# Progress and Trends in Image Processing Applications in Civil Engineering: Opportunities and Challenges

Authors: Ashwini A. Salunkhe; R. Gobinath; S. Vinay; Leo Joseph
Journal: Advances in Civil Engineering
 (2022)

---

## Abstract

Technological advancements in electronic storage have been trending for cloud computing. The revolution of this computer technology with machine learning and artificial intelligence has created prodigious platforms to the various disciplines of science and technology. Civil engineering is the oldest discipline, and due to the never-ending demand of this domain, it is rapidly adapting to newer computer techniques like image processing, deep learning, big data analysis, neural networks, building information modeling (BIM), unmanned aerial vehicle (UAV) system, digital image correlation (DIC), and many more. In the current paper, we portrayed the primary research and achievements of AI and image processing applications in the civil domain. The paper is divided in two parts. The first part provides analysis of existing methods along with examples relevant to the civil domain where it is incorporated. The second part elaborates scientometric study constituting 605 documents (Science Direct database) published in the last two decades. The bibliometrics are further used for producing analytical frameworks based on publications, citations, top journals, top institutions, and funding sources. This study serves as a guide for readers to identify research gaps and use the review for potential future study.


---

## Body

## 1. Introduction and Background

Paper forms have been a fundamental mode of transferring and storing data, but with technology advancements electronic storage has been trending for cloud computing. The revolution of this computer technology has created prodigious platforms to the various fields of science and technology which make researchers extend these techniques across all fields. However, when it comes to using digitalization to provide trustworthy data and enhance decision-making processes, the construction industry lags behind other industries [1]. According to Committee for European Construction Equipment (CECE), the civil industry is the least digitized sector. The climatic, environmental, economic, and sociopolitical challenges within civil projects make it difficult in digitization within the construction industry. Since last decade, computer-based monitoring systems and simulations using image processing, AI, ML, and others have piqued the interest among civil engineering community as a cost-effective and unobtrusive solution for image and video data collection and analysis in many construction organizations [1–13]. As a result, several applications were targeted by previous researchers in 3D imaging technologies for construction management, progress monitoring [9, 10], safety [14], and quality control [15, 16]. This review paper concisely emphasises imaging trends and techniques used for applications in civil engineering.Computer vision has evolved into a broad subject that includes everything from capturing raw data to extracting picture patterns and interpreting data [17]. Image processing involves analysis improving the pictorial information for better interpretation. Digital image processing (DIP) is concerned with performing systematic analysis of images using computers, whereas computer vision is concerned with creating models, data extraction, and information from photos [18]. DIP and computer vision technologies have been coupled in a wide range of research and project-based applications [19–21]. Figure 1 outlines evolution of image processing and its applications in diverse fields, starting from newspaper printing in 1920, while modern image processing with computing technology began in 1960 in the field astronomy, remote sensing, etc. [22]. Further developments were achieved in medical imaging, industrial inspection, and forensic applications in the early 2000s. In the twenty-first century, computer-based applications and image processing have become the need for a wide range of applications across all disciplines.Figure 1
Evolution of image processing.Mirsalar Kamari used a hand drafted and computer drawing for development of pattern recognition algorithm to understand structural notations marks from building pictures, and the knowledge was linked to analyze performance of structures. Here programming interface provides the freedom to integrate algorithms with software [23].Wei et al. carried out feasibility studies on using 3D imaging for construction and infrastructure management [24]. Likewise, Mróz et al. (2020) modelled the usefulness of digital image correlation (DIC), for testing concrete’s susceptibility to fire spalling [25]. While Zhang et al. verified the feasibility of DIC for quantifying displacement and strain fields in gravel particle breakage process under uniaxial process [26], del Rey Castillo et al. (2019) modelled the behaviour of fiber reinforced polymer (FRP) for obtaining displacement and strain field with DIC, where displacement measurement was satisfactory, but strain correlation showed dependency on direction of fibers (coincident with parallel direction of fibers and divergent for any other direction) [27].Globally, the interaction of new technologies is changing continuously, and with AI and ML systems it must cope with a variety of difficulties such as optimization and precise prediction. The advancement of image processing is providing innovative solutions to key challenges, achieving more reliable results at lower cost and time, opening new possibilities, and enabling the acquisition of accurate information from hypothesis testing systems. With digital cameras and sensors of high resolution, connectivity, and network communication, it is easy to upload the clicked photographs of construction activities [10] which are used for planning, monitoring, etc.Therefore, comprehensive guidance from literature review on DIP applications in civil engineering is presented herein. The purpose of this paper is to introduce the audience to the research by describing the potential of image processing in civil engineering. The following is a brief description of the sections in this paper. The advancement of image processing is generating a new philosophy in civil engineering, which is explored in Section2. Section 3 provides a detailed scientometric study of two-decade publications in this field, while Section 4 examines this research in terms of language, citations, top journals, top institutions, and funding agencies. Finally, Section 5 discusses the challenges and future scope of image processing before concluding with Section 6 with extracted outcomes and scope.


## 2. Civil Engineering Approach in Image Processing

The recent surge in the high demand for development and thrust for rapid infrastructure development has created an impact in modern technological advancements in this domain. Many researchers are working on finding the solutions to the problems raised at that society, technical expertise, and managers. Traditional methods for inspection or supervision involve visual assessment. However, a thorough inspection can disclose information that can be used to make various connections in order to determine structural integrity. This provides countless subdivisions to develop image processing applications (Figure2).Figure 2
Image processing applications in civil engineering.


### 2.1. Remote Sensing and GIS in Geological Applications

Remote sensing (RS) and GIS are linked feature extraction and analysis of ground surface data without coming in physical contact with the surface. It is utilised for urban planning, resource distribution, planning of water retention structures, detecting changes, hazards monitoring and mapping extent, estimating the damages [28], and so on. Despite the advances in RS technology, the applications in infrastructure and construction industry are limited [29]. Geotechnical information is imported for effective decision making [30]. In tunnel work, 2D geotechnical maps provide information of location and directions of weak planes. Particularly, complexity of tunnel work location and use of heavy machines like TBM (tunnel boring machines) make remote monitoring necessary. To overcome costly and time-consuming investigation, the image processing approach is adopted by researchers.Image acquisition of tunnel facing, its processing is used for rock mass rating, geological features (faults, joints, etc.), and interpolation technique on two consecutive images established in 3D visualization [31].In this context, Schindler et al. performed a virtual reality environment with 4D visualization by satellite and terrestrial based settlement data [7].One of the studies focused on the 1st of its trial on rock face characterization using images. The images on the rock surface provide comprehensive geological mapping. Additional rock mass parameters obtained by mapping are useful for tunnel stability estimation and support steps [32].


### 2.2. Hydrology and Water Resource Engineering

Management of earth’s water mass distribution and its interaction with natural and engineering material is essential for sustainable development. Computing technology offers a framework for interpretation as well as simulating the behaviour of water bodies along with their surroundings.Land use land cover (LULC) classification [33], changes detection [11, 12, 34–38], and its impact on water reservoirs were studied by Musaoglu, Moniruzzaman to help authorities and decision makers in changing quality conditions of reservoirs [39, 40].Chang et al. studied river behaviour using image processing technique on river-bed digital photographs. An image processing method was applied by fusing feedback pulse couple neural network (FPCNN) [41–44] and multilevel thresholding for image segmentation and grain size distribution [41]. In a similar context, Robert et al. have used image processing to establish granulometry of riprap. Along with rivers, the design of geotextile filters for water/soil retaining structures, drains, etc. requires morphological data where digital image processing is effectively applied for nonwoven geotextiles by Shobha in 1993 [19].Moreover, for applications in stream gauging, Lewis et al. evaluated the potential of unmanned aerial system (UAS) for quantifying flow in rivers [45].


### 2.3. Constructional Material

Construction engineering starts with study of construction material properties to acquire desired strength as per design requirement. The behaviour of any material is dependent on its morphological properties, and image processing is a very powerful tool to gather this information [46]. Digital image correlation (DIC) was used by McCormick et al., and Suleiman et al. measured deformation and strain by analyzing a pair of images of specimen surfaces taken before and after under various loading conditions [47, 48]. Image analysis for civil engineering materials, in the process of sampling and counting particles using specific methods for estimation of quantitative parameters, is useful for analyzing functions related to size distribution and anisotropy [46]. DIC has versatile abilities in adapting to any scalable applications allowing it to extend use in structural health monitoring [49].Aggregate gradation: in the construction sector, few are the key building materials such as cement, steel, and sand. Aggregates are one of the essential materials, and they play a vital role in asphalt pavements. Rutting of pavement gives major distress in pavement where aggregate gradation characteristics have a significant role in improving its performance [50]. The time-consuming methods aggregate gradation gives the motivation to the simple approach of image processing techniques. The method comprises curve fitting and image processing techniques to detect aggregates from cross-section images of cylindrical samples [51]. Anding et al. applied intelligent image processing algorithms in optical identification and analysis of mineral aggregates from construction and demolition waste [52].


### 2.4. Construction Execution and Safety

Execution work where there is major involvement of people and machinery together deals with site conditions, simultaneous activities, safety aspects, real-time progress, etc. Nowadays, drone technology is becoming very effective in making various surveys. Although the technology was introduced in risk assessment, disaster monitoring, etc., it is becoming famous in civil engineering works as well. The data gathered from these surveys can be analysed for civil engineering applications. Liu et al. used unmanned aerial vehicles (UAV) for construction execution [53]. The leading safety concerns at construction sites have attracted the researchers for smart techniques of safety assessment. This approach conducts dynamic measurement in the field of civil engineering through morphological operators for image filtering and a feature extraction technique: Hough Transform [14].


### 2.5. Repair and Maintenance

Every constructed structure, at some point of period, will require care in order to extend its lifespan. This part of the period focuses on the maintenance work of structures. Infrastructure challenges for constructing and maintaining the work include checking, making reports, and if required deciding what actions to be taken. Hence, the maintenance part requires the involvement of more people. In the major civil structures like bridges, high rise buildings, roads, etc., the crucial job is detecting the current state of structure and necessary maintenance. Due to vastness of the section, it is further categorized into two parts.


#### 2.5.1. Assessment of Structural Cracks

All structural assessment starts with crack detection, location, pattern, etc. However, the current evaluation techniques depend on people which attract more error and cost of execution. Valença et al. have developed image processing and laser scanning method to survey the cracks in bridges, while Kong et al. used DIP for fatigue crack detection in infrastructure [54]. Valença [55] used MCrack-TLS which is a combination of two technologies: image processing for crack detection (MCrack) and terrestrial laser scanning (TLS). In this technique, images are captured with different equipment depending on the location and type of structure, and further laser scanning helps to orthorectify the images for processing. Fangfei invented a comprehensive evaluator of defect detection in civil engineering structure [56]. The detailed module is shown in Figure 3.Figure 3
Defect detection system.


#### 2.5.2. Pavement Crack Monitoring and Analysis

Under pavement monitoring, asphalt resurfacing or overlay is a basic method of asphalt repair, and asphalt chip seals are the most common preventative maintenance technique employed by transportation/road agencies around the world. This requires precise percent aggregate in a thin bitumen sheet to avoid bleeding or aggregate loss. Ozdemir invented a standard procedure that uses digital image analysis to quantify the percent aggregate embedment in the bitumen layer [57].Further pavement monitoring entails the traditional assessment of pavement cracks via manual observation. To overcome the errors and drawbacks of conventional methods, a new automated method has been proposed for evaluation of crack sealing performance [58] while Chun in 2015 developed a semiautomatic method for asphalt pavement crack detection based on crack ratio [59]. Pavement surface drainage is a major concern to avoid hydroplaning in wet weather conditions. Mataei et al. created a pavement evaluation system that simulates the surface saturation condition from acquired images, and image processing generates indices to assess drainage quality [60].Uneven background illumination makes it challenging for automatic pavement defects detection and segmentation. For this, Oliveira and Correia (2014) recommended an image normalization approach [61]. This maintains the average intensities of the defect-containing regions while balancing the average intensities of defects-free regions. Figure 4 depicts image preprocessing for pavement. It is clear that background noise and sealed cracks are filtered to a large extent while pothole regions are kept for enabling further segmentation. The image-closing operator is one of the morphological operators used to improve segmented pothole shapes. This operator is illustrated in Figure 3 for extracting pothole shapes from pavement.Figure 4
Example of pothole extraction by 2D image pre- and postprocessing [62].The visual appearance of an object by the projective integral (PI) method is a one-dimensional pattern acquired by the addition of a set of pixels along the representative direction. PI is used to characterize the different forms of cracks based on the arithmetic mean computation of row or column pixels [63]. Figure 5 shows PI for longitudinal, transverse, and alligator forms of cracks and also a state of no crack.Figure 5
Images and SF response for various forms of cracks [64]. (a) Transverse crack. (b) Longitudinal crack. (c) Alligator crack. (d) No crack.
(a)(b)(c)(d)An image of adhesive loss and pull-out failure was used to identify the distress of a sealed crack. As shown in Figure6, the complement of the image has developed to represent a region of distress, further converting it to grayscale image for efficient computation which is further used to represent gradient mask by distinguishing texture of other regions from a uniform texture to sealed crack. A dilation operation is conducted on this for simplifying the sealed crack region from the background region (similar to morphological operation). The hole-filling method fills the noisy holes left over from the dilation process in the noncrack region. The finished image is binarized into black and white and compared to the binarized original image, resulting in the display of failure zones in sealed fractures.Figure 6
Crack sealing performance analysis from image processing (distress region extraction of sealed cracks) [58]. (a) Original figure. (b) Complement of image. (c) Grayscale image. (d) Binary gradient mask. (e) Dilated gradient mask. (f) Binary image with holes filled. (g) Binary image from the original figure. (h) Final extraction of the distress of a sealed crack.
(a)(b)(c)(d)(e)(f)(g)(h)


#### 2.5.3. Real-Time Monitoring

The large scale civil engineering structure itself demands maximum monitoring to reduce risk of failure or disaster management by providing early warnings. In this context, Vasileva et al. and Fukuda et al. came up with real-time and long-term remote monitoring of dynamic responses of major civil structures [5, 6]. Fukuda et al. used structure of target panel (predefined marks) transferring the signal in terms of movement captured by a connected camcorder. These signals are extracted in dynamic responses by the digital image processing software to compute displacement in real time while Vasileva et al. used FGPA (Field Programmable Gate Array) for high performance. Furthermore, Ehrhart et al. developed dynamic deformation monitoring by total station (for 3D movements) and image assisted total station (IATS) to compute absolute 3D coordinates and calculate structure’s natural frequencies [3]. However, to extract vibration behaviour of structure, sensors like accelerometers are used. The method uses IATS and marked targets fixed to the object. Total station telescope camera enabled real-time video streams with 10fps and 2”/px angular resolution. The electronic distance measurement (EDM) integrated in the total station gives precision to transform angular terms received from image processing to length units. The accuracy in case of dynamic application achieved better than 0.2 mm and for static applications better than 0.05 mm possible [4]. In 2018, Huang used stiffness coefficient and numerical analysis of a loaded structure for the life prediction module [65]. This evaluation system in predicting the life of structure involves a drone module for collecting data through image which is processed under image acquisition module.


## 2.1. Remote Sensing and GIS in Geological Applications

Remote sensing (RS) and GIS are linked feature extraction and analysis of ground surface data without coming in physical contact with the surface. It is utilised for urban planning, resource distribution, planning of water retention structures, detecting changes, hazards monitoring and mapping extent, estimating the damages [28], and so on. Despite the advances in RS technology, the applications in infrastructure and construction industry are limited [29]. Geotechnical information is imported for effective decision making [30]. In tunnel work, 2D geotechnical maps provide information of location and directions of weak planes. Particularly, complexity of tunnel work location and use of heavy machines like TBM (tunnel boring machines) make remote monitoring necessary. To overcome costly and time-consuming investigation, the image processing approach is adopted by researchers.Image acquisition of tunnel facing, its processing is used for rock mass rating, geological features (faults, joints, etc.), and interpolation technique on two consecutive images established in 3D visualization [31].In this context, Schindler et al. performed a virtual reality environment with 4D visualization by satellite and terrestrial based settlement data [7].One of the studies focused on the 1st of its trial on rock face characterization using images. The images on the rock surface provide comprehensive geological mapping. Additional rock mass parameters obtained by mapping are useful for tunnel stability estimation and support steps [32].


## 2.2. Hydrology and Water Resource Engineering

Management of earth’s water mass distribution and its interaction with natural and engineering material is essential for sustainable development. Computing technology offers a framework for interpretation as well as simulating the behaviour of water bodies along with their surroundings.Land use land cover (LULC) classification [33], changes detection [11, 12, 34–38], and its impact on water reservoirs were studied by Musaoglu, Moniruzzaman to help authorities and decision makers in changing quality conditions of reservoirs [39, 40].Chang et al. studied river behaviour using image processing technique on river-bed digital photographs. An image processing method was applied by fusing feedback pulse couple neural network (FPCNN) [41–44] and multilevel thresholding for image segmentation and grain size distribution [41]. In a similar context, Robert et al. have used image processing to establish granulometry of riprap. Along with rivers, the design of geotextile filters for water/soil retaining structures, drains, etc. requires morphological data where digital image processing is effectively applied for nonwoven geotextiles by Shobha in 1993 [19].Moreover, for applications in stream gauging, Lewis et al. evaluated the potential of unmanned aerial system (UAS) for quantifying flow in rivers [45].


## 2.3. Constructional Material

Construction engineering starts with study of construction material properties to acquire desired strength as per design requirement. The behaviour of any material is dependent on its morphological properties, and image processing is a very powerful tool to gather this information [46]. Digital image correlation (DIC) was used by McCormick et al., and Suleiman et al. measured deformation and strain by analyzing a pair of images of specimen surfaces taken before and after under various loading conditions [47, 48]. Image analysis for civil engineering materials, in the process of sampling and counting particles using specific methods for estimation of quantitative parameters, is useful for analyzing functions related to size distribution and anisotropy [46]. DIC has versatile abilities in adapting to any scalable applications allowing it to extend use in structural health monitoring [49].Aggregate gradation: in the construction sector, few are the key building materials such as cement, steel, and sand. Aggregates are one of the essential materials, and they play a vital role in asphalt pavements. Rutting of pavement gives major distress in pavement where aggregate gradation characteristics have a significant role in improving its performance [50]. The time-consuming methods aggregate gradation gives the motivation to the simple approach of image processing techniques. The method comprises curve fitting and image processing techniques to detect aggregates from cross-section images of cylindrical samples [51]. Anding et al. applied intelligent image processing algorithms in optical identification and analysis of mineral aggregates from construction and demolition waste [52].


## 2.4. Construction Execution and Safety

Execution work where there is major involvement of people and machinery together deals with site conditions, simultaneous activities, safety aspects, real-time progress, etc. Nowadays, drone technology is becoming very effective in making various surveys. Although the technology was introduced in risk assessment, disaster monitoring, etc., it is becoming famous in civil engineering works as well. The data gathered from these surveys can be analysed for civil engineering applications. Liu et al. used unmanned aerial vehicles (UAV) for construction execution [53]. The leading safety concerns at construction sites have attracted the researchers for smart techniques of safety assessment. This approach conducts dynamic measurement in the field of civil engineering through morphological operators for image filtering and a feature extraction technique: Hough Transform [14].


## 2.5. Repair and Maintenance

Every constructed structure, at some point of period, will require care in order to extend its lifespan. This part of the period focuses on the maintenance work of structures. Infrastructure challenges for constructing and maintaining the work include checking, making reports, and if required deciding what actions to be taken. Hence, the maintenance part requires the involvement of more people. In the major civil structures like bridges, high rise buildings, roads, etc., the crucial job is detecting the current state of structure and necessary maintenance. Due to vastness of the section, it is further categorized into two parts.


### 2.5.1. Assessment of Structural Cracks

All structural assessment starts with crack detection, location, pattern, etc. However, the current evaluation techniques depend on people which attract more error and cost of execution. Valença et al. have developed image processing and laser scanning method to survey the cracks in bridges, while Kong et al. used DIP for fatigue crack detection in infrastructure [54]. Valença [55] used MCrack-TLS which is a combination of two technologies: image processing for crack detection (MCrack) and terrestrial laser scanning (TLS). In this technique, images are captured with different equipment depending on the location and type of structure, and further laser scanning helps to orthorectify the images for processing. Fangfei invented a comprehensive evaluator of defect detection in civil engineering structure [56]. The detailed module is shown in Figure 3.Figure 3
Defect detection system.


### 2.5.2. Pavement Crack Monitoring and Analysis

Under pavement monitoring, asphalt resurfacing or overlay is a basic method of asphalt repair, and asphalt chip seals are the most common preventative maintenance technique employed by transportation/road agencies around the world. This requires precise percent aggregate in a thin bitumen sheet to avoid bleeding or aggregate loss. Ozdemir invented a standard procedure that uses digital image analysis to quantify the percent aggregate embedment in the bitumen layer [57].Further pavement monitoring entails the traditional assessment of pavement cracks via manual observation. To overcome the errors and drawbacks of conventional methods, a new automated method has been proposed for evaluation of crack sealing performance [58] while Chun in 2015 developed a semiautomatic method for asphalt pavement crack detection based on crack ratio [59]. Pavement surface drainage is a major concern to avoid hydroplaning in wet weather conditions. Mataei et al. created a pavement evaluation system that simulates the surface saturation condition from acquired images, and image processing generates indices to assess drainage quality [60].Uneven background illumination makes it challenging for automatic pavement defects detection and segmentation. For this, Oliveira and Correia (2014) recommended an image normalization approach [61]. This maintains the average intensities of the defect-containing regions while balancing the average intensities of defects-free regions. Figure 4 depicts image preprocessing for pavement. It is clear that background noise and sealed cracks are filtered to a large extent while pothole regions are kept for enabling further segmentation. The image-closing operator is one of the morphological operators used to improve segmented pothole shapes. This operator is illustrated in Figure 3 for extracting pothole shapes from pavement.Figure 4
Example of pothole extraction by 2D image pre- and postprocessing [62].The visual appearance of an object by the projective integral (PI) method is a one-dimensional pattern acquired by the addition of a set of pixels along the representative direction. PI is used to characterize the different forms of cracks based on the arithmetic mean computation of row or column pixels [63]. Figure 5 shows PI for longitudinal, transverse, and alligator forms of cracks and also a state of no crack.Figure 5
Images and SF response for various forms of cracks [64]. (a) Transverse crack. (b) Longitudinal crack. (c) Alligator crack. (d) No crack.
(a)(b)(c)(d)An image of adhesive loss and pull-out failure was used to identify the distress of a sealed crack. As shown in Figure6, the complement of the image has developed to represent a region of distress, further converting it to grayscale image for efficient computation which is further used to represent gradient mask by distinguishing texture of other regions from a uniform texture to sealed crack. A dilation operation is conducted on this for simplifying the sealed crack region from the background region (similar to morphological operation). The hole-filling method fills the noisy holes left over from the dilation process in the noncrack region. The finished image is binarized into black and white and compared to the binarized original image, resulting in the display of failure zones in sealed fractures.Figure 6
Crack sealing performance analysis from image processing (distress region extraction of sealed cracks) [58]. (a) Original figure. (b) Complement of image. (c) Grayscale image. (d) Binary gradient mask. (e) Dilated gradient mask. (f) Binary image with holes filled. (g) Binary image from the original figure. (h) Final extraction of the distress of a sealed crack.
(a)(b)(c)(d)(e)(f)(g)(h)


### 2.5.3. Real-Time Monitoring

The large scale civil engineering structure itself demands maximum monitoring to reduce risk of failure or disaster management by providing early warnings. In this context, Vasileva et al. and Fukuda et al. came up with real-time and long-term remote monitoring of dynamic responses of major civil structures [5, 6]. Fukuda et al. used structure of target panel (predefined marks) transferring the signal in terms of movement captured by a connected camcorder. These signals are extracted in dynamic responses by the digital image processing software to compute displacement in real time while Vasileva et al. used FGPA (Field Programmable Gate Array) for high performance. Furthermore, Ehrhart et al. developed dynamic deformation monitoring by total station (for 3D movements) and image assisted total station (IATS) to compute absolute 3D coordinates and calculate structure’s natural frequencies [3]. However, to extract vibration behaviour of structure, sensors like accelerometers are used. The method uses IATS and marked targets fixed to the object. Total station telescope camera enabled real-time video streams with 10fps and 2”/px angular resolution. The electronic distance measurement (EDM) integrated in the total station gives precision to transform angular terms received from image processing to length units. The accuracy in case of dynamic application achieved better than 0.2 mm and for static applications better than 0.05 mm possible [4]. In 2018, Huang used stiffness coefficient and numerical analysis of a loaded structure for the life prediction module [65]. This evaluation system in predicting the life of structure involves a drone module for collecting data through image which is processed under image acquisition module.


## 2.5.1. Assessment of Structural Cracks

All structural assessment starts with crack detection, location, pattern, etc. However, the current evaluation techniques depend on people which attract more error and cost of execution. Valença et al. have developed image processing and laser scanning method to survey the cracks in bridges, while Kong et al. used DIP for fatigue crack detection in infrastructure [54]. Valença [55] used MCrack-TLS which is a combination of two technologies: image processing for crack detection (MCrack) and terrestrial laser scanning (TLS). In this technique, images are captured with different equipment depending on the location and type of structure, and further laser scanning helps to orthorectify the images for processing. Fangfei invented a comprehensive evaluator of defect detection in civil engineering structure [56]. The detailed module is shown in Figure 3.Figure 3
Defect detection system.


## 2.5.2. Pavement Crack Monitoring and Analysis

Under pavement monitoring, asphalt resurfacing or overlay is a basic method of asphalt repair, and asphalt chip seals are the most common preventative maintenance technique employed by transportation/road agencies around the world. This requires precise percent aggregate in a thin bitumen sheet to avoid bleeding or aggregate loss. Ozdemir invented a standard procedure that uses digital image analysis to quantify the percent aggregate embedment in the bitumen layer [57].Further pavement monitoring entails the traditional assessment of pavement cracks via manual observation. To overcome the errors and drawbacks of conventional methods, a new automated method has been proposed for evaluation of crack sealing performance [58] while Chun in 2015 developed a semiautomatic method for asphalt pavement crack detection based on crack ratio [59]. Pavement surface drainage is a major concern to avoid hydroplaning in wet weather conditions. Mataei et al. created a pavement evaluation system that simulates the surface saturation condition from acquired images, and image processing generates indices to assess drainage quality [60].Uneven background illumination makes it challenging for automatic pavement defects detection and segmentation. For this, Oliveira and Correia (2014) recommended an image normalization approach [61]. This maintains the average intensities of the defect-containing regions while balancing the average intensities of defects-free regions. Figure 4 depicts image preprocessing for pavement. It is clear that background noise and sealed cracks are filtered to a large extent while pothole regions are kept for enabling further segmentation. The image-closing operator is one of the morphological operators used to improve segmented pothole shapes. This operator is illustrated in Figure 3 for extracting pothole shapes from pavement.Figure 4
Example of pothole extraction by 2D image pre- and postprocessing [62].The visual appearance of an object by the projective integral (PI) method is a one-dimensional pattern acquired by the addition of a set of pixels along the representative direction. PI is used to characterize the different forms of cracks based on the arithmetic mean computation of row or column pixels [63]. Figure 5 shows PI for longitudinal, transverse, and alligator forms of cracks and also a state of no crack.Figure 5
Images and SF response for various forms of cracks [64]. (a) Transverse crack. (b) Longitudinal crack. (c) Alligator crack. (d) No crack.
(a)(b)(c)(d)An image of adhesive loss and pull-out failure was used to identify the distress of a sealed crack. As shown in Figure6, the complement of the image has developed to represent a region of distress, further converting it to grayscale image for efficient computation which is further used to represent gradient mask by distinguishing texture of other regions from a uniform texture to sealed crack. A dilation operation is conducted on this for simplifying the sealed crack region from the background region (similar to morphological operation). The hole-filling method fills the noisy holes left over from the dilation process in the noncrack region. The finished image is binarized into black and white and compared to the binarized original image, resulting in the display of failure zones in sealed fractures.Figure 6
Crack sealing performance analysis from image processing (distress region extraction of sealed cracks) [58]. (a) Original figure. (b) Complement of image. (c) Grayscale image. (d) Binary gradient mask. (e) Dilated gradient mask. (f) Binary image with holes filled. (g) Binary image from the original figure. (h) Final extraction of the distress of a sealed crack.
(a)(b)(c)(d)(e)(f)(g)(h)


## 2.5.3. Real-Time Monitoring

The large scale civil engineering structure itself demands maximum monitoring to reduce risk of failure or disaster management by providing early warnings. In this context, Vasileva et al. and Fukuda et al. came up with real-time and long-term remote monitoring of dynamic responses of major civil structures [5, 6]. Fukuda et al. used structure of target panel (predefined marks) transferring the signal in terms of movement captured by a connected camcorder. These signals are extracted in dynamic responses by the digital image processing software to compute displacement in real time while Vasileva et al. used FGPA (Field Programmable Gate Array) for high performance. Furthermore, Ehrhart et al. developed dynamic deformation monitoring by total station (for 3D movements) and image assisted total station (IATS) to compute absolute 3D coordinates and calculate structure’s natural frequencies [3]. However, to extract vibration behaviour of structure, sensors like accelerometers are used. The method uses IATS and marked targets fixed to the object. Total station telescope camera enabled real-time video streams with 10fps and 2”/px angular resolution. The electronic distance measurement (EDM) integrated in the total station gives precision to transform angular terms received from image processing to length units. The accuracy in case of dynamic application achieved better than 0.2 mm and for static applications better than 0.05 mm possible [4]. In 2018, Huang used stiffness coefficient and numerical analysis of a loaded structure for the life prediction module [65]. This evaluation system in predicting the life of structure involves a drone module for collecting data through image which is processed under image acquisition module.


## 3. Review of Image Processing Tools and Algorithms

The summary of image processing tools researchers have used with their usefulness in their study is shown in Table1. In summary, notable researchers in the field of material image processing have used wavelet transformation methods, although the method has the limitation that particle diameters cannot change by more than a factor of two in area analysis [66]. This necessitates particle separation prior to picture acquisition. In addition, the researchers’ software selection scenario reveals a preference for MATLAB software. This could be due to the image processing library’s built-in tools, which include standard algorithms, visualization functions, and applications.Table 1
Image processing tools in civil engineering applications.
ResearchersTools and techniquesTool applicationUsefulness in researchSou-Sen Leu et al. (2005) [31]Borland C++, Slice dicerBorland C++ developed an image processing subsystem while Slice dicer integrated 3D reconstruction and visualization in the main systemTechnique can effectively guide aspirants to visualize geological features from tunnel excavationYoshio Fukuda et al. (2010) [5]Digital camcorder and a notebook computer equipped with DirectShow technology (Microsoft corporation) software using C++ languageImage processing modules were implemented with DirectShow software. The module helps to detect geometric patterns with trigonometric transformsSystem is capable for real-time dynamic response measurement and ideal for monitoring civil engineering structuresNick McCormick et al. (2012) [47]ARAMIS softwareThe deformation measurement was initially done on integer and further targeted in subpixel interpolation by 2D spline fitting and Newton Raphson techniqueFull-field deformation measurements of civil structures with DIC is effective and can be tested for various sites with sensorsMorteza Vadood et al. (2013) [51]MATLAB R2006aDistance and modified watershed transform methodology with Canny’s filterThe method effectively detects aggregate gradationKatharina Anding et al. (2013) [52]Halcon 8.0.3A machine vision software with feature algorithms for supervised learning classifierAlgorithms integrated for sorting optical aggregates in construction demolition waste (CDW)Peter Liu et al. (2014) [53]Unmanned aerial vehicle (UAV), ground controlling station (GCS)UAV acquires aerial perspective info and GCS software performs simulations functionPotential use of UAV in mapping methods, geometric analysis, and civil engineering applicationsHyon-Sohk Ohm et al. (2014) [66]MATLAB R2013aImage analysis method using wavelet transformation coded in software gives sediment size distributionSedimaging and sieving test for nine soils give identical results on soil gradation plot by both methodsBehrouz Mataeia et al. (2017) [60]Innovative device & softwareDevice consists of several parts: imaging, saturation, constant lighting, temperature recorder, and movement arrangement. Image processing software using sharelet transformPavement surface drainage quality was examined and classified into “good,” “normal,” and “bad”Can Wang et al. (2018) [67]Particle flow code (PFC) 3D (version 5)Software allows three-dimensional mapping of restitution tests for mapping segregated mixtureMaterial heterogeneity from digital image processing generates discrete element model which helps to examine influence of particle properties on segregationSaghafi et al. (2019) [68]Img. Pro. Plus 6Software measurement feature counts total aggregate loss by summing up pixels with loss areaAbrasion of slurry seal mixtures was investigated to find out loss of aggregate and bitumenFaria Borges et al. (2019) [69]SkyScan reconstruction software (CTAn)Software was capable of scanning hundreds of 2D images and combined to project 3D imageAlgorithm generated supervised information (grain size, void ratio, and soil tortuosity) through material microtomography for sandy materialBidgoli et al. (2020) [70]MATLAB R2014aMATLAB code was developed to determine the percentage of pixels for each surface fractureDamage investigation between binder and aggregate was done by moisture susceptibility of asphalt mixture by registration image-processing methodDetcev et al. (2014)Eight digital cameras and two projectorsCamera was used for capturing concrete beam which is being deformed by actuator loading. Projectors were used for performing 3D reconstruction and point cloud processing for deflection measurementPrecise deflection measurement of concrete beam loaded by hydraulic actuatorImage processing starts with the image acquisition process either with powerful camera capturing images [19, 47, 48] or video recording [66, 67]. The typical image analysis procedure can be summarized: (i) image acquisition, (ii) image preprocessing, (iii) image enhancement, (iv) image segmentation, (v) image postprocessing, and (vi) detection or prediction. Image preprocessing can be done with various algorithms with image graying like single component method, maximum weighted method, best weighted average method [2], etc. Further image filtering processes usually retain useful characteristics of image by removing noise. It can be done by mean filtering method, median filtering method, and morphology filtering method. Segmentation process labels every pixel for a certain characteristic which is useful for object recognition, edge detection, etc. Table 2 provides the summary of such algorithms and their briefing for getting insights of algorithms.Table 2
Algorithmic analysis for civil engineering application.
ReferencesApplicationAlgorithmBrief[29, 58, 59, 61, 64]Intelligent approach in pavement crack classificationImage processing and machine learning algorithmsImage processing tool: nonlocal means, steerable filter, projective integral, and image thresholding machine learning tool: support vector machine and artificial colony bee[2]Crack detection algorithm for crack skeletonZhang-Suen skeleton extraction algorithmOTSU automatic threshold, gamma filtering, and gamma image enhancement[71]Damage evaluation through tunnel crackImage threshold segmentation algorithm based on fuzzy setImage analysis with cracks characteristic[22, 46]Image analysis for civil engineering materialsProbabilistic models: tessellation and polyphaseImage acquisition, extraction of feature, segmentation, and binary image treatment[72]Interphase contact angle for unsaturated geomaterialImage-based contact angle measurementZero-intercept approach compared with liquid pixel approach and regression approach[73]Aggregate gradationLaser imaging and neural networkWavelet transform to extract features of material gradation[51]Separate aggregate from bitumen mastic and split joint aggregateNumerical and image processing techniques such as fitting equation and colour space systemBinary images with colour space method can accurately and easily remove bitumen from cross-sections images compared to the shape method. Then, the distance and modified watershed transform was applied on the obtained binary images, and the aggregate gradation was extracted from images[52]Optical identification of recycled aggregates from CDWIntelligent image processing algorithmsFeature extraction and machine learning classifiers[66]Size distribution of coarse-grained soil (0.075–2.0 mm)Image analysis based on wavelet transformationWavelet transformation decomposes soil image of kth pixel size intok decomposition level to produce size distribution[57]Quantifying aggregate embedded in asphalt binderPeak valley method, embedment of each aggregate method, and surface coverage area methodA digital image analysis to calculate percent aggregate embedded in asphalt[4, 5]Displacement measurement system for real-time monitoring for large civil structuresImage processing algorithm for target panel detectionGeometry of target panel analysis by position of white spots on panel[2, 4, 47, 74]Measurement of deformation of structureProcessing pair of images under different loading conditionsHigh resolution cameras for displacement measurement


## 4. Scientometric Analysis of Image Processing Applications in Civil Engineering

Scientometrics is the analysis of quantitative qualities and qualities of science and scientific study. The location of emphasis is on inquiries which concentrate on the processes and mechanisms of science that are explored using statistical mathematical approaches. Scientometrics is the area of study that is concerned with the collection, analysis, and evaluation of academic publications. The application of the methodologies used in bibliometrics (i.e., scientometrics) is a subfield of bibliometrics. There are several concerns in major research that need to be examined, including the study of the influence of research articles and academic journals, the comprehension of scientific citations, and the utilisation of research measurements in policy and management. Although there is some overlap, in practice there is a strong overlap between the areas of scientometrics and other scientific topics such as information systems, information science, science of science policy, sociology of science, and metascience. There have been critics who suggest that an overreliance on scientometrics has established a system of perverse incentives, which has produced a publish or perish atmosphere that encourages low-quality research. In 1969, Vasily Nalimov, in his capacity as the Chairman of the Russian Academy of Sciences’ Institute of Scientific Information, proposed that the field of scientometrics be renamed Naukometriya, which translates to “Scientometrics” in English [54, 75–77]. In modern scientific methodology, nearly all of the effort is focused on the work of Derek J. de Solla Price and Eugene Garfield. Later on, this individual established the Scientific Index and started the Institute for Scientific Information (also known as ISI). This is commonly used for scientometric analysis. In 1978, the Scientometrics journal was launched for committed researchers only. The expansion in the industrialization of science led to an increase in publications and research outputs, as well as the rise of computers, which allowed for effective analysis of the data. While the focus of the sociological study of science was on the conduct of scientists, the focus of the scientometric study was on examination of publications [78]. Similarly, science and technology analytics are referred to as the scientific and empirical examination of science and its outcomes.In this work, we had tried to analyze the progress of research in the image processing applications vertical to understand the quantum of research work being undertaken in various levels and its field implications. We had used SCOPUS core collection, a product of Elsevier Co for the analysis fully to understand how much research work is in progress. A multilevel bibliometric information retrieval was done on SCOPUS website on single day June 4, 2021, to avoid bias caused by updates of open databases. First, a search was made on SCOPUS with the following query:((TITLE-ABS-KEY(image processing) AND TITLE-ABS-KEY(civil engineering)) AND PUBYEAR >1999)The search comprises a total of 605 documents, containing various open access journals and articles in which all peer-reviewed scholarly articles are online and available without restriction. The distribution is shown in Figure7.Figure 7
Journal distribution.The documents are collected with details of year of publication, type of document like conference, editorial, book chapters, etc., keywords, name of author(s) with affiliation(s), and country. The bibliometric analysis was carried out for identifying trends, leading authors, funding agencies, and countries. The major key points in the publication datasets which relate to the civil engineering field were “deformation,” “construction equipment,” “concrete crack,” “structural health monitoring,” “building material,” etc. as shown in Figure8(a). For the dataset, key points with computer techniques were “digital image correlation,” “algorithm,” “data processing,” “image enhancement,” “image segment,” etc. as shown in Figure 8(b).Figure 8
(a) Key points related to civil engineering. (b) Key points related to image processing.
(a)(b)


## 5. Database Analysis

### 5.1. Type of Document Analysis with Language

For the +20-year period of 2000 to 2021, researchers globally published 604 documents related to image processing in civil engineering. Using this SCOPUS data, we performed categorial analysis based on year of publication, type of document, language, and country origin. Figure9 shows that research conducted over two decades has consistently contributed the most to conference papers, followed by articles. Data shows that the highest record was 370 (61.25828%) documents from conference paper followed by articles 173 (28.64238%), followed by conference review 37 (6.12582%) and review 11 (1.82119). The remaining 13 documents (2.152317%) were published as book, book chapters, editorial erratum, and short survey which constitute 1392 citations due to one Springer book having the highest citation of 1343. The maximum number of papers 574 (95.03%) was published in English language followed by Chinese 22 (3.64%). Hence, around 98.67% are in these two languages out of total documents findings. The remaining 1.49% were scattered in 5 different languages “GJSPT” around the globe constituting the language with papers: German (3), Japanese (2), Spanish (2), Portuguese (1), and Turkish (1). However, one of the papers is available in two languages, English and Japanese. By looking at the overall scenario in Figure 9, it can be concluded that image processing in the civil engineering field comprises the maximum number of papers in English language with very few documents from other 6 languages.Figure 9
Publication overview with language and country for the period from 2001 to 2021.


### 5.2. Publication Analysis

In the publication analysis process, bibliometrics allows us to perform statistical projection in acquiring reliable indicators. Along with the number of research publications, citations have been one of the most important indicators for research quality. Citations are a form of scientific performance metric that measures the impact of an article. Figure10 depicts the multiple axis (4Y) combination to one axis (X) for better understanding of publication and citation trends from research output in image processing in civil engineering during the period 2000–2021. The median for the publications is 22, while the deviation value for publication is 20. The average publications were 27 per year, and the highest number of documents published was 70 in 2012. It indicates that the global growth rate is constant in the period from 2000 to 2008. The data output shows stagnant growth in the first 10 years, and suddenly it reached a peak in 2011, which stayed there for the next 2 years.Figure 10
Publication trend from 2000 to 2021.


### 5.3. Top Journals Contributing More Research

Targeted at analyzing the research contribution from top sources, Figure11 shows most article contributed journals ranking in the top ten list. Journal named “Applied Mechanics & Materials” is conducting the most rigorous research in the field of image processing for civil engineering. The scientific explanation could be that they prefer to publish comprehensive volumes on specific topics rather than individual papers by authors. With around 30+ papers gap, the remaining sources come into the view like SPIE proceedings (dedicated to advancing the science and application of light and light-based technologies), Advanced Materials Research, etc.Figure 11
Top 10 sources for 2001 to 2021.


### 5.4. Top Universities or Institutions

The world of scientific research has become more collaborative with research papers by authors from multiple affiliations (same or different country, continent). This affiliation represents research integrity of paper and also the place of research conducted. China and the United States of America (USA) continue to be the most prolific vendors in generating quality research output. Figure12 shows the ranking of top 15 affiliations by researchers, where 60% of researchers’ affiliations are with universities from China, followed by USA (20%).Figure 12
Top 15 affiliations for 2001 to 2021.


### 5.5. Top Funding Sources

Research funding is a common feature for all mature innovation systems. The data presented in Figure13 does not focus on the quantity of money that agencies have given, but rather on the number of research entities that have been supported. The majority of the articles in this scientometric study were funded by China’s National Natural Science Foundation, which promotes and finances basic research and applied research in China. The second largest agency is from the USA, “National Science Foundation,” which supports fundamental research in the nonmedical field of science and engineering.Figure 13
Top 10 funding agencies for 2001 to 2021.


## 5.1. Type of Document Analysis with Language

For the +20-year period of 2000 to 2021, researchers globally published 604 documents related to image processing in civil engineering. Using this SCOPUS data, we performed categorial analysis based on year of publication, type of document, language, and country origin. Figure9 shows that research conducted over two decades has consistently contributed the most to conference papers, followed by articles. Data shows that the highest record was 370 (61.25828%) documents from conference paper followed by articles 173 (28.64238%), followed by conference review 37 (6.12582%) and review 11 (1.82119). The remaining 13 documents (2.152317%) were published as book, book chapters, editorial erratum, and short survey which constitute 1392 citations due to one Springer book having the highest citation of 1343. The maximum number of papers 574 (95.03%) was published in English language followed by Chinese 22 (3.64%). Hence, around 98.67% are in these two languages out of total documents findings. The remaining 1.49% were scattered in 5 different languages “GJSPT” around the globe constituting the language with papers: German (3), Japanese (2), Spanish (2), Portuguese (1), and Turkish (1). However, one of the papers is available in two languages, English and Japanese. By looking at the overall scenario in Figure 9, it can be concluded that image processing in the civil engineering field comprises the maximum number of papers in English language with very few documents from other 6 languages.Figure 9
Publication overview with language and country for the period from 2001 to 2021.


## 5.2. Publication Analysis

In the publication analysis process, bibliometrics allows us to perform statistical projection in acquiring reliable indicators. Along with the number of research publications, citations have been one of the most important indicators for research quality. Citations are a form of scientific performance metric that measures the impact of an article. Figure10 depicts the multiple axis (4Y) combination to one axis (X) for better understanding of publication and citation trends from research output in image processing in civil engineering during the period 2000–2021. The median for the publications is 22, while the deviation value for publication is 20. The average publications were 27 per year, and the highest number of documents published was 70 in 2012. It indicates that the global growth rate is constant in the period from 2000 to 2008. The data output shows stagnant growth in the first 10 years, and suddenly it reached a peak in 2011, which stayed there for the next 2 years.Figure 10
Publication trend from 2000 to 2021.


## 5.3. Top Journals Contributing More Research

Targeted at analyzing the research contribution from top sources, Figure11 shows most article contributed journals ranking in the top ten list. Journal named “Applied Mechanics & Materials” is conducting the most rigorous research in the field of image processing for civil engineering. The scientific explanation could be that they prefer to publish comprehensive volumes on specific topics rather than individual papers by authors. With around 30+ papers gap, the remaining sources come into the view like SPIE proceedings (dedicated to advancing the science and application of light and light-based technologies), Advanced Materials Research, etc.Figure 11
Top 10 sources for 2001 to 2021.


## 5.4. Top Universities or Institutions

The world of scientific research has become more collaborative with research papers by authors from multiple affiliations (same or different country, continent). This affiliation represents research integrity of paper and also the place of research conducted. China and the United States of America (USA) continue to be the most prolific vendors in generating quality research output. Figure12 shows the ranking of top 15 affiliations by researchers, where 60% of researchers’ affiliations are with universities from China, followed by USA (20%).Figure 12
Top 15 affiliations for 2001 to 2021.


## 5.5. Top Funding Sources

Research funding is a common feature for all mature innovation systems. The data presented in Figure13 does not focus on the quantity of money that agencies have given, but rather on the number of research entities that have been supported. The majority of the articles in this scientometric study were funded by China’s National Natural Science Foundation, which promotes and finances basic research and applied research in China. The second largest agency is from the USA, “National Science Foundation,” which supports fundamental research in the nonmedical field of science and engineering.Figure 13
Top 10 funding agencies for 2001 to 2021.


## 6. Challenges and Future Suggestions (Gap and Issues)

Technical reliability of image processing addresses the uncertainty of construction sites, severity of weather conditions, and real-time image acquisition for smooth application in the civil domain. Despite the fact that civil engineering has a plethora of subdivisions, some of them have excelled in image processing applications. Some of these subdivisions come from “infrastructure repair and maintenance,” which has been extensively researched by researchers in conjunction with computer vision. For deep understanding between the research gap and presenting it in a narrative manner, a network visualization graph of keywords is helpful. Network visualization graph in Figure14 features the relevance for major image processing research work carried out on the basis of keywords. The graphics use colour, geometry for interfacing, and overall reasoning. The thematic study results in major work in areas of remote sensing, motion or deformation estimates, and disaster analysis. However, due to technical limitations such as restricting the image frame and depth to which data extraction is necessary, some domains such as geotechnical engineering and concrete technology remain unexplored.Figure 14
Network of keyword interrelation.On one hand, civil work is responsible for developing the most efficient structures using modern equipment in order to deal with harsh site circumstances, while on the other hand, it is also responsible for maintaining and repairing ageing infrastructure [73]. The versatile nature of civil work makes it more complex to develop one tool or algorithm for a particular job and standardize for future similar works. As civil engineering activities are carried out in the field, a picture frame for this sort of work comprises a lot of noise and extraneous items, making it challenging for image acquisition and training the model for object detection. This makes image classification difficult, as each class requires hundreds of images. For better performance of the model, large amounts of data with accurate labelling are required. As a result, it is crucial to operate on the best hardware that supports the complex operations and processing of images.


## 7. Extraction Outcome and Potential Scope

In civil engineering, computer vision refers to artificial intelligence and machine learning, and further it can be closely related to image processing. The imaging applications give opportunity to civil engineers for the challenges of handling simultaneous work. From this automation, major benefits can be acquired by construction material companies. Various image processing techniques have been developed and used in the civil engineering field, yet so many areas are untouched.This paper provides an up-to-date review of image processing applications in the field of civil engineering. Image acquisition that has been used in controlled indoor setup and on-site equipment is described. Developing trends toward portable devices like digital camera, UAV, and mobile camera to acquire images are noted. From the research carried out in this article, it can be seen that digital imaging research in the context of civil engineering has heavily focused on object tracking like crack identification, classification, etc., and activity recognition like displacement or deformation, etc. The benefit of this approach is that it allows you to analyze the deformations without direct physical contact with structure. The applications of automation from image processing can save time and cost in some tedious jobs like material characterization in CDW, measuring strain, strength, and displacements. The future scope of this study can be considered as the use of handy tools (mobile camera) for image acquisition, transferring, and processing image data. In addition, one of the potential areas is making an image cloud database for civil engineering work. This will help multiple aspirants to easily access data and collaborate for the work in developing algorithms for more better results.Finally, this research paper is projection of progress and trends of image processing in virtue of civil engineering. This will help researchers to get background, challenges, and future scope of using image processing work in civil engineering.



---

*Source: 6400254-2022-05-05.xml*