Patent Description:
Structural health monitoring (SHM) is an essential tool for the effective maintenance of civil infrastructure, with a number of SHM systems employed in real-world applications.

Data acquisition of structural responses is a fundamental step in SHM systems where the data is subsequently processed for condition assessment and decision-making.

Displacement responses from a civil engineering structure are considered to be informative in evaluating the structure's current structural condition and safety. As it is directly related to structural stiffness and loadings, displacement can be an indicator of structural changes and excessive external loadings. For example, the plastic deformation ratios of building structures are estimated by drift displacement data.

Most design codes used in modern countries specify maximum displacement levels for civil structures to assure structural safety and usability. Thus, displacement information is commonly employed for infrastructure maintenance purposes.

Displacement sensors, such as linear variable differential transformers (LVDT) and strain-based displacement transducers, are widely adopted for conducting displacement measurements in practice.

These sensors are typically placed between a target point on a structure and a fixed reference point, measuring relative displacements. A sensor's installation requires additional supporting structures with respect to the fixed references that are often unavailable or difficult to prepare in field testing involving full-scale civil engineering structures. Furthermore, vibrations of the supporting structures can significantly degrade measurement accuracy. Thus, using traditional sensors to measure displacement responses from a full-scale structure is considered to be inefficient.

Recent research efforts have focused on effectively addressing this issue and providing a practical means for displacement measurement. These efforts include the development of indirect displacement estimation algorithms to convert other physical quantities, such as acceleration and strain, to displacement and the applicability for this purpose of relatively new sensors, including the laser Doppler vibrometer (LDV), global positioning systems (GPS) and computer vision-based approaches.

Precision in measurement is a fundamental aspect to record the behavior of a bridge deck against any external action and the passing of time. However, this precision is especially relevant to evaluate its behavior against the action of external loads such as the transit of vehicles, pedestrians or even wind. Depending on the type of structure, the amplitude of these oscillations can vary from several centimeters in the lightest or longest structures, to the order of a millimeter or a few millimeters in the most rigid structures. Having sufficient precision in measuring the vertical movement of a, i.e. a bridge deck is, therefore, a guarantee when it comes to identifying anomalous behavior, since the difference between the theoretical oscillation and the registered oscillation is a reliable parameter to identify early processes of deterioration or structural anomalies.

So, we conclude that measuring displacement with the required precision is a critical indicator of the condition of an infrastructure, but to assess the criticality of such measurement it is important that it is done in real time and under a continuous monitorization. All these factors together make of displacement monitorization a critical indicator of the infrastructure behavior in real time, supporting the comparison between the theoretical model of the infrastructure and the empirical one.

A significant amount of existing infrastructures require inspection. Thus, monitorization solutions capable of being easily installed on existing infrastructures are required. The easier to install and cost competitive, the better.

Autonomy is also a fundamental requirement for these displacement measuring solutions, as most of the times infrastructures are located in remote areas. So, to support the continuous operation in an unattended environment the solution must be designed attending this requirement.

Solutions described in the technical and scientific literature are, most of the times, tailor made solutions, costly ones or solutions requiring support, what makes then unsuitable for a generalized application.

There are currently a range of devices on the market that can be used to measure the deflection in bridge decks, although none of them constitutes an integral tool for the direct measurement of the deflection parameter.

There is below a brief description of different measurement systems available in the market and their limitations in relation to the precision of the measurement and their autonomy as monitoring systems.

Topography is a highly accurate direct measurement procedure, although it is not an unattended technology. Currently, there are robotic stations on the market capable of performing precision measurements and operating autonomously for a certain period of time, however, these are equipment with a permanent need for monitoring and surveillance. The absence of remote control over the measurement records may lead to possible undetected errors only revealed in the periodic download of the data to a computer and its subsequent processing which, in turn, requires intervention of specialized technicians in the instrumentation process. Likewise, this lack of remote control means that the monitoring of other operational parameters such as equipment status, power or battery condition must be carried out in the same location, once again requiring the intervention of a specialist technician.

Other systems for measuring vertical movement, such as GPS systems, digital distance meters or measuring movement by water levels, although they make direct measurements of the vertical movement, do not meet the precision required for the detection of very small range movement oscillations.

GPS measurement systems, with measurement errors of the order of centimeters, do not have sufficient precision to monitor the vertical movements of a bridge deck, since what is sought is high precision in the continuous recording of movements to detect any anomalous displacement, or out of range, of the deck on the passage of a moving load.

On the other hand, the digital distance meters that exist in the market work by emitting a laser signal to a target and calculating the time of flight that said signal takes to go back and forth the target, or the phase difference between emitted and reflected signals from the target. In this way, if we want to measure the vertical movement of a board, the EDM must measure the vertical distance between the deck and the ground. That is why, in this case, the target to which the laser signal is sent plays a fundamental role in the accuracy of the measurement obtained, but in turn it must be located on the surface of the obstacle saved (rivers, valleys, roads, etc.), being certainly complex that this objective surface remains invariable in time.

In the case of the measurement of movements by water levels, there are hydraulic instrumentation systems on the market capable of measuring the vertical movement by means of the variation of the height of a liquid contained in a container with respect to an initial height used as a reference. However, the accuracy of the measurement is influenced by various factors such as the presence of bubbles within the system, temperature changes, etc. It is also a technology that requires continuous maintenance for the deaeration of the liquid used.

There are also other systems on the market that make it possible to determine the vertical movements of a deck by indirectly measuring the elongation of a material attached to the surface of the bridge. This extensometric measurement procedure is based on the piezoresistive effect, that is, the property that some conductive and semiconductor materials have to vary their electrical resistance due to the effect of a deformation. In this way, once the deformation of the material is known, it is possible to know by applying formulas from classical mechanics the vertical displacement of the surface to which the element is attached. Among these measurement systems we find, for example, the extensometer bands embedded in the concrete or the fiber optic bands that allow their attached installation. In any case, these are indirect measurement systems that require post-processing of the recorded data. These systems usually must be embedded in the structure to measure, being this a complication when using them on existing structures.

Vision-based Sensing technologies possess the potential to address the issues in existing techniques. The existing vision-based methods differ by non-target approaches, feature detection, and coordinate transforms. The non-target approaches utilize noticeable features from a structure, which are tracked to measure displacement. Once a feature is detected, the position of the feature is transformed to the physical domain by using a coordinate transform. Several different transformation methods have been employed, such as simple scaling the affine transform, extrinsic parameters acquisition, and the homography transform.

Previous studies have shown the immense potential of computer vision for displacement sensing and other SHM applications. Several practical issues in computer vision-based displacement sensing have been identified in the literature, including the use of target markers, the selection of camera locations, and light-induced error.

The non-target approaches are convenient in that they do not need an installation of target markers. Despite the convenience, target-based measurement becomes useful when combined with the homography transform, which can greatly increase field applicability by allowing cameras to be arbitrarily placed. Regarding light-induced error, few studies have examined feature detection in a harsh field-testing environment, particularly those with adverse light conditions. Sunlight causes an image blur of target markers and thereby leads to significant error in finding features in the captured images.

As an example of the state of the art the following reference documents may be mentioned <CIT>, <CIT> and <CIT>.

Document <CIT> defines a system for measuring a structural displacement of an object using one or more digital video cameras adapted to generate digital image information corresponding to two or more features of the object in a sequence of frames having a frame rate, one of the features being a substantially stationary feature and at least one of the features being a nonstationary feature. The system comprises an input to receive the digital image information corresponding to the two or more features, a converter, coupled to the input, configured to convert each frame of the digital image information corresponding to the two or more features into a template; and a comparator, coupled to the converter, to compare the templates in sequence and to subtract displacement of the substantially stationary feature to thereby measure spatial displacement of the at least one nonstationary feature over time.

This system uses displacement monitoring by processing video images with so-called vision-based displacement sensors, using one or more cameras.

In the reference document <CIT> it is determinate a system that captures with a single video camera a grating maker that is attached or transferred to an object surface and using the sampling Moiré method quantitatively finds by image processing the out-of-plane displacement amount from a minute charge in the phase of the grating pitch in an image due to out-of-plane displacement.

Document <CIT> describes a measurement apparatus for easily measuring motion of a measurement object, such as a bridge girder, and a bridge inspection method. This measurement apparatus includes an imaging section and a processing section which processes image data output by the imaging section. The imaging section images a plurality of grid patterns arranged on the measurement object at predetermined time intervals, and output image data. The processing section calculates displacement of the grid patterns between two points of time with respect to each of image data in a plurality of image processing areas including the grid patterns. The displacement of the plurality of grid patterns <NUM> are calculated at a time.

The first document is a system without target and the second and third systems are based on vision through targets. In all the cases the systems present problems due to light-induced errors and don't allow continuous monitoring.

Document <CIT> discloses a fixing element for fixing a sensor to a structure to be monitored, in particular a rail, a sensor unit having such a fixing element, a sensor device for fixing the sensor unit to the structure to be monitored, and a method for disposing the sensor unit. The fixing element has at least one light source, in particular a light-emitting diode.

<CIT> discloses a single digital (video) camera that captures a grating marker that is attached or transferred to an object surface, and using the sampling moire method quantitatively finds by image processing the out-of-plane displacement amount from a minute change in the phase of the grating pitch in an image due to out-of-plane displacement. Also, the sampling moire method is used to quantitatively find the out-of-plane/in-plane displacement amount from a minute change in the phase of the same grating due to in-plane/out-of-plane displacement, and finds the three-dimensional displacement amount by excluding the effect of the apparent in-plane displacement.

<CIT> discloses a process for non-destructive testing, the process comprising: a) applying a photo-curable dye to a surface of an article; b) selectively curing an array of dots of the photo-curable dye on the surface; c) removing the photo-curable dye that has not been selectively cured; d) mechanically testing the article; and e) direct strain imaging the article during the mechanical testing based on the array of dots.

<CIT> discloses a method that includes generating image data of an interior of a fuel tank disposed within a wing of an aircraft, and determining, by a processing device, an amount of wing bending of the wing of the aircraft based on the generated image data of the interior of the fuel tank. The disclosed system generates, using one or more image capturing devices, reference image data of the interior of the fuel tank: and determines, based on the active image data and the reference image data, a displacement of one or more physical features of the interior of the fuel tank and determines the amount of wing bending based on the determined displacement of the one or more physical features.

Summarizing, it is necessary to find a system that meets certain important attributes like a very precise displacement measurement, a real-time and continuous monitorization, that it has an autonomous operation and a very competitive cost compared to existing alternative partial solutions. It also needs to be easy to install, both in existing infrastructures as well as in new infrastructures.

It is also relevant the recorded data, how it is managed and presented, so it is provided in a useful format for the decision maker with a minimum requirement of training effort.

The system for measuring displacement in structures is defined in claim <NUM> and is based on vision through targets presented here, comprises one or more measuring devices wherein each measuring device is attachable to a fixed area (i.e. stable surface) of the structure and is associated with a target attachable to an area of the structure to be analyzed.

The target comprises an outward facing face with two or more through holes and an interior illumination device facing said holes.

In addition, each measuring device comprises a high resolution camera with a long focal length lens capable of continuously capturing images of a target, wherein the resolution value and focal length are determined as a function of the distance between the camera lens and the target.

Also, the measuring device presents a processing unit for processing said target images continuously and autonomously, capable of calculating the vertical and/or horizontal and/or angular variation of the area to be analyzed by detecting the target holes, calculating their centroid and comparing the position of said centroid in different image processing.

The measuring device comprises as well a wireless communications module of the processing unit with a web platform for storage, processing and presentation of the results comprising data analysis and control software, and means of autonomous power supply.

With this displacement measuring system for civil structures a significant improvement over the state of the art is obtained.

The improvement is primarily due to the fact that it is a vision-based sensing solution developed with two main objectives: first, to achieve highly precise measurements of the displacement of an infrastructure, i.e. a bridge deck, in the submillimeter range and, second, to operate as an unattended technology, that is, that data registered by the sensor are a direct measurement of the target parameter to be monitored and that data recording is autonomously done by the system in real time.

The system is a target-based measurement system with the particularity that the target is internally illuminated to eliminate light-induced errors, and is able to work day and night, twenty four hours a day, seven days a week, in harsh environments with adverse light conditions. The system has been conceived as an unattended technology, that is, it performs a direct measurement of the parameter to be evaluated (vertical/horizontal/angular movement) and performs it autonomously. Measurement data are continuously reported and in real time to an on-line platform. Direct measurement facilitates the interpretation of the data by any user, without the need for technical support performing complex conversions or interpretations of the recorded data.

In addition, the system incorporates programmable alert thresholds that generate alarms when the registered value exceeds the threshold set for the structure under supervision, which further reinforces the autonomy of the equipment as a structure monitoring system.

Said drawings form an integral part of the description and illustrate preferred embodiments of the invention, which should not be interpreted as restricting the scope of the invention, but just as an example of how the invention can be embodied.

Throughout the drawings, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the disclosed invention will now be described in detail with reference to the Figs. , it is done so in connection with the illustrative embodiments.

In view of the figures provided, it can be observed how in a preferred embodiment of the invention, the system for measuring displacement in structures based on vision through targets proposed here comprises one or more measuring devices (<NUM>) wherein each measuring device (<NUM>) is attached to a fixed area and is associated with a target (<NUM>) attached to an area of the structure to be analyzed. The fixed area on which the measuring device is attached can be a stable surface that is part of the structure under analysis or it can be a stable surface outside the structure (a nearby ground surface or a surface of a different structure).

As show in <FIG>, <FIG> and <FIG>, the target (<NUM>) comprises an outward facing face with two or more through holes (<NUM>) and an interior illumination device (<NUM>) facing said holes (<NUM>).

As depicted in <FIG>, each measuring device (<NUM>) comprises a high resolution camera (<NUM>) with a long focal length lens (<NUM>) capable of continuously capturing images of a target (<NUM>), where the resolution value and focal length are determined as a function of the distance between the camera lens (<NUM>) and the target (<NUM>).

The measuring device (<NUM>) presents a processing unit (<NUM>) for processing said target (<NUM>) images continuously and autonomously, capable of calculating the vertical (y) and/or horizontal (x) and/or angular (α) variation of the area to be analyzed by detecting the target holes (<NUM>), calculating their centroid and comparing the position of said centroid in different image processing.

The measuring device (<NUM>) further comprises a wireless communications module (<NUM>) of the processing unit (<NUM>) with a web platform for storage, processing and presentation of the results, comprising data analysis and control software, and means of autonomous power supply.

In the present invention, an image of an internally illuminated target (<NUM>) as shown in <FIG> is captured and processed by the measuring device (<NUM>) to calculate the displacement of a civil structure.

The image obtained from a high megapixel camera (<NUM>) as shown in <FIG> is processed by the processing unit (<NUM>).

The combination of a high megapixel camera (<NUM>) and a long focal length lens (<NUM>) (for example, a <NUM>-megapixel, <NUM> x <NUM> pixels, camera and a <NUM> lens) results in an actual pixel width of less than one millimeter for distances between the measuring device (<NUM>) and the target (<NUM>) up to <NUM> meters. The number of pixels of the camera (<NUM>) and the focal length of the lens (<NUM>), as indicated, can be adapted to the required distance between the measuring device (<NUM>) and the target (<NUM>). The actual pixel width Px,y (mm) is calculated dividing the distance in millimeters between the centers of the circles of the target (<NUM>) by the number of pixels in the image between the centers of the circles.

The internal illumination device (<NUM>) improves the contrast to eliminate light-induced errors. In one embodiment the lighting device comprises means of wireless connection to the processing unit (<NUM>) so that the illumination device (<NUM>) can be controlled by this processing unit (<NUM>) to be adapted to the external light conditions. This solve limitation of previous systems and make this system able to work day and night, twenty four hours a day, seven days a week, in harsh environments with adverse light conditions.

In the exemplary embodiment shown in <FIG>, the illumination device (<NUM>) comprises a low power led, in particular a microled (<NUM>). In other embodiments different types of Illumination devices (<NUM>) may be used like an SMD led module or a COB led module or devices comprising fiber optics and light diffusers or a combination of them. <FIG> shows a combination of SMD / COB modules and stripes (<NUM>).

In the exemplary embodiment depicted in <FIG>, the measuring device (<NUM>) is installed in a fixed area of the structure, in this case the bridge pier (<NUM>), and the target (<NUM>) is installed in the bridge deck (<NUM>) to measure the mid-span (Y) as shown in <FIG>. This vision-based displacement measurement system can be installed in any in-service structure to measure displacement as shown in <FIG>.

Due to the low power requirements, in this embodiment the autonomous power supply means comprises a battery arranged inside the device and a solar panel connected to the inner battery. In others embodiments the system can be powered by wind power generators generator connected to the interior battery or any other autonomous power system.

This vision-based displacement measurement system can measure vertical (y), horizontal (x) and angular (α) displacement of any infrastructure as shown in <FIG>, <FIG>.

In a preferred embodiment the target (<NUM>) has two target holes (<NUM>) configured as circular orifices but in other embodiments the target (<NUM>) can have a different number of target holes (<NUM>) and the orifices can have a different shapes.

<FIG> shows a schematic block diagram of the steps performed by the processing unit (<NUM>). The processing unit (<NUM>) continuously performs the image acquisition process (<NUM>). Then the image processing (<NUM>) is in charge of detecting the two circular target holes (<NUM>) of the target (<NUM>) and calculating the centroid coordinates of the circles.

This step of image processing (<NUM>) involves several steps as shown in <FIG>.

The first step is the region of interest (ROI) selection (<NUM>), this ROI (<NUM>) is the area of the image that will be further processed as shown in <FIG>.

Next, binarization (<NUM>) of the cropped image is performed using a thresholding method as show in <FIG>. This binarized image (<NUM>) helps in clearly separating the circles from the background as shown in <FIG>.

A "binarized image" (<NUM>) is the result of converting the original image to black and white. Since for the purposes of the present invention only the two "big bright spots" corresponding to the target holes (<NUM>) are of interest, the best way is to "separate" them from the background (which, although black, is not a pure black in the image produced by any camera).

<FIG> shows some white "minor spots" (<NUM>). These are "glows" which correspond to flashes produced by any lens when there are very bright spots on a dark background. Glare can also be produced by reflections of powerful lights or the sun on the target.

The next step is a noise reduction process (<NUM>), as show in <FIG> This process of noise reduction through morphological transformations such as erosion and dilation generates an image like the one shown in <FIG>.

Morphological transformations basically clean up an image that is already black and white. Morphological transformations are some operations based on the image shape. Usually, as in this case, they are performed on binary images. It needs two inputs, one is the original image, second one is called structuring element or kernel which decides the nature of operation. Two basic morphological operators are Erosion and Dilation:.

The target holes (<NUM>) in this embodiment are circular and the image should ideally look like this. But due to the flashes and the fact that the "ideal" does not exist, the image does not come out perfectly circular (in the attached figures this deformation has been exaggerated for a better understanding of the process).

The final objective is that only the two "big spots" corresponding to the target holes (<NUM>) are left so that the necessary precision can be achieved when calculating the centroid.

The final step of the image processing (<NUM>) is the centroid calculation (<NUM>) as shown in <FIG>. The centroid of a shape is the arithmetic mean (i.e. the average) of all the points in a shape.

Suppose a shape consists of n distinct points X<NUM>. Xn, then the centroid is given by <MAT>.

In the context of image processing and computer vision, each shape is made of pixels, and the centroid can be calculated using Image Moments. Image Moment, Mij, is a particular weighted average of image pixel intensities I(x,y) and are calculated by: <MAT>.

The centroid (center of gravity) is given by the formula: <MAT> <MAT>.

Cx is the x coordinate and Cy is the y coordinate of the centroid and M denotes the Moment.

Once the centroids are computed, the pixel displacement is calculated subtracting the values of the previously acquired image.

Once we have obtained the image processing (<NUM>), the next step performed by the processing unit (<NUM>), as shown in <FIG>, is the displacement calculation process (<NUM>) that determines the corresponding distance to the pixel movement through a simple scaling coordinate transformation. Simple scaling multiplies the scaling factor (actual pixel width Px,y in mm) to the measured image pixel coordinate displacement. Due to the small actual pixel width, submillimeter precision can be obtained with simple scaling and low computational load, and there is no need to apply more complex subpixel analysis algorithms, as the one disclosed in <NPL>). This low computational load permits the use of an unattended low power embedded processor in the processing unit (<NUM>).

The threshold alert process (<NUM>) generates alerts with predefined thresholds and the data is sent to the web platform through the wireless communication module (<NUM>). The displacement data is then presented in a graphic window as shown in <FIG>.

The preceding processes are provided for purpose of illustration rather than limitation.

Claim 1:
A system for measuring displacement in civil structures based on vision through targets, comprising
• A target (<NUM>)
• One or more measuring devices (<NUM>) wherein each measuring device (<NUM>) is configured to be attached to a fixed area and is associated with the target (<NUM>) attachable to an area of the structure to be analyzed, wherein the target (<NUM>) comprises an outward facing face with two or more through holes (<NUM>) and an interior illumination device (<NUM>) facing said through holes (<NUM>), wherein each measuring device (<NUM>) comprises
• a high resolution camera (<NUM>) with a long focal length lens (<NUM>) capable of continuously capturing images of the target (<NUM>), wherein the resolution value and focal length are determined as a function of the distance between the camera lens (<NUM>) and the target (<NUM>);
• a processing unit (<NUM>) for processing said target (<NUM>) images continuously and autonomously, capable of calculating the vertical (y) and/or horizontal (x) and/or angular (α) variation of the area to be analyzed by detecting the target holes (<NUM>), calculating their centroid and comparing the position of said centroid in different image processing;
• a wireless communications module (<NUM>) of the processing unit (<NUM>) with a web platform for storage, processing and presentation of the results comprising data analysis and control software, and;
• means of autonomous power supply.