Patent ID: 12235191

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, a preferable embodiment of a flight imaging system and a method according to an embodiment of the present invention will be described in accordance with appended drawings.

[Outline of Present Invention]

FIG.1is a diagram showing a schematic configuration of a flight imaging system according to the present invention.

The flight imaging system1shown inFIG.1includes a drone100which is an unmanned flying object, a remote controller200, a user terminal300, and a server400. The user terminal300may be a personal computer (PC) (laptop PC, tablet PC), a smartphone, or the like that can communicate with the server400.

FIG.2is a diagram showing a state of the drone during flight, andFIG.3is a diagram showing a state of the drone at the time of imaging the structure with a visible camera or the like mounted on the drone. As shown inFIGS.2and3, a visible camera120and a millimeter-wave radar (first millimeter-wave radar)130are mounted on the drone100.

The flight imaging system1sequentially moves to each imaging point with the drone100, images the structure with the visible camera120at each imaging point, and acquires a visible image of the surface layer of the structure. In addition, in order to acquire a millimeter-wave image corresponding to the visible image, the flight imaging system1transmits a millimeter wave toward the structure at each imaging point from the millimeter-wave radar130, and receives the reflected wave of the millimeter wave from the structure.

The reflected wave of the millimeter wave from the structure is reflected by the surface of the structure, but a part of the reflected wave reaches a certain distance (for example, several centimeters) from the surface of the structure, so that the millimeter-wave reception data received from millimeter-wave area includes three-dimensional information up to the certain distance from the surface of the structure. In this example, acquiring the millimeter-wave reception data is also referred to as imaging a millimeter-wave image by the millimeter-wave radar.

The structures10shown inFIGS.2and3are social infrastructure structures such as bridges and tunnels, and buildings.

For example, the drone100autonomously flies along a predetermined flight route, or semi-autonomously flies in response to an instruction from the remote controller200. It is preferable that the flight route is designed so as to connect each imaging point of the structure with the shortest route.

The visible image and the millimeter-wave image captured at each imaging point are transmitted from the drone100to the server400via the user terminal300after the imaging at all the imaging points is completed. The data of the visible image and the millimeter-wave image is transferred from the drone100to the user terminal300by a memory card150in which the visible image and the millimeter-wave image are recorded, or by connecting the drone100and the user terminal300via a universal serial bus (USB) cable or short-range wireless connection.

First Embodiment

FIGS.2and3are diagrams showing an outline of a first embodiment of the flight imaging system according to the present invention,FIG.2shows a state of the drone at the time of flight, andFIG.3is a state of the drone at the time of imaging.

[At the Time of Flight]

The millimeter-wave radar130shown inFIG.2is configured such that the entire or a part (for example, a transmission/reception module) of the millimeter-wave radar130can be rotated (tilted) with respect to the drone100.

During the flight of the drone100, the millimeter-wave radar130is directed downward. That is, in the millimeter-wave radar130, the rotation angle of the millimeter-wave radar130is controlled such that the transmission/reception direction of the millimeter wave from the millimeter-wave radar130is the downward direction of the drone100.

In this example, the visible camera120and the millimeter-wave radar130are integrated and the rotation angle is controlled at the same time. However, the rotation angle may be individually controlled. In addition, “at the time of flight” or “during flight” of the drone100includes flight in a case where the drone100is moving in the air and stationary flight (hovering) in which the drone100is stationary in the air.

As the millimeter-wave radar130, for example, various methods such as a beam switching method, a phased array method, and a digital beam forming method can be applied.

The beam switching method is a method of forming a plurality of fixed beams having a narrow beam width in which the oriented directions are slightly different from each other and electrically switching the fixed beams by time division.

The phased array method uses a phased array antenna in which a plurality of element antennas are arranged at regular intervals, and controls the phase of a signal by a phase controller connected to the plurality of element antennas to form and emit a beam in a high frequency band.

Similar to the phased array method, the digital beam forming method uses a phased array antenna, detects signals received by a plurality of element antennas in the reception, converts the signal into a baseband signal, converts the signal into a digital signal, and then forms a beam by a signal processing operation. In this method, since the waveform information of the received signal is stored as numerical data, it is possible to form a beam or the like having various characteristics and shapes by operation.

In the millimeter-wave radar130, in a case where the transmission/reception direction of the millimeter wave from the millimeter-wave radar130is controlled to be downward direction of the drone100, a distance in the millimeter-wave area (area where the millimeter wave is transmitted and received)134on a reference surface20of the ground or the water surface can be measured. The shortest distance among the distances measured in the millimeter-wave area134is the height (altitude) from the reference surface20of the drone100.

In the first embodiment of the flight imaging system according to the present invention, the millimeter-wave radar130during flight of the drone100functions as an altitude meter and measures the altitude of the drone100. The altitude information indicating the measured altitude is used as information in a case where the drone100is made to fly (autonomous flight or semi-autonomous flight).

In general, a drone is equipped with a global positioning system (GPS) module, and the GPS module receives GPS signals transmitted from a plurality of GPS satellites, performs positioning calculation processing based on the received plurality of GPS signals, and detects position information including a latitude, a longitude, and an altitude of the GPS module.

However, the GPS module has a drawback that it cannot acquire the position information (altitude information) in a place where the GPS signal cannot be received, such as in a tunnel or under a bridge.

On the other hand, according to the millimeter-wave radar130mounted on the drone100, the altitude of the drone100can be measured under any environment, and the measured altitude can be used for autonomous flight of the drone100or the like.

[At the Time of Imaging]

The visible camera120is configured such that the entire or a part of the visible camera120(for example, an imaging unit including a lens and an image sensor) can be rotated (tilted) with respect to the drone100.

As shown inFIG.3, at the time of imaging with the drone100, the drone100transitions to hovering stationary in the air. During hovering, the position and the posture of the drone100are controlled to be maintained based on the sensor output of the gyro sensor, the acceleration sensor, or the like mounted on the drone100. While the drone100is stationary, the altitude of the drone100is controlled to be a desired altitude based on the altitude information measured by the millimeter-wave radar130.

In addition, while the drone100is stationary, the rotation angle of the visible camera120is controlled such that the imaging direction of the visible camera120is the direction of the structure10. Similarly, the rotation angle of the millimeter-wave radar130is controlled such that the transmission/reception direction of the millimeter wave from the millimeter-wave radar130is the direction of the structure10.

In this case, it is preferable that the visible camera120and the millimeter-wave radar130are rotationally controlled so as to face the structure10.

After that, in a case where the drone100receives the imaging instruction from the remote controller200by a user operation or in a state in which the drone100is stationary and ready to image, the drone100automatically causes the visible camera120to capture a visible image of the surface layer of the structure10, or transmits the millimeter wave toward the structure10from the millimeter-wave radar130, receives the reflected wave of the millimeter wave from the structure10, and acquires the millimeter-wave reception data.

<Relationship between Visible Image and Millimeter-Wave Image>

FIG.4is a diagram showing a relationship between a visible image and a millimeter-wave image.

InFIG.4,124indicates a first imaging range (visible image area)124corresponding to a visible image, and134indicates a second imaging range (millimeter-wave area) corresponding to a millimeter-wave image.

As shown inFIG.4, in an xy coordinate system (the coordinate system of the image sensor of the visible camera120) that specifies the visible image area124, the coordinates of the four points A, B, C, and D of the millimeter-wave area134(A(x1(d), y1(d)), B(x2(d), y2(d)), C(x3(d), y3(d)), D(x4(d), y4(d))) are obtained in advance. For example, at a factory, it is measured in advance which positions in the visible image correspond to the four points of the millimeter-wave image at a certain distance d. An aluminum metal plate that strongly reflects the millimeter wave is disposed in the imaging area of the visible image, the reflection intensity of the millimeter wave is observed while shifting the aluminum metal plate, and the millimeter-wave area134is fixed based on the position in the visible image area124of the aluminum metal plate, at which the reflected wave of the millimeter wave cannot be detected first.

Then, the distance d is changed, and the coordinates of the four points A, B, C, and D of the millimeter-wave area134are obtained for each of the plurality of distances (d1, d2, d3, . . . ).

FIG.5is a table showing a relationship between a plurality of distances and coordinates of four points of a millimeter-wave image at each distance.

During operation, the distance d between the drone100and the structure10is measured, and a table shown inFIG.5is used to estimate which pixel positions in the visible image are the four points A, B, C, and D of the millimeter-wave area134. In a case where the measured distance d does not exist in the table, the pixel positions of the four points A, B, C, and D corresponding to the distances before and after the measured distance d are linearly interpolated.

In a case where the structure10is a concrete structure and there are internal defects12such as fissuring generated inside the structure10or floating of concrete, the millimeter-wave radar130can also receive reflected waves reflected by the internal defects12. In addition, in a case where the outer wall tile is attached to the structure10, the millimeter-wave radar130can receive the reflected wave reflected by defects such as floating of the outer wall tile.

FIG.6is a diagram schematically showing a relationship between a reflection intensity of a millimeter wave and a distance at a certain pixel position of the millimeter-wave image.

InFIG.6, P1indicates a reflected wave from the surface of the structure10, and P2indicates a reflected wave from the internal defect12of the structure10.

The distance between the reflected waves P1and P2corresponds to a depth of the internal defect12.

FIG.7is a diagram showing a relationship between a visible image and a millimeter-wave image showing an internal defect.

(A) ofFIG.7is a front view in a case where the visible image and the millimeter-wave image are superimposed, and (B) ofFIG.7is a perspective view of the structure10which is an image target of the visible image and the millimeter-wave image viewed from above.

In (B) ofFIG.7, in a case where a millimeter wave T is emitted from the millimeter-wave radar130to a certain position of the structure10, a reflected wave P1from the surface of the structure10and a reflected wave P2from the internal defect12which is present an inside detection area10A of the structure10are can be detected. The reflection intensities and distances of the reflected waves P1and P2are as shown inFIG.5.

Therefore, in a case where the millimeter-wave image is generated based on the reflected wave P2from the internal defect12, the millimeter-wave image becomes an image showing the internal defect12. In addition, since the positional relationship between the visible image and the millimeter-wave image can be specified based on the table shown inFIG.6, the millimeter-wave image can be combined with the visible image.

In (A) ofFIG.7,136represents a certain pixel of the millimeter-wave image. In addition, it is preferable that the millimeter-wave image has a color different from that of the visible image such as red, and is represented by a shade according to the depth and the reflection intensity so that the millimeter-wave image can be distinguished from the visible image of the surface layer of the structure10.

Returning toFIG.3, the millimeter-wave radar130can measure the distance d between the drone100and the structure10at the time of imaging.

The distance d measured by the millimeter-wave radar130can be used in a case of reading out the coordinates of the four points A, B, C, and D of the millimeter-wave area134from the table shown inFIG.5. In addition, the distance d can be used to control the position of the imaging point of the drone100in a case where the imaging distance from the drone100to the structure10is maintained at a desired imaging distance, and further, the imaging lens of the visible camera120can be used for focusing.

In this way, in a case where the drone100captures the visible image and the millimeter-wave image at a certain imaging point, the drone100flies to the next imaging point and captures the visible image and the millimeter-wave image. This is repeated until the imaging of the visible image and the millimeter-wave image at all the imaging points of the structure10is completed.

Second Embodiment

FIG.8is a diagram showing an outline of a second embodiment of the flight imaging system according to the present invention. Components shown inFIG.8common to the flight imaging system according to the first embodiment shown inFIG.2or the like will be denoted by the same reference numerals as those shown inFIG.2or the like, and the detailed description thereof will be omitted.

The flight imaging system of the second embodiment shown inFIG.8is different from the flight imaging system of the first embodiment shown inFIG.2or the like in that a millimeter-wave radar131is added.

That is, the millimeter-wave radar (first millimeter-wave radar)130is directed toward the structure10and is used for capturing a millimeter-wave image of the structure10. The millimeter-wave radar (second millimeter-wave radar)131is directed downward of the drone100, and functions as an altitude meter that measures the altitude of the drone100.

Other configurations and functions of the flight imaging system according to the second embodiment are common to the flight imaging system according to the first embodiment shown inFIGS.2to7.

Third Embodiment

FIGS.9and10are diagrams showing an outline of a third embodiment of the flight imaging system according to the present invention,FIG.9shows a state of the drone at the time of flight, andFIG.10is a state of the drone at the time of imaging. Components shown inFIGS.9and10common to the flight imaging system according to the first embodiment shown inFIG.2or the like will be denoted by the same reference numerals as those shown inFIG.2or the like, and the detailed description thereof will be omitted.

The flight imaging system of the third embodiment shown inFIGS.9and10is different from the flight imaging system of the first embodiment shown inFIG.2or the like in that a laser distance meter140is added.

[At the Time of Flight]

The millimeter-wave radar130shown inFIG.9measures the altitude of the drone100, and during flight of the drone100, the millimeter-wave radar130is directed downward as in the first embodiment.

The millimeter-wave radar130has lower accuracy in measuring the distance than the laser distance meter140, but can measure the distance (altitude) to such an extent that the flight of the drone100is not hindered. In particular, it is advantageous as compared with the laser distance meter140in that the altitude can be measured even in a case where the reference surface20is a water surface.

In addition, the millimeter-wave radar130is also used as a reference surface detecting unit that detects whether or not the reference surface20that is a reference for measuring the altitude of the drone100is a reference surface that is difficult to detect with the laser distance meter140. Here, the surface that is difficult to detect with the laser distance meter140is the water surface.

As will be described later with reference toFIG.10, an altitude h of the drone100at the time of stationary flight of the drone100is measured by the laser distance meter140. This is because the laser distance meter140can measure the distance (altitude) more accurately than the millimeter-wave radar130.

In a case where the millimeter-wave radar130detects that the reference surface in the vertical downward direction of the drone100is a surface (water surface) that is difficult to detect with the laser distance meter140, a new reference surface (ground) that is newly set and can be detected by the laser distance meter140is used as the reference surface for altitude measurement of the drone100.

The millimeter-wave radar130measures the altitude of the drone100during the moving flight of the drone100, and in a case where the drone100is stationary in the air, discriminates between a ground area and a water surface area in the millimeter-wave area.

In a case where the drone100is stationary, the intensity of the reflected wave from the water surface area changes with time due to the influence of the wave on the water surface, whereas the intensity of the reflected wave from the ground area hardly changes with time.

Accordingly, by analyzing the reflected wave obtained from the millimeter-wave radar130before the drone100is stationary in the air and switches the direction of the millimeter-wave radar130for imaging the structure10, the ground and the water surface area in the millimeter-wave area can be discriminated.

The laser distance meter140measures the distance to the measurement target by measuring the time during which the emitted laser light is reflected by the measurement target and received the light, and it is preferable to use light detection and ranging (LiDAR). LiDAR is used in an automatic operation system, and can measure a position and a shape of an object in space in addition to a distance to the object.

The laser distance meter140detects an obstacle on the flight route at the time of the moving flight of the drone100, or measures the distance d from the drone100to the structure10. The information of the object measured by the laser distance meter140is used in a case where the drone100is made to fly autonomously or semi-autonomously. For example, the information can be used for autonomous flight of the drone100such that the distance d between the drone100and the structure10is maintained at a desired distance.

[At the Time of Imaging]

As shown inFIG.10, at the time of imaging with the drone100, the drone100transitions to hovering stationary in the air.

In a case where the drone100is stationary, as described above, before switching the direction of the millimeter-wave radar130for imaging the structure10, the reflected wave obtained from the millimeter-wave radar130is analyzed, and the ground area and the water surface area in the millimeter-wave area in the downward direction of the drone100are discriminated.

In a case where the area in the vertical downward direction of the drone100is discriminated to be the ground area, the ground corresponding to the ground area is set as the reference surface20A, and the distance to the reference surface20A is measured by the laser distance meter140. Then, the drone100acquires the shortest distance among the distances measured by the laser distance meter140as the altitude h (h1) of the drone100.

In a case where an area in the vertical downward direction of the drone100is discriminated to be the water surface area, a new reference surface20A (a reference surface of the ground corresponding to the ground area other than the vertical downward direction) that can be detected by the laser distance meter140is set instead of the reference surface20B corresponding to the water surface area, and the distance L between the drone100and the new reference surface20A is measured by the laser distance meter140.

Then, based on the angle θ formed between the measured distance L and the vertical downward direction of the laser light at the time of measuring the distance L, the altitude h (h2) of the drone100is calculated by the following equation.
h2=L×cos θ  [Equation1]

The altitude information of the drone100measured by the laser distance meter140can be used for controlling the altitude of the drone100in a stationary state in a case where an imaging point (at least an altitude) of the drone100is set.

After that, the rotation angle of the visible camera120is controlled such that the imaging direction of the visible camera120becomes the direction of the structure10, as in the first embodiment shown inFIG.2or the like. Similarly, the rotation angle of the millimeter-wave radar130is controlled such that the transmission/reception direction of the millimeter wave from the millimeter-wave radar130is the direction of the structure10.

The drone100automatically captures a visible image of the surface layer of the structure10by the visible camera120in a case of receiving an imaging instruction from the remote controller200by a user operation or in a state in which the drone100is stationary and capable of imaging. In addition, a millimeter-wave image is imaged by the millimeter-wave radar130.

In addition, at the time of capturing the visible image and the millimeter-wave image, the distance d between the drone100(the visible camera120and the millimeter-wave radar130) and the structure10is measured by the laser distance meter140. The distance d measured by the laser distance meter140can be used in a case of reading out the coordinates of the four points A, B, C, and D of the millimeter-wave area134from the table shown inFIG.5. In addition, the distance d can be used to control the position of the imaging point of the drone100in a case where the imaging distance from the drone100to the structure10is maintained at a desired imaging distance, and further, the imaging lens of the visible camera120can be used for focusing.

[Hardware Configuration of Flight Imaging System]

FIG.11is a block diagram showing an embodiment of a hardware configuration of the flight imaging system according to the present invention.

The drone100shown inFIG.11corresponds to the third embodiment shown inFIGS.9and10, and includes a processor102, a gyro sensor104, a GPS module106, an acceleration sensor108, a memory110, and a communication interface (communication I/F)112, an input-output I/F114, a plurality of propellers119, a plurality of motors118for driving each of the plurality of propellers119, and a propeller control unit116for controlling the operation of the plurality of motors118, a visible camera120, a millimeter-wave radar130, a laser distance meter140, and the like.

The memory110includes a random access memory (RAM), a read only memory (ROM), a flash ROM, or the like, and an operating system, a program for executing a flight imaging method, flight plan information such as a flight route, the table shown inFIG.5, or the like are stored in the ROM or the flash ROM. The RAM temporarily stores a program read out from the ROM or the flash ROM, and functions as a work area of the processor102. In addition, the flash ROM can function as an internal memory that stores a visible image captured by the visible camera120and a millimeter-wave image captured by the millimeter-wave radar130.

The processor102is a part that reads various programs from the memory110and controls each part in an integrated manner. The processor102performs flight control of the drone100, control of capturing a visible image by the visible camera120, control of capturing a millimeter-wave image and the measurement of the altitude with the millimeter-wave radar130, and the measurement of the altitude with the laser distance meter140.

Further, each output signal of the gyro sensor104, the GPS module106, and the acceleration sensor108is input to the processor102. The processor102can know the posture, the angular velocity, and the angular acceleration of the drone100based on the output signal of the gyro sensor104. In addition, the processor102can know the position (latitude, longitude, altitude) of the drone100based on the GPS signal of the GPS module106. Furthermore, the processor102can know the acceleration of the drone100based on the output signal of the acceleration sensor108.

The processor102executes various operations of autonomous flight such as takeoff, moving flight, hovering, revolution, and landing of the drone100according to a flight route set in advance by the flight plan, or semi-autonomous flight in which the drone100receives an instruction about a part of flight from the remote controller200and flies in response to the instruction by controlling each of the plurality of motors118via the propeller control unit116based on the output signals of the gyro sensor104, the GPS module106, and the acceleration sensor108.

Obviously, the drone100can arbitrarily fly based on a flight instruction from the remote controller200, in addition to the autonomous flight or the semi-autonomous flight. In addition, at the time of hovering, it is possible to automatically control to hold the position and the posture of the drone100based on the detection output of the gyro sensor104or the acceleration sensor108. Furthermore, the gyro sensor104can detect the angular velocity along the movement of the drone100and can detect the angle by the integral calculation of the angular velocity, and the acceleration sensor108can detect the tilt (direction of gravitational force) of the drone100, the parallel translation, and the speed by the integral.

The processor102performs processing of capturing a visible image of the surface layer of the structure by the visible camera120and processing of transmitting the millimeter wave toward the structure from the millimeter-wave radar130and receiving the reflected wave of the millimeter wave from the structure to capture a millimeter-wave image. The processor102, during the moving flight of the drone100, causes the millimeter-wave radar130to measure the altitude from the reference surface by and causes the laser distance meter140to measure the altitude from the reference surface at the time of stationary flight of the drone100. The altitude information indicating these altitudes is used for flight control of the drone100.

The drone100uses information of the altitude and the distance to the structure measured by the millimeter-wave radar130and the laser distance meter140to perform autonomous flight or semi-autonomous flight in an environment in which GPS radio waves do not reach.

The drone100can fly while keeping a constant distance (altitude) from the reference surface measured by the millimeter-wave radar130or the laser distance meter140and the distance to the structure10. The altitude at which the drone100should fly and the distance to the structure10can be acquired from the memory110as flight plan information such as a flight route, or can be received from the remote controller200by a user instruction.

In addition, since the specific control of the drone100at the time of flight and at the time of imaging by the processor102is the same as that of the third embodiment described with reference toFIGS.9and10, the description thereof will be omitted here.

The remote controller200is used to remotely control the drone100by a user operation, and includes a communication I/F210, a central processing unit (CPU)220, an operation unit230, a memory240, and a monitor250.

The CPU220integrally controls each unit by executing the firmware stored in the memory240. The processor102of the drone100and the CPU220of the remote controller200can exchange necessary information via the communication I/Fs112and210, and the remote controller200receives a live view image captured by the visible camera120via the communication I/Fs112and210and can display it on the monitor250.

The user can operate the operation unit230while viewing the live view image displayed on the monitor250to guide the drone100to fly to the first imaging point of the structure10. In a case where the flight route to the first imaging point is in an environment capable of receiving the GPS radio wave, by setting the flight route to the drone100in advance, the drone100can fly autonomously to the first imaging point while capturing the current position with the GPS module106. In this case, the user gives an instruction for autonomous flight to the first imaging point from the remote controller200to the drone100.

In a case where the drone100reaches the first imaging point, the user gives an instruction from the remote controller200to execute the flight imaging method according to the present invention to the drone100. As a result, the drone100acquires information such as an altitude measured by the millimeter-wave radar130and the laser distance meter140, a distance to the structure10, and the drone100moves each imaging point while maintaining a preset altitude and the distance to the structure10. Then, the drone100is stationary at each imaging point to capture a visible image by the visible camera120and capture a millimeter-wave image by the millimeter-wave radar130.

The capturing of the visible image by the visible camera120and the capturing of the millimeter-wave image by the millimeter-wave radar130at each imaging point may be automatically performed at a time when the hovering of the drone100at each imaging point is stable or may be performed by receiving the imaging instruction from the remote controller200by the user operation.

The drone100moves to the next imaging point after the imaging at a certain imaging point is completed. The distance between the imaging points can be preset based on the imaging area on the structure10corresponding to the visible image to be captured. For example, it is preferable that the distance between the imaging points is set to be shorter than the length of the long side of the imaging area in a case where the drone100is moved in the horizontal direction, and the distance is set to be shorter than the length of the short side of the imaging area in a case where the drone100is moved in the vertical direction. This is because, in a case of panorama composition of captured visible images, overlapping image areas between adjacent visible images are required.

Since the moving distance of the drone100can be estimated based on the detection output of the acceleration sensor108, the drone100can be stopped at a position where the moving distance of the drone100from a certain imaging point reaches a preset distance as the next imaging point. In addition, in a case where the speed pattern from the certain imaging point to the next imaging point is determined, the flight time can be controlled to move the drone100to the next imaging point.

After the imaging at the last imaging point is completed by the drone100, the user can operate the operation unit230while viewing the live view image displayed on the monitor250to return the drone100. In addition, in an environment in which the GPS radio wave can be received, the user can automatically return the drone100by instructing the return from the remote controller200.

FIG.12is a block diagram showing an embodiment of a server applied to the flight imaging system according to the present invention.

The server400shown inFIG.12processes a visible image and a millimeter-wave image uploaded from the user terminal300or collected from the user terminal300, and returns the processing result to the user terminal300. The server400includes a communication I/F410, a CPU420, a memory430, and database440.

The CPU220controls each part in an integrated manner by executing a program stored in the memory240, and performs processing of detecting damage appearing on the surface layer of the structure10based on a visible image as described later, processing of generating a millimeter-wave image inside the structure from millimeter-wave reception data including three-dimensional information up to a predetermined depth from the surface layer of the structure, and processing of combining the visible image and the millimeter-wave image.

The database440stores and manages the visible image and the millimeter-wave reception data acquired from the user terminal300via the communication I/F410, and the processing result for the visible image and the millimeter-wave reception data.

FIG.13is a functional block diagram showing functions of the server shown inFIG.12.

The server400shown inFIG.13mainly includes an input unit410A, a damage detection unit422, a millimeter-wave image generation processing unit424, a combination processing unit426, and an output unit410B.

The input unit410A acquires the visible image122and the millimeter-wave reception data132from the user terminal300. The input unit410A corresponds to the communication I/F410of the server400that receives the visible image122and the millimeter-wave reception data132transmitted from the user terminal300.

The damage detection unit422performs processing of detecting damage (for example, fissuring, free lime, exposed reinforcing bar, peeling of concrete) appearing on the surface layer of the structure10based on the visible image122, specifies a damage area and a damage type, and outputs the damage image.

The damage detection unit422may be performed by an image processing algorithm or may be performed by artificial intelligence (AI). For example, a trained model using a convolution neural network (CNN) can be used as AI.

The millimeter-wave image generation processing unit424generates a millimeter-wave image showing the inside of the structure based on the millimeter-wave reception data132indicating the reflected wave of the millimeter wave from the structure. As shown inFIG.6, the reflected wave of the millimeter wave includes a reflected wave from the surface of the structure and a reflected wave from an internal defect of the structure, and the former reflected wave has a higher reflection intensity than the latter reflected wave. It is preferable that the millimeter-wave image generation processing unit424extracts millimeter-wave reception data indicating a reflected wave other than the reflected wave from the surface of the structure, and generates a millimeter-wave image based on the extracted millimeter-wave reception data. In addition, it is preferable that the millimeter-wave image has a color different from that of the visible image such as red, and is represented by a shade according to the depth and the reflection intensity so that the millimeter-wave image can be distinguished from the visible image of the surface layer of the structure10.

The combination processing unit426combines the visible image, the damage image output from the damage detection unit422, and the millimeter-wave image output from the millimeter-wave image generation processing unit424. Since the positional relationship between the visible image and the millimeter-wave image can be specified based on the table shown inFIG.6, the millimeter-wave image can be combined with the visible image.

In this example, the visible image, the damage image, and the millimeter-wave image are combined, but the present invention is not limited to this. The visible image and the millimeter-wave image may be combined, or the damage image and the millimeter-wave image may be combined. In addition, the server400can also perform panorama composition of the visible image or the like.

The damage detection unit422, the millimeter-wave image generation processing unit424, and the combination processing unit426correspond to each function of the CPU420of the server400.

The combined image combined by the combination processing unit426is transmitted to the user terminal300via the output unit410B. The input unit410A corresponds to the communication I/F410of the server400.

As a result, the user can confirm the internal defects in addition to the damage of the surface layer of the structure.

The user terminal300may be configured to have the processing function of the server400of this example.

[Flight Imaging Method]

FIG.14is a flowchart showing an embodiment of a flight imaging method according to the present invention.

The processing of each step shown inFIG.14is performed by, for example, the processor102of the drone100shown inFIG.11.

InFIG.13, the drone100is moved (flied) to the first imaging point where imaging of the structure is performed (step S10). The flight of the drone100may be performed by autonomous flight of the drone100according to a flight plan, or may be performed by an instruction from the remote controller200operated by the user.

The processor102determines whether or not the drone100is moved to the first imaging point (step S12). In a case where determination is made that the drone100is moved to the first imaging point (in the case of “Yes”), the processor102causes the drone100to fly stationary at the imaging point, and causes the laser distance meter140to measure the distance (altitude) of the drone100from the reference surface (step S13).

Subsequently, the processor102causes the visible camera120to capture the visible image of the structure (step S14), and causes the millimeter-wave radar130to capture the millimeter-wave image of the structure (step S16). The capturing of the millimeter-wave image is performed by transmitting the millimeter wave to the structure, receiving the reflected wave from the millimeter-wave area, and acquiring the millimeter-wave reception data.

The processor102stores the captured visible image and the millimeter-wave image in the memory110in association with each other (step S18).

After the imaging of the structure at the imaging point is completed, the processor102moves the drone100to the next imaging point according to the flight plan (step S20).

In a case where determination is made that the drone100is moved to the next imaging point (in a case of “Yes”), the processor102transitions to step S13. Here, a stationary flight is performed at the imaging point, and the altitude of the drone100is measured by the laser distance meter140.

The processor102repeats the processing from step S13to step S22until the imaging at all the imaging points is completed.

[Others]

The processor of the unmanned flying object of the flight imaging system according to the present invention and the various processors of the server include a central processing unit (CPU) that is a general-purpose processor functioning as various processing units by executing programs, a programmable logic device (PLD) that is a processor of which circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), a dedicated electrical circuit that is a processor having circuit configuration designed exclusively to perform specific processing, such as an application specific integrated circuit (ASIC), and the like. One processing unit constituting the flight imaging system may be composed of one of the various processors or two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs or a combination of a CPU and an FPGA. In addition, a plurality of processing units may be composed of one processor. As an example of configuring the plurality of processing units with one processor, first, as represented by a computer such as a client or a server, a form of configuring one processor with a combination of one or more CPUs and software and causing the processor to function as the plurality of processing units is present. Secondly, as represented by system on chip (SoC) or the like, there is a form in which a processor that realizes all functions of a system including a plurality of processing units into one integrated circuit (IC) chip is used. Accordingly, various processing units are configured using one or more of the various processors as a hardware structure. Furthermore, the hardware structure of those various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

In addition, the present invention is not limited to the embodiment and can be subjected to various modifications without departing from a spirit of the present invention.

EXPLANATION OF REFERENCES

1: flight imaging system10: structure12: internal defect20,20A,20B: reference surface100: drone102: processor104: gyro sensor106: GPS module108: acceleration sensor110: memory112: communication I/F114: input-output I/F116: propeller control unit118: motor119: propeller120: visible camera122: visible image124: visible image area130,131: millimeter-wave radar132: millimeter-wave reception data134: millimeter-wave area140: laser distance meter150: memory card200: remote controller210: communication I/F220: CPU230: operation unit240: memory250: monitor300: user terminal400: server410: communication I/F410A: input unit410B: output unit420: CPU422: damage detection unit424: millimeter-wave image generation processing unit426: combination processing unit430: memory440: databaseS10to S22: step