Patent ID: 12242269

DESCRIPTION OF EMBODIMENTS

Hereinafter, the same or equivalent components, members, and processes illustrated in each drawing are denoted by the same reference signs, and the repeated description will be omitted as appropriate. Additionally, in each drawing, some of the members that are not important for the description are omitted.

FIG.1is a schematic diagram for describing an inspection support system2according to an embodiment. The inspection support system2supports an inspection of an object to be inspected that is performed by a worker (user4) using an unmanned flying device such as a drone8. In the present embodiment, a base station apparatus6of a mobile telephone network is assumed as the object to be inspected, but the object to be inspected is not limited to the base station apparatus6, and in other embodiments, for example, a power-related infrastructure facility such as a transmission line and an architectural structure such as a building, bridge, and dam may be an object to be inspected.

The inspection support system2includes a mobile terminal10of the user4and the drone8. The mobile terminal10and the drone8are configured to be communicable, and this communication may be achieved by a direct wireless communication means such as Bluetooth (registered trademark) or WiFi (registered trademark), or may be achieved by a mobile phone network or the Internet. The mobile terminal10is a terminal that controls the drone8.

The mobile terminal10is a mobile terminal such as a smartphone, a tablet terminal, a laptop personal computer, and a dedicated controller. The user4downloads an inspection support application program (hereinafter referred to as an inspection support application) from a download site to the mobile terminal10via a network and install the inspection support application program. Alternatively, the inspection support application may be preinstalled on the mobile terminal10. Furthermore, the inspection support application may be configured to be of an application service provider (ASP) type or a software as a service (SaaS) type. The mobile terminal10executes the inspection support application, whereby the mobile terminal10communicates with the drone8and achieves various functions. Hereinafter, functions achieved by (processing units such as a central processing unit (CPU) of) the mobile terminal10executing the inspection support application may be described as functions of the mobile terminal10, but those functions are functions that the inspection support application causes the mobile terminal10to achieves.

The drone8is a relatively small device that flies in an unmanned manner, and may fly by remote control via wireless communication or may fly autonomously. In the present embodiment, as the drone8, a general-purpose drone having an imaging function such as a camera, a positioning function such as a global positioning system (GPS), and a function of communicating with the mobile terminal10, for example, a DJI Mavic Pro (Non Patent Literature 1) is assumed.

Referring toFIG.1, first, the user4designates an area to be imaged by the drone8(hereinafter, referred to as an imaging area12) on a display102of the mobile terminal10. In particular, the user4designates the base station apparatus6that is the object to be inspected on an electronic map displayed on the display102, and then sets the imaging area12so that the base station apparatus6enters the imaging area12. The object to be inspected such as the base station apparatus6may be displayed as an object that can be designated on the electronic map. The imaging area12may be designated by a polygon connecting points14tapped by the user4on the display102. Alternatively, when the object corresponding to the base station apparatus6is selected on the electronic map, the imaging area12may be automatically set. For example, the imaging area12may be set as a circular area having a predetermined radius centered on the position of the base station apparatus6.

FIG.2is a schematic diagram illustrating a situation in which the drone8flies along an outline flight path16that is set while corresponding to the base station apparatus6and images the base station apparatus6. When the user4designates the base station apparatus6and the imaging area12related to the base station apparatus6, the mobile terminal10sets the outline flight path16of the drone8so that an image necessary to generate an outline or coarse three-dimensional model of the designated base station apparatus6(hereinafter, referred to as an outline three-dimensional model) can be acquired.

Referring toFIG.2, the drone8captures an image of the base station apparatus6while flying according to the outline flight path16, and transmits the acquired image to the mobile terminal10. The mobile terminal10uses the acquired image to generate an outline three-dimensional model of the base station apparatus6. Setting of the outline flight path16of the drone8and generation of the three-dimensional model using the image acquired from the drone8may be achieved using known drone three-dimensional (3D) Mappping technology (e.g., refer to Non Patent Literature 2).

FIG.3is a schematic diagram illustrating a situation in which the user4designates a part whose details are to be confirmed in a generated outline three-dimensional model18. The mobile terminal10causes the display102to display the generated outline three-dimensional model18of the base station apparatus6. The user4designates a part of the base station apparatus6whose details the user4desires to confirm among parts of the base station apparatus6, by drawing a rectangle20on the outline three-dimensional model18displayed on the display102. The mobile terminal10specifies the part in the rectangle20as a designated part. The mobile terminal10sets a detailed flight path22of the drone8so that an image necessary to generate a detailed or fine three-dimensional model of the designated part (hereinafter, referred to as a detailed three-dimensional model) can be acquired on the basis of the positioning information when the designated part was imaged in the outline flight path16. The detailed three-dimensional model has a smaller granularity than the outline three-dimensional model and/or a higher resolution than the outline three-dimensional model, and/or a larger data amount for each object than the outline three-dimensional model.

Note that a plurality of types of rectangles20having different colors, line types, and shapes may be used. In this case, different types of rectangles may correspond to different instruction contents. For example, in a case where a designation is made by a red frame rectangle, the mobile terminal10may interpret the designation as an instruction to image only the surface of the rectangle in detail. In a case where a designation is made by a blue frame rectangle, the mobile terminal10may interpret the designation as an instruction to image a part in the rectangle in all directions in detail. Additionally, the shape may be not only a rectangle but also a circle, a triangle, or the like, as long as an area can be designated. Instead of the area, a user may designate only a point and set an area around the point as a target area.

FIG.4is a schematic diagram illustrating a situation in which the drone8flies according to the detailed flight path22that is set while corresponding to the designated part of the base station apparatus6and images the designated part. The drone8captures an image of the designated part of the base station apparatus6while flying according to the detailed flight path22, and transmits an acquired image to the mobile terminal10. The mobile terminal10uses the acquired image to generate a detailed three-dimensional model24of the designated part.

FIG.5is a schematic diagram illustrating a situation in which the user4is confirming the generated detailed three-dimensional model24. The mobile terminal10causes the display102to display the generated detailed three-dimensional model24of the designated part. The user4confirms the state of the designated part (a flaw, crack, discoloration, deterioration, dropout, foreign matter attachment, and the like) from a displayed detailed image of the designated part.

An inspection of an infrastructure or building is usually required to find an abnormality such as a crack on the order of one mm to several mm. In order to find an abnormality at this level, it is necessary to acquire a relatively detailed image. However, it is not realistic to acquire an image at that level when generating a three-dimensional model of the whole of the object to be inspected in terms of time and processing load. Therefore, in a support system2according to the present embodiment, first, the outline three-dimensional model18of the object to be inspected is generated, and the outline three-dimensional model18is used to receive the designation of a part to be inspected from the user4. Next, the drone8is flown once again so as to image the part designated by the user4in detail, and the detailed three-dimensional model24is generated. Thus, the user4can confirm a state of the necessary part in detail after grasping the whole of the object to be inspected schematically in the inspection. In addition, since there is no need to generate a detailed three-dimensional model of the whole object to be inspected, time required for the inspection can be reduced. Furthermore, since higher communication performance is not required for the drone8, costs can be reduced.

FIG.6is a hardware configuration diagram of the mobile terminal10ofFIG.1. The mobile terminal10includes a memory104, a processor106, a communication interface108, the display102, and an input interface110. These elements are each connected to a bus112and communicate with each other via the bus112.

The memory104is a storage area for storing data and a program. The data and the program may be permanently stored in the memory104or may be temporarily stored. In particular, the memory104stores the inspection support application. The processor106achieves various functions in the mobile terminal10by executing the program stored in the memory104, particularly, the inspection support application. The communication interface108is an interface for transmitting and receiving data to and from the outside of the mobile terminal10. For example, the communication interface108includes an interface for accessing a network and an interface for performing direct wireless communication with the drone8. The display102is a device for displaying various information, and is, for example, a liquid crystal display or an organic electroluminescence (EL) display. The input interface110is a device for receiving an input from the user. The input interface110includes, for example, a touch panel provided on the display102and various input keys.

FIG.7is a block diagram illustrating functions and a configuration of the mobile terminal10inFIG.1. In terms of hardware, each block illustrated in the block diagram of the present specification can be achieved by an element and a mechanical device including a CPU of a computer. In terms of software, each block can be achieved by a computer program and the like. Here, the block diagram depicts a functional block to be achieved by cooperation therebetween. Therefore, it is to be understood by those skilled in the art that these functional blocks can be achieved in various forms by combinations of hardware and software.

The mobile terminal10includes an outline processing unit114, a details processing unit116, a model using unit118, an outline image information holding unit120, a three-dimensional model holding unit122, a detailed image information holding unit124, and a detailed flight history holding unit126. The outline processing unit114performs a process related to generation of the outline three-dimensional model18. The details processing unit116performs a process related to generation of the detailed three-dimensional model24. The model using unit118performs a process related to the use of the generated detailed three-dimensional model24.

FIG.8is a data structure diagram illustrating an example of the outline image information holding unit120inFIG.7. The outline image information holding unit120holds a situation when the drone8acquires an outline or coarse image (hereinafter, referred to as an outline image) while flying according to the outline flight path16and a situation when the outline image was acquired. The outline image information holding unit120holds an outline flight ID that specifies a flight of the drone8that follows the outline flight path16, time when the drone8acquired the outline image, and a position of the drone8when the drone8acquired the outline image, and a file of the outline image in association with each other. The position of the drone8when the drone8acquired the outline image may be given by the positioning function of the drone8. The size of the file of the outline image may be relatively small.

FIG.9is a data structure diagram illustrating an example of the detailed image information holding unit124inFIG.7. The detailed image information holding unit124holds a detailed or fine image (hereinafter, referred to as a detailed image) that the drone8acquires while flying according to the detailed flight path22, and a situation when the detailed image was acquired. The detailed image information holding unit124holds a detailed flight ID that specifies a flight of the drone8that follows the detailed flight path22(hereinafter, referred to as a detailed flight), a designated part ID that specifies a part designated by the user4, and a drone8holds time when the detailed image was acquired, a position of the drone8when the drone8acquired the detailed image, and a file of the detailed image in association with each other. The size of the file of the detailed image may be relatively large, and may be, for example, larger than the size of the outline image file.

FIG.10is a data structure diagram illustrating an example of the detailed flight history holding unit126inFIG.7. The detailed flight history holding unit126holds a detailed flight history. The detailed flight history holding unit126holds the detailed flight ID, an object ID for specifying the object to be inspected, the designated part ID, a designated rectangular image file, time when the detailed flight was performed, and information on the detailed flight path22in the detailed flight, the file of the detailed image of the designated part in association with each other. The designated rectangular image file is a file of an image including the rectangle20drawn by the user4with respect to the outline three-dimensional model18when the designation of the designated part is received from the user4. For example, the designated rectangular image file may be a screen capture of the display102illustrated inFIG.3. The file of the detailed image of the designated part may be a file of a designated detailed image described later.

Returning toFIG.7, the outline processing unit114includes an imaging area acquisition unit128, an outline flight path setting unit130, an outline image acquisition unit132, and an outline model generation unit134. The imaging area acquisition unit128acquires information for setting an operation of the drone8so that the base station apparatus6is imaged. The imaging area acquisition unit128displays the electronic map on the display102of the mobile terminal10, and receives, from the user4, the designation of the base station apparatus6as the object to be inspected and the designation of the imaging area12via the electronic map.

The outline flight path setting unit130generates the operation of the drone8, that is, the outline flight path16so as to image the base station apparatus6on the basis of the information acquired by the imaging area acquisition unit128. The outline flight path setting unit130sets the outline flight path16in the drone8by transmitting the generated outline flight path16to the drone8.

The outline image acquisition unit132acquires, from the drone8, the outline image acquired as a result of the drone8performing a flight according to the outline flight path16(hereinafter, referred to as an outline flight). The outline image acquisition unit132receives the outline image sequentially transmitted from the drone8during the outline flight of the drone8, time when the outline image was acquired and a position when the outline image was acquired, and stores the time and the position in the outline image information holding unit120.

The outline model generation unit134reads the outline image held in the outline image information holding unit120, and generates the outline three-dimensional model18of the base station apparatus6on the basis of the read outline image. The outline model generation unit134registers the generated outline three-dimensional model18in the three-dimensional model holding unit122.

The details processing unit116includes a designation receiving unit136, a detailed flight path setting unit138, a position and orientation control unit140, a detailed image acquisition unit142, and a detailed model generation unit144. The designation receiving unit136uses the outline three-dimensional model18generated from the outline image to receive the designation of the part of the base station apparatus6from the user4. When receiving a request for confirming the details of the base station apparatus6from the user4, the designation receiving unit136reads the outline three-dimensional model18of the base station apparatus6held in the three-dimensional model holding unit122and causes the display102to display the outline three-dimensional model18. The designation receiving unit136specifies, as the designated part, the part of the base station apparatus6displayed in the rectangle20drawn by the user4, with respect to the displayed outline three-dimensional model18. A function at the time of a re-designation by the designation receiving unit136will be described later.

The detailed flight path setting unit138generates the operation of the drone8, that is, the detailed flight path22so that the detailed image of the designated part that is more detailed than the outline image of the designated part acquired in the outline flight is acquired. The detailed flight path setting unit138sets the detailed flight path22to the drone8by transmitting the generated detailed flight path22to the drone8.

When generating the detailed flight path22, the detailed flight path setting unit138uses the positioning information acquired in the outline flight. More specifically, the detailed flight path setting unit138first specifies the outline image in which the designated part is captured. This may be achieved, for example, by specifying the outline image that is a source for forming the designated part in the outline three-dimensional model18. Alternatively, this may be achieved by specifying, from the outline images held in the outline image information holding unit120, an outline image corresponding to the rectangle20drawn by the user4when the user designates the part. The detailed flight path setting unit138refers to the outline image information holding unit120and acquires the position associated with the specified outline image. The detailed flight path setting unit138sets a position of a first destination of the detailed flight path22on the basis of the acquired position. The detailed flight path setting unit138sets the flight path of the drone8after the drone8arrives at the first destination so that the designated part is imaged from a plurality of different viewpoints. A shape of the detailed flight path22from the start point to the first destination may be set so that the detailed flight path22is the shortest path to the designated part (or the position of the first destination) and avoids any obstacles on the way.

The position and orientation control unit140compares the outline image in which the designated part specified by the detailed flight path setting unit138is captured with a current image acquired by the imaging function of the drone8, thereby controlling the drone8so that a position or orientation for imaging the designated part is adjusted. The position and orientation control unit140transmits, to the drone8, the outline image in which the designated part specified by the detailed flight path setting unit138is captured. When the drone8flies along the detailed flight path22and reaches near the first destination, the drone8compares the received outline image with the current image of the designated part acquired by the imaging function of the drone8. The drone8adjusts the position and orientation of the drone8and/or a line-of-sight direction and a focal length of the camera of the drone8so that a difference between both the images is reduced. The detailed image of the designated part imaged by the drone8whose position and orientation have been adjusted in this way is referred to as a designated detailed image.

Note that the position and orientation of the drone8can be adjusted by other methods. For example, the position and orientation control unit140may use a display direction of the outline three-dimensional model18displayed on the display102when the user4designates the part and a position corresponding to the drawn rectangle20to adjust the detailed flight path22generated by the detailed flight path setting unit138. Alternatively, when relatively detailed positioning information is given to each object of the outline three-dimensional model18(for example, see Non Patent Literature 3), the position and orientation control unit140may generate or adjust the detailed flight path22on the basis of the positioning information of the object corresponding to the part designated by the rectangle20.

A specific example of the detailed flight path22will be described below.(1) A straight flight from the starting point to the first destination→adjustment of the position and orientation by image comparison→capturing of the designated detailed image→imaging from multiple angles while flying around the designated part→return(2) A circular flight from the starting point to the first destination (to avoid obstacles)→controlling of pan, tilt, and zoom (PTZ) to adjust the camera to a designated direction and a focal length while hovering at the first destination→capturing of the designated detailed image→imaging from multiple angles while flying around the designated part→return

The detailed image acquisition unit142acquires, from the drone8, the detailed image acquired as a result of a detailed flight performed by the drone8. The detailed image acquisition unit142receives the detailed image sequentially transmitted from the drone8during the detailed flight of the drone8, and time when the detailed image was acquired and a position when the detailed image was acquired, and stores the time and the position in the detailed image information holding unit124.

The detailed model generation unit144reads the detailed image held in the detailed image information holding unit124, and generates the detailed three-dimensional model24of the designated part on the basis of the read detailed image. The detailed model generation unit144registers the generated detailed three-dimensional model24in the three-dimensional model holding unit122.

When one detailed flight is completed and a corresponding detailed three-dimensional model24is stored in the three-dimensional model holding unit122, the details processing unit116registers information on the detailed flight in the detailed flight history holding unit126.

The model using unit118includes a detailed image providing unit146and a comparison image providing unit148. The detailed image providing unit146reads the detailed three-dimensional model24of the designated part held in the three-dimensional model holding unit122and causes the display102to display the detailed three-dimensional model24. Alternatively, the detailed image providing unit146may cause the display102to display the designated detailed image instead of the detailed three-dimensional model24. A function of the comparison image providing unit148will be described later.

An operation of the mobile terminal10having the above configuration will be described.

FIG.11is a flowchart illustrating a flow of a series of processes in the mobile terminal10inFIG.1. The mobile terminal10receives the designation of the imaging area12by the user4(S202). The mobile terminal10sets the outline flight path16on the basis of the imaging area12that has been received (S204). The mobile terminal10acquires the outline image from the drone8(S206). The mobile terminal10uses the acquired outline image to generate the outline three-dimensional model18(S208). The mobile terminal10receives the designation of a part that is an object for a detailed inspection from the user4via a display of the outline three-dimensional model18(S210). The mobile terminal10sets the detailed flight path22so that a detailed image of the designated part is acquired (S212). The mobile terminal10acquires the detailed image from the drone8(S214). The mobile terminal10uses the acquired detailed image to generate the detailed three-dimensional model24(S216). The mobile terminal10presents the detailed three-dimensional model24or the designated detailed image of the designated part to the user4(S218). The mobile terminal10registers the designated part ID, the detailed flight path22, and the designated detailed image in the detailed flight history holding unit126(S220).

FIG.12is a representative screen diagram of a detailed imaging history selection screen150displayed on the display102of the mobile terminal10. The detailed imaging history selection screen150displays, for each past detailed flight, information152for specifying the past detailed flight, and a confirm with image button154. The information152for specifying the past detailed flight includes a date on which the detailed flight was performed, a name of the object to be inspected, and a name of the designated part. Upon receiving a request of displaying an imaging history from the user4, the designation receiving unit136generates the detailed imaging history selection screen150by referring to the detailed flight history holding unit126, and causes the display102to display the detailed imaging history selection screen150. The date on the detailed imaging history selection screen150corresponds to the time when the detailed flight was performed in the detailed flight history holding unit126, and the name of the object to be inspected on the detailed imaging history selection screen150corresponds to the object ID in the detailed flight history holding unit126, and the name of the designated part on the detailed imaging history selection screen150corresponds to the designated part ID in the detailed flight history holding unit126.

FIG.13is a representative screen diagram of a designated part confirmation screen156displayed on the display102of the mobile terminal10. When the user4taps a confirm button154of a desired detailed flight on the detailed imaging history selection screen150, the designation receiving unit136reads the designated rectangular image file corresponding to the detailed flight of the confirm button154tapped from the detailed flight history holding unit126, and generates the designated part confirmation screen156. The designated part confirmation screen156is an image including the rectangle20drawn by the user4with respect to the outline three-dimensional model18when the designation of the designated part was received from the user4with respect to the detailed flight of the tapped confirm button154and a re-designate button158. For example, when the user4taps the confirm button of the past detailed flight specified by “8/15, BS #1000, number 1 antenna lower end” among the detailed flight histories displayed on the detailed imaging history selection screen150illustrated inFIG.12, the screen changes and the designated part confirmation screen156is displayed. On the designated part confirmation screen156, a rectangle that the user4drew to designate the lower end of a number 1 antenna (designated part) on the display102where the outline three-dimensional model of BS #1000 (name of the base station apparatus6) was displayed on 8/15 (past time) is displayed together with the outline three-dimensional model.

The user4confirms the designated part on the designated part confirmation screen156, and taps the re-designate button158if there is no problem. Then, the designation receiving unit136receives the tap as a request for re-imaging the designated part. The detailed flight path setting unit138reads the detailed flight path corresponding to the designated rectangular image file read at the time of generation of the designated part confirmation screen156from the detailed flight history holding unit126, and transmits the detailed flight path to the drone8, thereby being able to set again the same detailed flight path as the past detailed flight path in the drone8.

Note that in another embodiment, a designation receiving unit136may cause a display102to display a screen that displays a designated detailed image or a detailed three-dimensional model acquired in the past instead of the designated part confirmation screen156. In this case, when a user4requests a re-designation, the designation receiving unit136reads a detailed flight path corresponding to a designated detailed image being displayed or the detailed three-dimensional model being displayed from the detailed flight history holding unit126, and sets again the detailed flight path in the drone8.

In an inspection of an infrastructure facility such as a base station apparatus and a building, it has been empirically known that a pat to be inspected with priority and a part to be always inspected every time. In an example of the base station apparatus, since degradation starts from the tip of the antenna, the tip is a part that is always confirmed in every inspection. In the present embodiment, once the designated part that needs to be repeatedly inspected is specified from the outline three-dimensional model in the first inspection, the setting of the detailed flight path to the drone8in the next and subsequent inspections is simply completed just by selecting the designated part from the detailed flight history. As a result, users' convenience is improved.

FIG.14is a representative screen diagram of a designated part transition screen160displayed on the display102of the mobile terminal10. The comparison image providing unit148causes the display102to display the designated part transition screen160that displays, in a comparable manner, the detailed images or the detailed three-dimensional models of the same designated part acquired in the detailed flight path22at different time points in a comparable manner. When the comparison image providing unit148receives a comparison request including the designations of the object to be inspected and the designated part from the user4, the comparison image providing unit148acquires, from the detailed flight history holding unit126, the file of the designated detailed image corresponding to the designated object to be inspected (object ID) and the designated designated part (designated part ID). The comparison image providing unit148generates the designated part transition screen160by arranging, in chronological order, the designated detailed images that have been acquired. Note that the detailed three-dimensional model that can be acquired from the three-dimensional model holding unit122may be displayed on the designated part transition screen160instead of or in addition to the designated detailed image.

In the designated detailed image displayed on the designated part transition screen160, a difference from an immediately preceding designated detailed image displayed on the same screen is displayed in a highlighted manner. For example, in the designated detailed image corresponding to “8/17” on the designated part transition screen160inFIG.14, a new crack162that is a difference from the designated detailed image of “8/15” that is an immediately preceding designated detailed image is displayed in a different manner from other cracks, for example, in an emphasized manner in which the crack162is displayed in bold.

In this way, the designated detailed images of the designated designated part are displayed in chronological order, whereby the user4can more accurately confirm a deteriorated part and the progress of degradation at a glance.

In the above embodiment, examples of the holding unit are a hard disk and a semiconductor memory. Additionally, it is to be understood by those skilled in the art referred to the present specification that on the basis of the description in the present specification, each part can be achieved by a CPU (not illustrated), a module of an installed application program, a module of a system program, a semiconductor memory that temporarily stores the content of data read from a hard disk, or the like.

According to the inspection support system2according to the present embodiment, a three-dimensional model is generated in two stages of an outline and a detail, by aerial photography of the drone8, whereby there is no need to generate a detailed three-dimensional model of a part of an object to be inspected other than a part thereof that is desired or required to be inspected. Thus, it possible to reduce the processing load and processing time while achieving a detailed confirmation of the desired or required part. Additionally, since the communication performance required for the drone8is not so high, an increase in cost can be suppressed by the use of a relatively inexpensive drone.

The configuration and operation of the inspection support system2according to the embodiment have been described above. It is to be understood by those skilled in the art that the embodiment is illustrative, that various modifications can be made to the combination of each constituent element and each process, and that such modifications are also within the scope of the present invention.

In the embodiment, a case where the user4performs the inspection by going to the vicinity of the base station apparatus6and operating the mobile terminal10has been described, but the present invention is not limited to this. For example, the designation of the base station apparatus6as the object to be inspected, the designation of the imaging area12, the designation of the designated part using the acquired outline three-dimensional model18, and the confirmation of the designated detailed image may be performed by an operator of a center facility located at a position remote from the base station apparatus6. The worker only needs to carry the drone8to the vicinity of the base station apparatus6. A process in this case is understood by replacing the mobile terminal10of the embodiment with a desktop terminal of the operator.

Alternatively, a three-dimensional model of each component of the base station apparatus6may be stored in advance on the desktop terminal of the operator, and the operator may select a designated part from the three-dimensional model.

REFERENCE SIGNS LIST

2Inspection support system4User6Base station apparatus8Drone10Mobile terminal