Crane information display system

This crane information display system is provided with a terminal device having a camera and captures an image of a crane with the camera to obtain a camera image, the crane information display system including a crane information acquisition unit that reads display information of an information display unit, which is mounted in the crane, from the camera image and acquires information about the crane; a position/orientation calculation unit that calculates the position and orientation of the crane from the camera image; an information processing unit that converts the information about the crane acquired by the crane information acquisition unit into three-dimensional image information corresponding to the position and orientation of the crane calculated by the position/orientation calculation unit; and an image display unit that overlays the information about the crane converted by the information processing unit on the camera image and displays the same.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2020/036270 (filed on Sep. 25, 2020) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2019-176758 (filed on Sep. 27, 2019), which are all hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present invention relates to a crane information display system.

BACKGROUND ART

Conventionally, a crane information display system that displays information of a crane is known (for example, see Patent Literature 1).

Patent Literature 1 discloses a configuration that visualizes information on an operating state of a crane and displays a display screen on a display unit of a mobile terminal. As a result, even a person outside the crane can see the operating state of the crane.

CITATION LIST

Patent Literature

Patent Literature 1: JP 2014-227281 A

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

However, in the configuration disclosed in Patent Literature 1, information on an operating state of a crane is visualized and displayed by drawing the crane from above or from the side. Therefore, in the configuration disclosed in Patent Literature 1, there is a problem in that it is necessary to examine it in two-dimensions.

Therefore, an object of the present invention is to provide a crane information display system capable of enabling information about a crane at a work site to be three-dimensionally examined.

Solutions to Problems

A main disclosure for solving the above-described problem is a crane information display system that is provided with a terminal device having a camera and captures as image of a crane with the camera to obtain a camera image, the crane information display system including a crane information acquisition unit that reads display information of an information display unit, which is mounted in the crane, from the camera image and acquires information about the crane, a position/orientation calculation unit that reads display information of the information display unit, acquires information about a reference shape of the information display unit and a mounting position of the information display unit on the crane from the camera image, extracts a contour shape of the information display unit reflected in the camera image, and calculates the position and orientation of the crane based on the information about the reference shape of the information display unit, the mounting position of the information display unit, and the contour shape of the information display unit, an information processing unit that converts the information about the crane acquired by the crane information acquisition unit into three-dimensional image information corresponding to the position and orientation of the crane calculated by the position/orientation calculation unit, and an image display unit that overlays the information about the crane converted by the information processing unit on the camera image and displays the information.

Effects of the Invention

In the crane information display system of the present invention configured as described above, information about a crane can be examined three-dimensionally at a work site.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments for achieving a crane information display system according to the present invention will be described with reference to Example 1 illustrated in the drawings.

[Configuration of Crane Information Display System]

FIG.1is a diagram illustrating a crane information display system of Example 1. Hereinafter, a configuration of the crane information display system of Example 1 will be described.

As illustrated inFIG.1, an example in which, in the crane information display system100of Example 1, a worker M captures an image of a crane1placed at a work site with a tablet terminal50as a user terminal provided with a camera51will be described. An example in which the crane information display system100of Example 1 captures an image of the crane1with the camera51in a state where an outrigger80does not protrude will be described. The front-rear direction of the crane1is defined as a front-rear direction D.

As illustrated inFIG.1, the crane1includes a traveling body10, a swirling body20, and a boom30.

The traveling body10includes a vehicle body, frame11, the outrigger80, a traveling device for self-traveling on a road or a work site, and the like.

The outrigger80includes a rear outrigger81attached to a rear side surface of the vehicle body frame11of the traveling body10and a front outrigger82attached to a front side surface of the vehicle body frame11. The outrigger80is stored in the vehicle body frame11during travel. On the other hand, at the time of work, the outrigger80protrudes in the horizontal direction and the vertical direction, lifts the entire vehicle body, and stabilizes the orientation.

An April Tag (April Tag) is attached to the vehicle body frame11of the traveling body10as an information display unit40(that is, a marker). The information display unit40includes a first information display unit41attached to the front surface of the vehicle body frame11, a second information display unit42attached to the rear surface of the vehicle body frame11, a third information display unit43attached to the right side surface of the vehicle body frame11, and a fourth information display unit44attached to the left side surface of the vehicle body frame11. The first information display unit41, the second information display unit42, the third information display unit43, and the fourth information display unit44each have separate display information (that is, code information) of, for example, text, a code, a symbol, a pattern, or the like attached to the surface thereof. The display information (that is, the code information) held by each of the first information display unit41, the second information display unit42, the third information display unit43, and the fourth information display unit44can be read from the camera image generated by the camera51using a decoding program stored in advance in a control unit60(described later) or the like.

The first information display unit41records information about the model of the crane1, information about the position where the first information display unit41is attached in the crane1, and the shape and size of the first information display unit41. The second information display unit42records information about the model of the crane1, information about the position where the second information display unit42is attached in the crane1, and the shape and size of the second information display unit42. The third information display unit43records information about the model of the crane1, information about the position where the third information display unit43is attached in the crane1, and the shape and size of the third information display unit43. The fourth information display unit44records information about the model of the crane1, information about the position where the fourth information display unit44is attached in the crane1, and the shape and size of the fourth information display unit44. The shapes and sizes recorded in the first information display unit41, the second information display unit42, the third information display unit43, and the fourth information display unit44are, for example, shapes and sizes obtained from a predetermined distance in front (hereinafter referred to as a “reference shape” and a “reference size”).

The first information display unit41, the second information display unit42, the third information display unit43, and the fourth information display unit44have the same outer shape (for example, a rectangle) and the same size. The first information display unit41, the second information display unit42, the third information display unit43, and the fourth information display unit44can be identified from the display information (that is, the code information) held by each.

The swirling body20is provided above the traveling body10and is rotatable about a vertical axis C1with respect to the traveling body10. The swirling body20includes a cabin21. The cabin21includes an operation unit (for example, a steering wheel, a shift lever, an accelerator pedal, a brake pedal, and the like) for controlling traveling of the traveling body10. The cabin21includes an operation unit that operates the swirling body20, a boom30, a winch, or the like. The operator riding in the cabin21operates the operation unit to swing the swirling body20, raise and lower the boom30, extend and contract the boom30, and rotate the winch to perform the work.

The proximal end side of the boom30is supported by the swirling body20, and is attached to the swirling body20to be raised and lowered. The boom30is raised and lowered by a raisable/lowerable cylinder22provided in the swirling body20, and is expanded and contracted by a telescopic cylinder (not illustrated).

The boom30includes intermediate booms32to35between a proximal end boom31on the proximal side and a distal end boom36on the distal end side. The intermediate booms32to35and the distal end boom36are telescopically stored in the proximal end boom31in sequence.

A sheave37is disposed in a boom head36aprovided at the distal end of the distal end boom36. A wire rope36for a suspended load is wound around a winch provided near the proximal end of the boom30of the swirling body20. The wire rope38is disposed along the boom30from the winch to the sheave37in the axial direction, and the wire rope38wound around the sheave37is suspended downward in the vertical direction from the sheave37. A hook39is provided at the lowermost portion of the wire rope38.

A load is hung on the hook39, the wire rope38wound around the winch is unwound to lower the hook39, and the wire rope38is wound up to raise the hook39.

The crane1configured as described above moves the load hung on the hook39to a predetermined position by unwinding and winding the wire rope38by the winch, raising, lowering, extending, and contracting the boom30, and swinging the swirling body20.

As illustrated inFIG.1, the tablet terminal50includes the camera51, an image display unit52, and an input unit53.

An image captured with the camera51is displayed on the image display unit52. The image display unit52is also configured as a touch panel as the input unit53.

[Functional Configuration of Crane Information Display System]

FIG.2is a block diagram illustrating a functional configuration of the crane information display system100of Example 1.FIG.3is a diagram illustrating an image displayed on an image display unit of Example 1. Hereinafter, a functional configuration of the crane information display system100of Example 1 will be described.

In the crane information display system100, an image of the information display unit40captured with the camera51and the input information input by the input unit53are input to the control unit60(in the present embodiment, a control unit built in the tablet terminal50), and the information controlled by the control unit60is output by the image display unit52.

The camera51can be, for example, a camera51provided in a general tablet terminal50. The camera51can capture an image of the crane1, a site environment around the crane1, and the information display unit40.

The input unit53can input an overhanging amount of the outrigger80in the horizontal direction, a suspended load, a length of the boom30, and the like. The length of the boom30is, for example, the length of the boom30in a state where the distal end boom36and the intermediate booms32to35are stored in the proximal end boom31(a fully contracted state), the length of the boom30in a state where the distal end boom36is extended, or the length in a state in which the distal end boom36and the intermediate booms32to35are extended (a fully extended state).

The control unit60includes a storage unit61, a crane information acquisition unit62, a position/orientation calculation unit65, and an information processing unit66. The control unit60is a known microcomputer including, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), and the like, and functions of the control unit60(the crane information acquisition unit62, the position/orientation calculation unit65, and the information processing unit66) are achieved by, for example, the CPU referring to a control program or various data stored in the storage unit61(for example, an HDD), the ROM, or the RAM.

The storage unit61stores performance information about various cranes. The performance information includes, for example, information on the shape of the outrigger80, information about a workable area in the horizontal direction and information about a workable area in the height direction according to the overhanging amount of the outrigger80and the length of the boom30, information about a derricking angle range of the boom30, information on a load factor, information on a tail swing area, and the like.

The crane information acquisition unit62acquires information about the model of the crane1based on the image of the information display unit40captured with the camera51, and acquires performance information about the acquired model of the crane1from the storage unit61. That is, the crane information acquisition unit62reads the image of the information display unit40captured with the camera51(that is, the display information of the information display unit40is read from the camera image), acquires the model of the crane1, and acquires the performance information of the acquired model from the storage unit61.

The crane information acquisition unit62includes a virtual outrigger generation unit63and a workable area calculation unit64.

The virtual outrigger generation unit63generates three-dimensional data (that is, image information about the three-dimensional image of the outrigger80) of the virtual outrigger as the information of the crane1based on the information on the shape of the outrigger80stored in the storage unit61(that is, the shape of the outrigger80corresponding to the model of the crane1specified from the display information of the information display unit40) and the overhanging amount of the outrigger80in the horizontal direction input to the input unit53.

The workable area calculation unit64refers to the information about the model of the crane1specified from the display information of the information display unit40, and calculates the workable area of the crane1as the information about the crane1based on the suspended load input to the input unit53and the length of the boom30. The workable area calculation unit64can calculate workable areas with a plurality of load factors. In Example 1, the workable area calculation unit64calculates a workable area with a load factor of 80% and a workable area with a load factor of 100%. A workable area is an area where work can be performed in the horizontal direction of the crane1on the installation surface of the crane1.

The position/orientation calculation unit65calculates the position and orientation of the crane1based on the image of the information display unit40captured with the camera51.

Specifically, the position/orientation calculation unit65reads the image of the information display unit40captured with the camera51(that is, the display information of the information display unit40is read from the image of the camera51), acquires information about the position (that is, the mounting position of the information display unit40) where the information display unit40is attached to the crane1, and calculates the orientation of the crane1.

For example, when the image of the information display unit40captured with the camera51shows the first information display unit41, since the first information display unit41is attached to the front surface of the vehicle body frame11, the camera51captures an image of the crane1facing forward. When the image of the information display unit40captured with the camera51shows the second information display unit42, since the second information display unit42is attached to the rear surface of the vehicle body frame11, the camera51captures an image of the crane1facing rearward. When the image of the image of the information display unit40captured with the camera51shows the third information display unit43, since the third information display unit43is attached to the right side surface of the vehicle body frame11, the camera51captures an image of the crane1facing rightward. When the image of the information display unit40captured with the camera51shows the fourth information display unit44, since the fourth information display unit44is attached to the left side surface of the vehicle body frame11, the camera51captures an image of the crane1facing leftward.

The position/orientation calculation unit65acquires information about the orientation of the crane1based on the shape (hereinafter, referred to as a “contour shape of the information display unit40”) of the information display unit40captured with the camera51and the reference shape of the information display unit40recorded in the information display unit40. That is, the position/orientation calculation unit65reads the display information (that is, code information) of the information display unit40from the image of the camera51to acquire information related to the reference shape of the information display unit40, and extracts the contour shape of the information display unit40reflected in the camera51from the image of the camera51by known pattern matching or the like. Then, the position/orientation calculation unit65calculates information about the orientation of the crane1by comparing the contour shape of the information display unit40reflected in the image of the camera51with the reference shape of the information display unit40.

For example, in a case where the information display unit40captured with the camera51is the first information display unit41and has a rectangular shape recorded in the first information display unit41(that is, in a case where both the contour shape of the first information display unit41reflected in the image of the camera51and the reference shape of the first information display unit41specified from the display information of the first information display unit41are rectangular), the crane1is in an orientation facing straight forward with respect to the camera51. When the information display unit40captured with the camera51is the first information display unit41, is not the rectangular shape recorded in the first information display unit41, and is a trapezoid in which the left edge of the information display unit40is longer (higher) than the right edge (that is, the contour shape of the first information display unit41reflected in the image of the camera51is a trapezoid of which the left edge is longer than the right edge, and the reference shape of the first information display unit41specified from the display information of the first information display unit41is a rectangular shape), the crane1is in an orientation facing slightly left from the straight front with respect to the camera51. When the information display unit40captured with the camera51is the first information display unit41, is not the rectangular shape recorded in the first information display unit41, and is a trapezoid in which the right edge of the information display unit40is longer than the left edge (that is, the contour shape of the first information display unit41reflected in the image of the camera51is a trapezoid of which the right edge is longer than the left edge, and the reference shape of the first information display unit41specified from the display information of the first information display unit41is a rectangular shape), the crane1is in an orientation facing slightly right from the straight front with respect to the camera51.

That is, the position/orientation calculation unit65compares the shape of the information display unit40captured with the camera51with the shape of the information display unit40recorded in the information display unit40, and acquires information about the orientation of the crane1.

The position/orientation calculation unit65acquires information about the position of the crane1based on the size of the contour shape of the information display unit40captured with the camera51and the reference size of the information display unit40recorded in the information display unit40. Specifically, the position/orientation calculation unit65compares the reference size of the information display unit40recorded in the information display unit40with the size of the contour shape of the information display unit40captured with the camera51, and calculates the distance from the camera51to the information display unit40.

The information processing unit66processes the performance information of the crane1acquired by the crane information acquisition unit62, the virtual outrigger generated by the virtual outrigger generation unit63, and the workable area of the crane1calculated by the workable area calculation unit64into information corresponding to the position and the orientation of the crane1calculated by the position/orientation calculation unit65.

That is, the information processing unit66processes the information about the crane1acquired by the crane information acquisition unit62into information corresponding to the position and the orientation of the crane1calculated by the position/orientation calculation unit65. In other words, the information processing unit66converts the information about the crane1acquired by the crane information acquisition unit62into three-dimensional image information corresponding to the position and the orientation of the crane1calculated by the position/orientation calculation unit65. For example, when the crane1is viewed from the capturing position of the camera51, the information processing unit66converts the three-dimensional image of a virtual outrigger80A so that the three-dimensional image of the virtual outrigger80A becomes an image simulating a state where the outrigger80is actually overhung. Such image processing of the information processing unit66is achieved by known coordinate conversion processing or the like.

The image display unit52displays the information (that is, information about the crane1converted into image information to be displayed by the information processing unit66) processed by the information processing unit66to be superimposed on the image of the camera51. Specifically, as illustrated inFIG.3, the image display unit52superimposes and displays the tail swing area73of the crane1acquired by the crane information acquisition unit62, the virtual outrigger80A generated by the virtual outrigger generation unit63, and workable areas71and72of the crane1calculated by the workable area calculation unit64on the image of the crane1and the site environment around the crane1captured with the camera51. The image display unit.52displays the suspended load input to the input unit53.

That is, the image display unit52displays the workable areas71and72of the crane1, the virtual outrigger80A of the crane1, and the tail swing area.73of the crane1in a three-dimensional image so as to be superimposed on the crane1or the surrounding environment of the crane1reflected in the image of the camera51so that the user can three-dimensionally examine how each part of the crane1affects the site environment when the crane1is actually operated on site. The image display unit52displays the virtual outrigger80A at the position of the outrigger of the crane1reflected in the image of the camera51, for example. In addition, the image display unit52also displays the workable areas71and72of the crane1around the crane1reflected in the image of the camera51, for example. In addition, the image display unit52displays the tail swing area73of the crane1around the swivel base of the crane1shown in the image of the camera51, for example.

The virtual outrigger80A includes a virtual front outrigger82A and a virtual rear outrigger81A. The workable area of the crane1includes a workable area72with a load factor of 80% and a workable area71with a load factor of 100%.

[Flow of Processing by Control Unit]

FIG.4is a flowchart illustrating a flow of processing by the control unit.60of the crane information display system100of Example 1. Hereinafter, a flow of processing by the control unit60of the crane information display system100according to Example 1 will be described.

When the worker M captures an image of the crane1installed at the work site and the surrounding site environment with the camera51of the tablet terminal50, as illustrated inFIG.4, the crane information acquisition unit62acquires information about the model of the crane1based on the information display unit40captured with the camera51(step S101).

Next, the position/orientation calculation unit65calculates the position and orientation of the crane1based on the information display unit40captured with the camera51(step S102).

Next, the control unit60determines whether the overhanging amount of the outrigger80has been input to the input unit53(step S103). In a case where it is determined that the overhanging amount of the outrigger80is input to the input unit53(YES in step S103), the virtual outrigger generation unit63generates the virtual outrigger80A (step S104), and the process proceeds to step S105. On the other hand, in a case where the overhanging amount of the outrigger80is not input to the input unit53(NO in step S103), the process proceeds to step S105.

Next, the control unit60determines whether the suspended load and the length of the boom30have been input to the input unit53(step S105). In a case where it is determined that the suspended load and the length of the boom30are input to the input unit53(YES in step S105), the workable area calculation unit64calculates the workable areas71and72(step S106), and the process proceeds to step107. On the other hand, in a case where it is determined that the suspended load and the length of the boom30are not input to the input unit53(NO in step S105), the process proceeds to step S107.

Next, the information processing unit66processes the information about the crane1acquired by the crane information acquisition unit62into information corresponding to the position and the orientation of the crane1calculated by the position/orientation calculation unit65(step S107). That is, the information processing unit66processes the performance information of the crane1acquired by the crane information acquisition unit62, the virtual outrigger80A generated by the virtual outrigger generation unit63, and the workable areas71and72of the crane1calculated by the workable area calculation unit64into information corresponding to the position and the orientation of the crane1calculated by the position/orientation calculation unit65.

Next, the image display unit52superimposes and displays the tail swing area73of the crane1acquired by the crane information acquisition unit62, the virtual outrigger80A generated by the virtual outrigger generation unit63, and workable areas71and72of the crane1calculated by the workable area calculation unit64on the image of the crane1and the site environment around the crane1captured with the camera51(step S108), and ends the processing.

[Operation of Crane Information Display System]

Hereinafter, an operation of the crane information display system100of Example 1 will be described.

The crane information display system100according to Example 1 includes: the crane information acquisition unit62that acquires information about the crane1by capturing the information display unit40that displays information about the crane1by the camera51, the information display unit40being provided in the crane1; the position/orientation calculation unit65that calculates a position and an orientation of the crane1based on the information display unit40captured with the camera51; the information processing unit66that processes the information about the crane acquired by the crane information acquisition unit62into information corresponding to the position and the orientation of the crane1calculated by the position/orientation calculation unit65; and the image display unit52that displays the information processed by the information processing unit66to be superimposed on an image captured with the camera51(FIG.2).

As a result, the information about the crane1can be superimposed and displayed on the image obtained by capturing the crane1and the surroundings thereof. Therefore, at the work site, the crane1, the site environment around the crane1, and the information about the crane1can be confirmed by a three-dimensional mage in real time. As a result, a work plan of the crane1can be examined in real time at the work site.

The crane information display system100of Example 1 includes the input unit53that inputs the overhanging amount of the outrigger80of the crane1, and the crane information acquisition unit62includes the virtual outrigger generation unit63that generates the three-dimensional virtual outrigger80A based on the input value to the input unit53(FIG.2).

As a result, the virtual outrigger80A having the overhanging amount corresponding to the input value can be superimposed and displayed on the image obtained by capturing the crane1and the surroundings thereof. Therefore, at the work site, the overhanging amount of the outrigger80can be confirmed on a three-dimensional image in real time.

The crane information display system100of Example 1 includes the input unit53that inputs the suspended load and the length of the boom30of the crane1, and the crane information acquisition unit62includes the workable area calculation unit64that calculates the workable areas71and72of the crane1based on the input value to the input unit53(FIG.2).

As a result, it is possible to display the workable areas71and72for a predetermined suspended load in an overlapping manner based on the length of the boom30on the image obtained by capturing the crane1and the surroundings thereof. Therefore, at the work site, the workable areas71and72can be confirmed by a three-dimensional image in real time.

In the crane information display system100of Example 1, the information about the crane1includes the tail swing area73of the crane1(FIG.3).

As a result, the tail swing area73can be superimposed and displayed on the image obtained by capturing the crane1and the surroundings thereof. Therefore, at the work site, the tail swing area73can be confirmed by a three-dimensional image in real time.

The crane information display system of the present invention has been described above based on Example 1. However, the specific configuration is not limited to this example, and design changes, additions, and the like are allowed without departing from the gist of the invention according to each claim of the claims.

In Example 1, an example in which the information display unit40is an April Tag has been described. However, the information display unit is not limited to this mode, and may be a two-dimensional code such as a QR code (registered trademark). In addition, a crane itself may be used as the information display unit, and information of the crane may be acquired by image recognition using deep learning.

In Example 1, an example in which one April Tag as the information display unit40is attached to each of the front surface, the rear surface, the right side surface, and the left side surface of the vehicle body frame11of the crane1has been described. However, two or more information display units40may be attached to each of the front surface, the rear surface, the right side surface, and the left side surface of the vehicle body frame11of the crane1.

In Example 1, an example has been described in which the workable area72with a load factor of 80% and the workable area71with a load factor of 100% are displayed on the image display unit52. However, one workable area may be displayed on the image display unit, or three or more workable areas may be displayed on the image display unit. The load factor of the workable area is not limited to 80% or 100%.

In Example 1, the user terminal is the tablet terminal50including the camera51, the input unit53, and the image display unit52. However, the user terminal may be a smartphone. Further, in the user terminal, the camera and the image display unit may be separate bodies.

In Example 1, an example is which the performance information of the crane1is stored in the storage unit61has been described. However, the performance information of the crane may be stored in the information display unit.

The entire disclosure of the specification, drawings, and abstract included in Japanese Patent Application No. 2019-176758 filed on Sep. 27, 2019 is incorporated herein by reference.

REFERENCE SIGNS LIST