Patent ID: 12221774

MODES FOR CARRYING OUT THE INVENTION

In the following, embodiments of the present invention are described with reference to the drawings. It is to be noted that, in the drawings, like members are denoted by like reference characters, and overlapping description of them is suitably omitted

First Embodiment

In the present embodiment, an example of an operation record analysis system for a construction machine in which contents that meet conditions are extracted from within operation records and are displayed preferentially.

FIG.1is a block diagram of the operation record analysis system for a construction machine of the present embodiment. Here, description is given taking an excavator as a target construction machine.

A machine body position sensor101is configured, for example, from a GPS device or a GNSS device, and senses the position of the excavator and sends out a result of the sensing.

A machine body posture sensor102is configured from angle sensors for sensing angles of a boom, an arm, and a bucket, an angle sensor for sensing a swing angle of an upper swing structure and so forth, and senses a positional relation between a lower track body and the upper swing structure of the excavator, a positional relation among the boom, arm, and bucket and so forth and sends out a result of the sensing.

An operation state sensor103is configured from a rotation speed sensor for sensing an engine speed, a pressure sensor for sensing a load pressure of an actuator, and so forth, and acquires a state relating to operation of the excavator such as output power of the engine, movable part hydraulic pressures, and so forth, and sends out resulting acquired state information.

An work position acquisition section104acquires a place at which the excavator is working and a positional relation between and states of the movable parts of the excavator on the basis of information received from the machine body position sensor101and the machine body posture sensor102, and sends out resulting acquired information.

An operation state acquisition section105acquires a state of the engine and a state of a hydraulic system on the basis of information received from the operation state sensor103, and sends out resulting acquired state information.

An work contents estimation section106performs such estimation of which one of, for example, excavation, earth removal and travel the work being currently performed by the excavator is on the basis of information received from the work position acquisition section104and the operation state acquisition section105, and sends out resulting estimated work contents.

An operation information calculation section107integrates work contents outputted from the work contents estimation section106and information representative of whether or not an object existing around the excavator and the excavator are close to each other. Then, when the operation information calculation section107decides that they are close to each other, it adds such setting as to display this preferentially to produce operation information and sends out the resulting operation information.

A communication device108is, for example, a portable communication device and is configured from a wireless LAN terminal, and sends out operation information produced by the operation information calculation section107to an apparatus outside the system through a communication line.

A recording device109is configured, for example, from a memory or a disk, and records operation information outputted from the operation information calculation section107such that it outputs this information on the basis of a request from the operation information calculation section107or the communication device108.

A display device110is configured, for example, from a console and a display, and displays operation information outputted from the operation information calculation section107on the display in response to an operation of the console.

An object sensor111includes cameras501and502(depicted inFIG.3) attached to the machine body, and images states around the excavator and outputs resulting camera images.

An object sensing section113receives images outputted from the object sensor111, senses an object existing around the excavator from within the images and sends out a result of the sensing.

A shape information retaining section114retains a three-dimensional shape of the excavator and outputs the three-dimensional shape in response to a request from an arrangement calculation section112.

An arrangement state decision section115compares current three-dimensional arrangement of parts of the excavator outputted from the arrangement calculation section112with a result of projection in which an object sensing result in camera images outputted from the object sensing section113is projected from2D to3D to decide whether or not an object sensed by the object sensing section113exists in the proximity of the position at which the machine body of the excavator currently exists, and sends out a result of the decision to the operation information calculation section107.

Examples of a display image of operation information at a point of time at which an event that may possibly make a factor of decrease in the operation rate has occurred, in the operation record analysis system for a construction machine described above, are described with reference toFIGS.2to10.

FIG.2depicts a flow of processing of the operation record analysis system according to the present invention. The flow of processing is roughly divided into two. In particular, the processing upon operation information recording is executed when the excavator is operating. The processing upon operation information verification is executed when operation information about the excavator recorded by the recording device109is verified using the display device110.

If operation upon operation information recording is started in the operation record analysis system for a construction machine according to the present invention, then a process in step201is performed first.

In step201, the work position acquisition section104acquires a work position on the basis of information inputted from the machine body position sensor101and the machine body posture sensor102. A state of the excavator in this example is depicted inFIG.3. The excavator301includes at least a lower track body302, an upper swing structure303swingably provided on the lower track body302, and a front work implement provided on the upper swing structure303and including a boom304, an arm305, and a bucket306. The excavator301exists on the ground surface having a slanting surface. The cameras501and502are attached to the machine body and image the surroundings of the machine body. The machine body position sensor101may use such a technique as to acquire a latitude and a longitude on the earth using, for example, the GPS (Global Positioning System). As another technique, also such a technique as to acquire a relative position in a target area of a construction plan using a total station or the like is possibly applied. The machine body posture sensor102senses the posture of the machine body using angle sensors and so forth mounted on the excavator301. The posture in this case is information in which a direction of the lower track body302, a difference in swing angle between the lower track body302and the upper swing structure303and differences in arrangement angle between portions of the upper swing structure303, boom304, arm305, and bucket306are summarized. Thereafter, the processing advances to step202.

In step202, the operation state acquisition section105acquires an operation state on the basis of information outputted from the operation state sensor103. The operation state sensor103outputs information obtained from sensors such as an engine output power sensor, a hydraulic pressure sensor, a speedometer, and an abnormality sensing sensor to the operation state acquisition section105. Thereafter, the processing advances to step203.

In step203, the work contents estimation section106estimates current work contents of the excavator on the basis of the information outputted from the work position acquisition section104and the operation state acquisition section105. An example of the work position outputted from the work position acquisition section104and the operation state outputted from the operation state acquisition section105is depicted inFIG.4. The work position here is three-dimensional coordinates of the position at which swing center on the bottom face of the excavator exists. Meanwhile, the operation state is angles θ1, θ2, θ3, and θ4 formed relatively by each of the parts of the excavator. In the estimation of the work contents, the work contents estimation section106estimates, on the basis of, for example, information about the posture of the machine body outputted from the machine body posture sensor102, that the work contents are traveling when the position of the bucket306is higher than the ground surface and besides the value of the speedometer is equal to or higher than a fixed value. On the other hand, the work contents estimation section106estimates that the work contents are excavating when the angle of the bucket306is continuing to change and besides the delivery pressure of the hydraulic pump is equal to or higher than a fixed value. The work contents estimation section106outputs the work contents estimated in this manner to the operation information calculation section107. Thereafter, the processing advances to step204.

In step204, the operation information calculation section107integrates the work contents inputted from the work contents estimation section106and numerical information about positions, speeds, angles, and so forth relating to the work contents. For example, in a case where it is estimated that the work contents are traveling, the operation information calculation section107integrates the traveling speed with the work contents. On the other hand, in a case where it is estimated that the work contents are excavating, the operation information calculation section107integrates angle information indicated as the posture. A result of such integration is retained in the operation information calculation section107. Thereafter, the processing advances to step205.

In step205, the arrangement calculation section112constructs a three-dimensional shape based on the current posture of the excavator301. The arrangement calculation section112receives a three-dimensional basic shape of the excavator301retained by the shape information retaining section114as an input thereto and applies, to the three-dimensional basic shape, the work position inputted from the work position acquisition section104and the operation state inputted from the operation state acquisition section105. For example, the arrangement calculation section112updates the shape, on the basis of the information about the posture, such that the angle differences between the lower track body302and the upper swing structure303, and between the parts including the boom304, arm305, and bucket306in the three-dimensional shape become equivalent according to the direction of the lower track body302, the difference in swing angle between the lower track body302and the upper swing structure303, and the differences in arrangement angle between the portions of the upper swing structure303, boom304, arm305, and bucket306. The arrangement calculation section112outputs a result of the update to the arrangement state decision section115. Thereafter, the processing advances to step206.

In step206, the object sensing section113performs an object sensing process using images around the excavator being captured by the cameras501and502. An example of the object sensing process is depicted inFIG.5. In this example, the front camera501and the rear camera502are incorporated on the upper swing structure303. The images captured by the cameras501and502are such as a front camera image503and a rear camera image504, respectively. Since no object exists in the rear of the excavator301, only the ground surface and the sky are reflected in the rear camera image504. Since a person505exists in front of the excavator301, the person505is reflected in the front camera image503. Further, depending upon the state of the posture of the machine body, the arm305and the bucket306are reflected in the front camera image503.

An example of surrounding object sensing by the front camera image503is depicted inFIG.6. The front camera image503indicates an appearance at time ti−1 at an upper left portion inFIG.6and indicates an appearance at time ti after that at a lower left portion ofFIG.6. Here, the person505is approaching the excavator, and the image of the person505is moving in a downward direction in the front camera image503as time passes. If the difference between the front camera images503at time ti−1 and time ti is acquired, then such a result as depicted on the right side inFIG.6is obtained. In the image, a difference in image appears only at a portion at which the person505moves. This is referred to as difference area601on the image. In a case where a difference appears in this manner, the difference area601on the image is outputted to the arrangement state decision section115. Thereafter, the processing advances to step207.

In step207, the arrangement state decision section115performs projection transformation between a coordinate system of the camera images and a coordinate system representative of the shape of the excavator301. The camera images are represented in a two-dimensional coordinate system and the shape of the excavator301is represented in a three-dimensional coordinate system. The arrangement state decision section115performs projection transformation between them and calculates at which position the difference area601sensed in the camera images is positioned relative to the excavator. The shape of the excavator301has been updated by the process in step S205such that it becomes equivalent to that of the current posture of the excavator301. The shape of the excavator301simultaneously holds also an installation position and an imaging direction of the front camera501. Further, also a relative positional relation between the ground surface and the front camera501has been known on the basis of the information about the work position. Since the person505exists on the ground surface, it is possible to find it by projection transformation at which position the difference area601sensed in the front camera image503exists on the ground surface in the three-dimensional space. This state is depicted inFIG.7. By the projection transformation, the position at which a person701exists in the three-dimensional space at time ti is acquired. Thereafter, the processing advances to step208.

In step208, the arrangement state decision section115calculates an overlapping area between the components of the excavator301and the person701at time ti. For example, a state is supposed in which time further passes from the state ofFIG.7and the person701further moves. This state is depicted inFIG.8. The person701at time ti has changed its position by the movement and moved to a position in the proximity of the bucket306as depicted by a person801at time ti+1. The overlapping area in this case is found in the following manner. A range in which the person801at time ti+1 may possibly move thereafter is calculated as a person movement range803in the form of a three-dimensional area. The person movement range803indicates a range in which the person801may possibly move till time ti+2 at a speed at which the person can possibly move. Similarly, a range in which the arm305and the bucket306of the excavator301at time ti+1 may possibly move thereafter is calculated as an excavator movement range804in the form of a three-dimensional area. The excavator movement range804indicates a range in which, where the arm305and the bucket306move while accelerating to the maximum from the speed of movement thereof at time ti+1, they may possibly move till time ti+2. The overlapping area802is calculated as an area (indicated by slanting lines inFIG.8) in which the person movement range803and the excavator movement range804overlap with each other. Thereafter, the processing advances to step209.

In step209, the arrangement state decision section115decides whether or not an overlapping area802has appeared by the process in step208. In a case where an overlapping area802has appeared, the processing advances to step210, but in any other case, the processing advances to step211.

In step210, the operation information calculation section107adds the appeared overlapping area802to the operation information calculated in step204to produce new operation information. Thereafter, the processing advances to step211.

In step211, the operation information calculation section107records the operation information into the recording device109. Alternatively, the operation information calculation section107records the operation information into an external recording device through the communication device108. Thereafter, the processing advances to step212.

In step S212, it is decided whether or not the recording of the operation information is completed. In a case where recording of the operation information is completed by stopping of the excavator301or the like, the process upon operation information recording depicted on the left side inFIG.2is ended, but in any other case, the processing advances to step201. Upon operation information recording, such a process as described above is performed.

Now, a process when operation information recorded in the recording device109is verified is described. A flow of the process is described at a portion of the operation upon operation information verification depicted on the right side inFIG.2.

After operation upon operation information verification is started in the operation record analysis system for a construction machine according to the present embodiment, a process in step213is performed first.

In step213, the operation information calculation section107acquires operation information recorded in the recording device109. Operation information at which point of time is to be acquired is set by an input from a system user using the display device110. It is to be noted that the operation information to be acquired may be operation information at a certain point of time or may be a range designated from within operation information at a plurality of consecutive points of time. Where a range is designated, processes upon operation information verification are executed consecutively within the range. Thereafter, the processing advances to step214.

In step214, addition information added to the acquired operation information is sensed. In a case where no addition information is added, nothing is sensed. Thereafter, the processing advances to step215.

In step215, it is decided whether or not the acquired operation information has addition information added thereto. In a case where addition information is added, the processing advances to step216, but in any other case, the processing advances to step217.

In step216, the acquired operation information is added to an extraction candidate. Thereafter, the processing advances to step S217.

In step217, it is decided whether or not operation information to which the process described above is not applied as yet remains in the operation information designated by the display device110. In a case where such operation information remains, the processing advances to step213, but in any other case, the processing advances to step218.

In step218, the acquired operation information is displayed on the display device110. Thereafter, the processing advances to step219.

In step219, the operation information added to the extraction candidate is highlighted. Thereafter, the process is ended.

Here, an example of the operation information is depicted inFIG.9. Here, operation information at time ti and operation information at time ti+1 are depicted. At time ti, the work contents of the excavator301are “excavating,” and the position and the posture of the excavator301are recorded in the operation information. At this point of time, the overlapping area802does not appear as yet, and therefore, the addition information is set to “absent.” At time ti+1, the work contents of the excavator301are “excavating,” and the position and the posture of the excavator301are recorded as the operation information. At this pint of time, the person701exists at such a position as indicated as a person801at time ti+1 as a result of movement thereof, and as a result of the overlapping area calculation process, an overlapping area802is calculated. On the basis of this result, the addition information is set to “present” and the overlapping area802is added as addition information to the operation information.

Now, an example of the highlighting in step218is depicted in DIF.10. In this example, such a graph as depicted inFIG.10is displayed on the display screen of the display device110. In the graph, the axis of abscissa indicates transition of time. In the row of the work contents, a change in work contents associated with transition of time is indicated. In the row of the work position, a change in work position is indicated similarly. Here, although this figure indicates a graph displayed by one line for the convenience of illustration, actually, a graph that indicates a change in three-dimensional coordinates, a change in latitude and longitude, or the like is displayed. In the row of the posture, a change in posture of the excavator301is indicated. Also here, although a graph is displayed as one line for the convenience of illustration, a change in angle such as θ1, θ2, θ3, or θ4 is displayed by a graph. That the overlapping area802exists at time at which an overlapping area802added to operation information exists is indicated by a display image of a belt (indicated by slanting lines inFIG.10) and characters of “appearance of overlapping area.” Consequently, an approaching state of the excavator301and the excavator301, which may possibly become a factor of operation rate decrease in the excavator301, is indicated clearly in display images of operation information at a plurality of points of time.

In the present embodiment, in the operation record analysis system for a construction machine that includes the machine body301including the front work implement having at least the arm305and the bucket306, the machine body position sensor101that senses a position of the machine body301, the machine body posture sensor102that senses a posture of the machine body301, the operation state sensor103that senses an operation state of the machine body301, and the controller100that calculates a work position of the machine body301on the basis of information from the machine body position sensor101and the machine body posture sensor102, estimates work contents of the machine body301on the basis of the work position and the operation state and outputs operation information including the work position and the work contents, the operation record analysis system includes the object sensor111that senses the object505,701,801existing around the machine body301, and the controller100calculates a position of the object505,701,801on the basis of the information from the object sensor111, decides on the basis of the information from the machine body position sensor101and the machine body posture sensor102whether or not the machine body301and the object505,701,801are close to each other, and adds a result of the decision to the operation information and outputs the operation information.

According to the present embodiment configured in such a manner as described above, since a decision result of whether or not the machine body301and the person505,701or801existing around the machine body301are close to each other is included in the operation information, a system user can quickly extract work contents when the machine body and an object existing the machine body are close to each other from within a huge amount of operation information. This makes it possible to improve the verification efficiency of the work situation of the excavator301.

Further, the object sensor111is the camera501,502that captures an image around the machine body301, and the controller100decides that the machine body301and the object801are close to each other in a case where at least part of the machine body movement range804in which the machine body301may possibly move in the image capturing interval of the camera501,502and at least part of the object movement range803in which the object801may possibly move in the image capturing interval overlap with each other. This makes it possible to decide, on the basis of the moving speeds of the excavator301and the object801, whether or not the excavator301and the object801come close to each other.

Further, the operation record analysis system for a construction machine in the present embodiment further includes the communication device108that transmits the operation information, the recording device109that records the operation information, and the display device110that displays the operation information. This improves the degree of freedom of the place at which the operation information is to be confirmed.

Second Embodiment

The present embodiment described below is directed to an example in which, where an overlapping area802appears, a factor that causes the overlapping area802is displayed.

An example of the front camera image503at time ti+1 in the present embodiment is depicted at an upper left portion inFIG.11. Here, a difference area1101at time t1+1 appears as a result of a movement of the person801at time ti+1. In a case where it is decided, by a process after this, that an overlapping area802has appeared, the difference area1101at time ti+1 is recorded into the recording device109. Since, where an overlapping area802appears, also the difference area1101exists, in such a situation as just described, the difference area1101is recorded into the recording device109without fail.

An example of a display image on the screen of the display device110in this example is depicted on the right side inFIG.11. At time at which the overlapping area802exists, the difference area1101recorded in the recording device109is displayed together with a highlighted image of appearance of an overlapping area.

In the present embodiment, the controller100places, when deciding that the excavator301and the object801are close to each other, the difference area1101indicative of a range in which the object801has moved in an imaging interval of the cameras501and502from within the images captured by the cameras501and502at the time of decision, into the operation information and outputs the operation information.

According to the present embodiment configured in such a manner as described above, when an overlapping area802appears, the difference area1101that has been made a decision criterion of the appearance can be displayed as an image, and the system user can visually recognize what object has caused the overlapping area802to be appeared. The system user can grasp the significance of analysis of the overlapping area802on the basis of the visually recognized contents and execute subsequent countermeasures rapidly.

Third Embodiment

The present embodiment described below is directed to an example in which, in a case where an overlapping area802appears, a factor that causes the overlapping area802to be appeared is displayed including also some other state.

An example of the front camera image503at time ti+1 in the present embodiment is depicted at an upper left portion inFIG.12. Here, a difference area1101at time ti+1 appears as a result of a movement of the person801at time ti+1. In a case where it is decided by a subsequent process that the overlapping area802has appeared, the difference area1101at time ti+1 and a camera image1201at time ti+1 are recorded into the recording device109. Since camera images are used for calculation of the overlapping area802and the difference area1101, in such a situation as just described, the camera image1201and the difference area1101are recorded into the recording device109without fail.

An example of a display image on the screen of the display device110in the present example is depicted on the right side inFIG.12. At time at which the overlapping area802exists, the camera image1201and the difference area1101recorded in the recording device109are displayed together with a highlighted image of appearance of an overlapping area.

In the present embodiment, the controller100places, when deciding that the excavator301and the object801are close to each other, information about the difference area1101indicative of a range in which the object801has moved in an image capturing interval of the cameras501and502in an image captured by the camera501and502at the time of decision and the images captured by the cameras501and502at the time of decision, into the operation information and outputs the resulting operation information.

According to the present embodiment configured in such a manner as described above, when an overlapping area802appears, together with the difference area1101that has been made a decision criterion of the appearance, a situation around the difference area1101can be displayed as an image. Thus, the system user can visually recognize in what situation the overlapping area802has appeared. The system user can grasp the significance of analysis of the overlapping area802on the basis of the contents of the visual recognition and execute subsequent countermeasures rapidly.

Fourth Embodiment

The present embodiment described below is directed to an example in which object sensing is performed using a LiDAR (Light Detection and Ranging).

A configuration of the excavator301in the present embodiment is depicted inFIG.13. In this configuration, the cameras501and502in the first embodiment are replaced with LiDARs. In this example, a front LiDAR1301whose sensing target is the front of the excavator301and a rear LiDAR1302whose sensing target is the rear of the excavator301are installed.

A front LiDAR output1303and a rear LiDAR output1304are depicted on the right side inFIG.13. The person505, arm305, and bucket306exist in the sensing range of the front LiDAR1301, and thus waveforms are generated on the graph of the front LiDAR output1303. The waveforms on the left side originate from the person505, and the waveforms on the right side originate from the arm305and the bucket306. It is to be noted that a target other than the ground surface does not exist in the sensing range of the rear LiDAR1302, and except this, a characteristic waveform does not exist on the rear LiDAR output1304.

The object sensing section113performs object sensing using such LiDAR outputs. As a result, the person505, arm305, and bucket306are sensed. On the basis of this result, the arrangement state decision section115excludes the waves originating from the arm305and the bucket306from comparison with the three-dimensional shape of the excavator301outputted from the arrangement calculation section112to acquire a waveform originating from the person505. Calculation of the overlapping area802is performed using a result of the acquisition.

In the present embodiment, the object sensor111is the LiDARs1301and1302that acquire information about a distance to a target, and the controller100acquires the information about the distance to the object505by excluding the information about the distance to the machine body301from the information about the distance to the target acquired by the LiDARs1301and1302.

According to the present embodiment configured in such a manner as described above, even in a case where the object sensor111is configured from the LiDARs1301and1302, a decision result of whether or not the excavator301and the object505existing around the excavator301are close to each other is recorded together with the operation information similarly as in the first embodiment. Therefore, the system user can rapidly extract, from within a huge amount of operation information, work contents when the excavator301and the object505existing around the excavator301come close to each other can be extracted rapidly. Consequently, the verification efficiency of the work situation of the excavator301can be improved.

Fifth Embodiment

The present embodiment described below is directed to an example in which object sensing is performed using LiDARs and camera images upon object sensing are displayed.

A configuration of the excavator301in the present embodiment is depicted inFIG.14. In this configuration, LiDARs are installed in addition to the cameras501and502and the outputs of the LiDARs are inputted to the object sensing section113. A detection process performed using the front LiDAR1301and the rear LiDAR1302is similar to that in the fourth embodiment. In the present embodiment, the front camera501and the rear camera502further capture images around the excavator301. An object sensing result using the LiDAR outputs is superposed on the contents of the images captured by the cameras501and502, and a result of this is recorded into the recording device109. A display image of the display device110in this case is such as depicted inFIG.15. At time at which appearance of an overlapping area802is sensed, the images recorded in the recording device109are displayed together.

In the present embodiment, the object sensor111is the LiDARs1301and1302that acquire information about a distance to a target, and the controller100acquires the information about the distance to the object505by excluding the information about the distance to the excavator301from the information about the distance to the target acquired by the LiDARs1301and1302. The operation record analysis system for a construction machine according to the present embodiment further includes the cameras501and502that capture images around the excavator301, and the controller100places, in a case where it decides that the excavator301and the object505are close to each other, the images captured by the front cameras501and502at the time of the decision, into the operation information and outputs the resulting operation information.

According to the present embodiment configured in such a manner as described above, when an overlapping area802appears, it is possible to display, together with an object sensing result that has been made a criterion of decision of the appearance, a situation around the object as an image, and it is possible for the system user to visually recognize in what situation the overlapping area802appears. The system user can grasp the significance of analysis of the overlapping area802on the basis of the visually recognized contents and execute subsequent countermeasures rapidly.

Although the embodiments of the present invention have been described in detail, the present invention is not restricted to the embodiments described above and include various modifications. For example, the embodiments described above have been described in detail in order to describe the present invention so as to facilitate understandings of the present invention and the present invention is not necessarily restricted to such embodiments that include all configurations described above. Also it is possible to add part of the configurations of a certain embodiment to the configurations of a different embodiment, or also it possible to delete part of the configurations of a certain embodiment or replace part of the configurations of a certain embodiment with part of a different embodiment.

DESCRIPTION OF REFERENCE CHARACTERS

100: Controller101: Machine body position sensor102: Machine body posture sensor103: Operation state sensor104: Work position acquisition section105: Operation state acquisition section106: Work contents estimation section107: Operation information calculation section108: Communication device109: Recording device110: Display device111: Object sensor112: Arrangement calculation section113: Object sensing section114: Shape information retaining section115: Arrangement state decision section201to219: Step301: Excavator (machine body)302: Lower track body303: Upper swing structure304: Boom305: Arm306: Bucket501: Front camera (object sensor)502: Rear camera (object sensor)503: Front camera image504: Rear camera image505: Person (object)601: Difference area701: Person (object) at time ti801: Person (object) at time ti+1802: Overlapping area803: Person movement range (object movement range)804: Excavator movement range (machine body movement range)1101: Difference area at time ti+11201: Camera image at time ti+11301: Front LiDAR (object sensor)1302: Rear LiDAR (object sensor)1303: Front LiDAR output1340: Rear LiDAR output