Patent Description:
A terminal device for a remote monitoring assisting system has been proposed for sharing information with sufficient accuracy between a worker who is performing patrol inspection in a plant and a person who is waiting outside a work site (for example, see Patent Literature <NUM>). The terminal device comprises a video input unit configured to input video data of the site, an input operation selection unit of a pen or mouse type, a detection unit configured to detect presence/absence of new video acquisition, a communication control unit configured to wirelessly transmit and receive data to/from outside, and an input/output screen display unit configured to display an input screen to input predetermined data.

However, when a picked-up image acquired through an image pickup device is displayed on an output interface (image display device) constituting a remote operating device and a plurality of work machines are shown in the picked-up image, there is a possibility that an operator erroneously recognizes a work machine which is an operation target and surrounding work machines which are not the operation target, in the displayed image. In particular, in a case where the work machines are respectively operated by a plurality of operators, the own machine which is the operation target of one operator and the other machines which are the operation targets of the other operators need to be recognized by the operators; however, it is difficult to share the same recognition among the plurality of operators. It is especially notable in the case where the plurality of work machines of the same model of the same maker are simultaneously displayed on the output interface. In this case, there is a possibility that the operator may operate the work machine using the remote operating device based on erroneous recognition, which may be improper from a viewpoint of target work execution.

Therefore, it is an object of the present invention to provide a server and a system capable of enabling an operator of a remote operating device to recognize which remote operating device remotely operates a work machine displayed on an output interface constituting the remote operating device.

The present invention relates to a work assisting server for assisting a remote operation of a work machine using a remote operating device.

The work assisting server of the present invention is according to claim <NUM>.

According to the work assisting server of the configuration, the work environment image indicating a situation of the work site acquired through the image pickup device is outputted on the output interface constituting each of the plurality of remote operating devices. According to the label image request accompanied with the identifier from the plurality of remote operating devices, whether or not the real space position of the linked work machine identified by the identifier is included in the real space image pickup range of the image pickup device is determined. When the result of the determination is positive, the label image indicating the presence of the linked work machine is outputted on the output interface of each of the plurality of remote operating devices. Thus, each of a plurality of operators of the plurality of remote operating devices can be made to recognize the presence of at least one linked work machine linked with at least one remote operating device in the work environment image.

Therefore, the plurality of operators can be enabled to recognize in common that the linked work machine linked with one remote operating device or the linked work machines linked with the remote operating devices are present in the work environment images outputted on the output interfaces constituting the remote operating devices.

The invention is defined in the appended independent claims. Preferred embodiments are defined in the appended dependent claims.

A work assisting system as one embodiment of the present invention illustrated in <FIG> is constituted of a work assisting server <NUM> and a plurality of remote operating devices <NUM> for remotely operating a plurality of work machines <NUM>. The work assisting server <NUM>, the remote operating devices <NUM> and the work machines <NUM> are configured to perform network communication with each other. The plurality of remote operating devices <NUM> may be configured to perform communication with each other through a network different from a communication network with the work assisting server <NUM>.

The work assisting server <NUM> comprises a database <NUM>, a first assisting processing element <NUM> and a second assisting processing element <NUM>. The database <NUM> stores and holds, in addition to a position and/or a track of each of the plurality of work machines <NUM>, a picked-up image obtained by a camera provided in each of the work machines <NUM>, a work environment image and a route guidance image provided to an operator operating each of the work machines <NUM>, and the like. The database <NUM> may be constituted of a database server different from the work assisting server <NUM>. Each assisting processing element is constituted of an arithmetic processing unit (a single-core processor, a multi-core processor or a processor core constituting it), reads required data and software from a storage device such as a memory and executes arithmetic processing to be described later according to the software with the data as a processing target.

The work machine <NUM> comprises an actual machine controller <NUM>, an actual machine input interface <NUM>, an actual machine output interface <NUM>, an actuation mechanism <NUM> and a positioning device <NUM>. The actual machine controller <NUM> is constituted of an arithmetic processing unit (a single-core processor, a multi-core processor or a processor core constituting it), reads required data and software from a storage device such as a memory and executes arithmetic processing according to the software with the data as a processing target.

The work machine <NUM> is a crawler shovel (construction machine), for example, and comprises, as illustrated in <FIG>, a crawler type lower traveling body <NUM> and an upper turning body <NUM> loaded on the lower traveling body <NUM> through a turning mechanism <NUM> so as to be turned. A cab (driving room) <NUM> is provided on a front left side part of the upper turning body <NUM>. A work attachment <NUM> is provided on a front center part of the upper turning body <NUM>.

The actual machine input interface <NUM> comprises an actual machine operation mechanism <NUM> and an actual machine image pickup device <NUM>. The actual machine operation mechanism <NUM> comprises a plurality of operation levers arranged similarly to a remote operation mechanism <NUM>, around a seat arranged inside the cab <NUM>. A driving mechanism or a robot which receives a signal according to an operation mode of a remote operation lever and moves an actual machine operation lever based on the received signal is provided in the cab <NUM>. The actual machine image pickup device <NUM> is installed inside the cab <NUM>, for example, and picks up an image of an environment including at least a part of the actuation mechanism <NUM> over a front window of the cab <NUM>.

The actual machine output interface <NUM> comprises an actual machine radio communication device <NUM>.

The work attachment <NUM> as the actuation mechanism comprises a boom <NUM> mounted on the upper turning body <NUM> so as to be hoisted, an arm <NUM> turnably connected to a distal end of the boom <NUM>, and a bucket <NUM> turnably connected to a distal end of the arm <NUM>. On the work attachment <NUM>, a boom cylinder <NUM>, an arm cylinder <NUM> and a bucket cylinder <NUM> each constituted of extendable hydraulic cylinder are mounted.

The boom cylinder <NUM> is interposed between the boom <NUM> and the upper turning body <NUM> so as to be extended and contracted by receiving supply of hydraulic oil to turn the boom <NUM> in a hoisting direction. The arm cylinder <NUM> is interposed between the arm <NUM> and the boom <NUM> so as to be extended and contracted by receiving supply of hydraulic oil to turn the arm <NUM> with respect to the boom <NUM> around a horizontal axis. The bucket cylinder <NUM> is interposed between the bucket <NUM> and the arm <NUM> so as to be extended and contracted by receiving supply of hydraulic oil to turn the bucket <NUM> with respect to the arm <NUM> around the horizontal axis.

The remote operating device <NUM> as a client comprises a remote controller <NUM>, a remote input interface <NUM> and a remote output interface <NUM>. The remote controller <NUM> is constituted of an arithmetic processing unit (a single-core processor, a multi-core processor or a processor core constituting it), reads required data and software from a storage device such as a memory and executes arithmetic processing according to the software with the data as a processing target. The remote input interface <NUM> comprises the remote operation mechanism <NUM>. The remote output interface <NUM> comprises an image output device <NUM> and a remote radio communication device <NUM>.

The client may be constituted of a mobile terminal such as a smartphone or a tablet terminal or a wearable terminal such as a pair of VR goggles, which are linked with the remote operating device <NUM> or have an intercommunication function. The mobile terminal or the wearable terminal may have a communication function with the work assisting server <NUM>.

The remote operation mechanism <NUM> includes a traveling operation device, a turning operation device, a boom operation device, an arm operation device and a bucket operation device. Each operation device includes an operation lever which receives a turning operation. The operation lever (traveling lever) of the traveling operation device is operated to move the lower traveling body <NUM> of the work machine <NUM>. The traveling lever may serve also as a traveling pedal. For example, the traveling pedal fixed to a base part or a lower end part of the traveling lever may be provided. The operation lever (turning lever) of the turning operation device is operated to move a hydraulic type turning motor constituting the turning mechanism <NUM> of the work machine <NUM>. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder <NUM> of the work machine <NUM>. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder <NUM> of the work machine <NUM>. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder <NUM> of the work machine <NUM>.

Each operation lever constituting the remote operation mechanism <NUM> is arranged around a seat St for an operator to sit, as illustrated in <FIG>, for example. The seat St is in a form like a high back chair with armrests; however, it may be in any form that a remote operator OP2 can sit, such as a form like a low back chair without a headrest or a form like a chair without a backrest.

A pair of left and right traveling levers <NUM> corresponding to left and right crawlers are arranged laterally side by side at a front of the seat St. One operation lever may serve as a plurality of operation levers. For example, a right side operation lever <NUM> provided on the front of a right side frame of the seat St illustrated in <FIG> may function as the boom lever when operated in front and back directions and may function as the bucket lever when operated in left and right directions. Similarly, a left side operation lever <NUM> provided on the front of a left side frame of the seat St illustrated in <FIG> may function as the arm lever when operated in the front and back directions and may function as the turning lever when operated in the left and right directions. A lever pattern may be arbitrarily changed by an operation instruction of the operator.

The image output device <NUM> is constituted of a diagonally right front image output device <NUM>, a front image output device <NUM> and a diagonally left front image output device <NUM> arranged at the diagonally right front, front and diagonally left front of the seat St, respectively, as illustrated in <FIG>, for example. The image output devices <NUM>-<NUM> may further comprise a speaker (voice output device).

The functions of the work assisting system of the configuration described above will be explained using flowcharts illustrated in <FIG> and <FIG>. In the flowcharts, a block "C•" is used to simplify description, means transmission and/or reception of data and means a conditional branch of executing processing in a branch direction on the condition that the data is transmitted and/or received.

In the remote operating device <NUM>, presence/absence of a first specifying operation by the operator through the remote input interface <NUM> is determined (<FIG>/STEP200). The "first specifying operation" is an operation of the remote input interface <NUM> for making the work machine <NUM> that the operator intends to remotely operate selectable and is the operation of tapping or the like through the remote input interface <NUM>, for example. When the result of the determination is negative (<FIG>/STEP200. NO), a series of processing is ended.

On the other hand, when the result of the determination is positive (<FIG>/STEP200. YES), the remote operating device <NUM> acquires construction machine candidates that can be a target of the remote operation by the remote operating device <NUM>, through the communication network, and causes various information such as positions, models and operation states of the construction machines to be displayed on the remote output interface <NUM> together with identifiers of the construction machines (<FIG>/STEP201).

Then, in the remote operating device <NUM>, the presence/absence of a second specifying operation by the operator through the remote input interface <NUM> is determined (<FIG>/STEP202). The "second specifying operation" is an operation for specifying the construction machine to be the target of the remote operation out of the construction machine candidates that can be the target of the remote operation, and is the operation of tapping or the like through the remote input interface <NUM>, for example. When the result of the determination is negative (<FIG>/STEP202. NO), a series of processing is ended. On the other hand, when the result of the determination is positive (<FIG>/STEP202. YES), a work environment image request is transmitted to the work assisting server <NUM> through the remote radio communication device <NUM> (<FIG>/STEP203). The work environment image request includes at least one of the identifier of the remote operating device <NUM> and the identifier of the operator.

In the work assisting server <NUM>, when the work environment image request is received, the work environment image request is transmitted to the work machine <NUM> corresponding to the work environment image request by the first assisting processing element <NUM> (<FIG>/C10).

In the work machine <NUM>, when the work environment image request is received through the actual machine radio communication device <NUM> (<FIG>/C41), the actual machine controller <NUM> acquires a picked-up image through the actual machine image pickup device <NUM> (<FIG>/STEP402). By the actual machine controller <NUM>, picked-up image data indicating the picked-up image is transmitted to the remote operating device <NUM> through the actual machine radio communication device <NUM> (<FIG>/STEP404).

In the work assisting server <NUM>, when the picked-up image data is received (<FIG>/C11), first work environment image data (data indicating whole or a part of the picked-up image itself or a simulated first work environment image generated based on it) according to the picked-up image data is transmitted to the remote operating device <NUM> (<FIG>/STEP112).

In the remote operating device <NUM>, when the first work environment image data is received through the remote radio communication device <NUM> (<FIG>/C20), first work environment image according to the first work environment image data is outputted on the image output device <NUM> (<FIG>/STEP204).

Thus, as illustrated in <FIG>, for example, the work environment image including the boom <NUM>, the arm <NUM>, the bucket <NUM> and the arm cylinder <NUM> which are parts of the work attachment <NUM> as the actuation mechanism is displayed on the image output device <NUM>.

In addition, based on the picked-up image acquired through an image pickup device C (see <FIG>) installed at a work site, as illustrated in <FIG>, for example, an image indicating a situation of the work site in which actual machine images Q1-Q4 of the plurality (four) of work machines <NUM> are present may be additionally or alternatively outputted on the image output device <NUM> as the first work environment image.

Further, as illustrated in <FIG>, for example, a bird's-eye picked-up image or a bird's-eye map indicating a general situation of the work site and indicating the actual machine images or icons Q1-Q4 indicating the work machines <NUM> present at the work site may be additionally or alternatively outputted on the image output device <NUM> as the first work environment image. The bird's-eye picked-up image may be acquired through the image pickup device loaded on an unmanned aircraft or the image pickup device installed at a structure such as a pole at the work site, for example. An image pickup part and a view angle of the picked-up image as the first work environment image may be arbitrarily changed. The bird's-eye map may be generated based on the bird's-eye picked-up image.

In the remote operating device <NUM>, an operation mode of the remote operation mechanism <NUM> is recognized by the remote controller <NUM> (<FIG>/STEP206), a signal according to the operation mode of the remote operation lever is transmitted to the work machine <NUM>, and it becomes ready to drive the actual machine operation lever based on the transmitted signal. Then, in the remote operating device <NUM>, the presence/absence of a third specifying operation by the operator through the remote input interface <NUM> is determined (<FIG>/STEP208). The "third specifying operation" is an operation for making the work machine <NUM> selected as the target of the remote operation operable by the remote operating device <NUM>, and is the operation of tapping or the like through the remote input interface <NUM>, for example. By the "third specifying operation", the remote operation of the work machine <NUM> by the remote operating device <NUM> is started. Then, a remote operation command according to the operation mode can be transmitted to the work assisting server <NUM> through the remote radio communication device <NUM> (<FIG>/STEP210).

In the work assisting server <NUM>, when the remote operation command is received, the remote operation command is transmitted to the work machine <NUM> by the first assisting processing element <NUM> (<FIG>/C12).

In the work machine <NUM>, when the operation command is received through the actual machine radio communication device <NUM> (<FIG>/C42), operations of the work attachment <NUM> or the like are controlled by the actual machine controller <NUM> (<FIG>/STEP406). For example, the work of scooping soil in front of the work machine <NUM> by the bucket <NUM>, turning the upper turning body <NUM> and then dropping the soil from the bucket <NUM> is executed.

In the remote operating device <NUM> (first client), the presence/absence of a fourth specifying operation by the operator through the remote input interface <NUM> is determined (<FIG>/STEP210). The "fourth specifying operation" is an operation for sharing the same recognition among the plurality of operators about which operator is to operate each of the own machine that is the operation target of one operator and the other machines that are the operation targets of the other operators, and is the operation of tapping or the like through the remote input interface <NUM>, for example. For example, the operator of the first client (first remote operating device <NUM>) requests intercommunication with the operator of a second client (second remote operating device <NUM>), and the operator of the second client (second remote operating device <NUM>) accepts the request; then, the forth specifying operation is triggered.

Then, a label image request can be transmitted to the work assisting server <NUM> through the remote radio communication device <NUM> (<FIG>/STEP212). The operation of selecting the target actual machine for which the operator wants to share the recognition corresponds to the label image request. For example, the operator selects the target actual machine for which the operator wants to share the recognition based on the various information such as the positions, models and operation states of the construction machines together with the identifiers of the construction machines, which are displayed on the remote output interface <NUM>. When displaying target actual machine candidates, the remote output interface <NUM> at least displays the work machine <NUM> which is the operation target of the first client and the work machine <NUM> which is the operation target of the operator of the second client (second remote operating device <NUM>) performing the intercommunication, and emphatic display makes it easy to share the recognition of the own machine and the other machine performing the communication. The label image request includes, in addition to an image identifier for identifying the first work environment image outputted on the image output device <NUM>, at least one of a first identifier for identifying the first client or its operator and a second identifier for identifying the second client or its operator.

In the work assisting server <NUM>, when the label image request is received (<FIG>/C13), the real space position of the target actual machine is recognized by the first assisting processing element <NUM>, by retrieval from the database <NUM> based on the first identifier (<FIG>/STEP122).

When the intercommunication of the remote operating device <NUM> (client) and the work machine <NUM> is established, the identifier of the remote operating device <NUM> (or its operator), the identifier of the work machine <NUM> and the real space position of the work machine <NUM> are registered in the database <NUM> in association with one another. The real space position of the work machine <NUM> is measured by the positioning device <NUM> utilizing a GPS and an acceleration sensor, as needed, which are loaded on the work machine <NUM>. By the work machine <NUM> transmitting the real space position or the time sequence thereof to the work assisting server <NUM>, the real space position of the work machine <NUM> registered in the database <NUM> is updated. Thus, the real space position of the target actual machine can be recognized based on each of the first identifier and the second identifier.

Subsequently, by the first assisting processing element <NUM>, whether or not the target actual machine is included in the first work environment image identified by the image identifier included in the label image request is determined (<FIG>/STEP123). The case where the target actual machine is included in the first work environment image is the case where the target actual machine <NUM> is included in the first work environment image (the work environment image acquired through the image pickup device <NUM> loaded on the work machine <NUM> different from the target actual machine), as illustrated in each of <FIG>.

Specifically, firstly, the shooting range of the first work environment image is identified. When the first work environment image is picked up by the image pickup device <NUM> of the work machine <NUM>, the real space position of the work machine <NUM> and a shooting direction (up-down and left-right attaching directions to the work machine <NUM>) of the image pickup device <NUM> are acquired by the positioning device <NUM>. When the first work environment image is by the image pickup device C installed at the work site, as illustrated in <FIG>, the real space position of the image pickup device C and the shooting direction of the image pickup device C are acquired. Further, by acquiring information regarding the view angle and scaling of the picked-up image, the shooting range of the first work environment image in a real space coordinate system is identified.

Then, whether or not the target actual machine is included in the shooting range of the first work environment image in the real space coordinate system is determined based on the real space position of the work machine <NUM> registered in the database <NUM> (<FIG>/STEP123).

When the result of the determination is positive (<FIG>/STEP123. YES), the position of the target actual machine in the real space coordinate system is transformed to the position in a first work environment image coordinate system. Then, image analysis processing is executed on the first work environment image and whether or not an actual machine image Qi indicating the work machine <NUM> is extracted at the position of the target actual machine in the first work environment image coordinate system is determined (<FIG>/STEP124).

When the result of the determination is positive (<FIG>/STEP124. YES), a first label image indicating that the actual machine image Qi is the target actual machine (the own machine or the other machine) is generated by the second assisting processing element <NUM>, and data indicating the first label image is transmitted to the first client by the second assisting processing element <NUM> (<FIG>/STEP125). Then, a specifying notice indicating that the first label image is superimposed on the first work environment image is transmitted (<FIG>/STEP132).

In the remote operating device <NUM> (first client), when first label image data is received through the remote radio communication device <NUM> (<FIG>/C21), the first label image which emphatically displays the presence of the target actual machine in the first work environment image is outputted on the image output device <NUM> constituting the output interface <NUM> (<FIG>/STEP214). Thus, as illustrated in <FIG>, for example, the result of superimposing a texture image T on the first work environment image in an area surrounded by a set EC of edge points in the actual machine image Qi is outputted as the first label image. A color of the texture image T is changeable according to preference of the operator. In addition, as illustrated in <FIG>, for example, the result of superimposing a mark image M on the first work environment image near the actual machine image Qi is outputted as the first label image. For the mark image M, a symbol (character) according to preference of the operator can be set.

The case where the result of the determination is negative (<FIG>/STEP124. NO) is the case where a work machine <NUM>-<NUM> is not facing the target real machine <NUM>-<NUM>, as illustrated in <FIG>, or the case where it is blocked by an obstacle such as the other work machine, for example.

When the target actual machine is not included in the shooting range of the first work environment image in the real space coordinate system (<FIG>/STEP123. NO) or when the actual machine image Qi indicating the work machine <NUM> is not extracted at the position of the target actual machine in the first work environment image coordinate system (<FIG>/STEP124. NO), whether or not the target actual machine is included in the shooting range of a second work environment image in the real space coordinate system is determined (<FIG>/STEP126). Here, the second work environment image is a work environment image provided for the second client and is a work environment image of the work machine <NUM> which is the target of the remote operation by the operator of the second client. The second work environment image for the second client is similar to the first work environment image for the first client, and the description of a generation method of the image is omitted.

When the result of the determination is positive (<FIG>/STEP126. YES), the position of the target actual machine in the real space coordinate system is transformed to the position in a second work environment image coordinate system. Then, the image analysis processing is executed on the second work environment image and whether or not the actual machine image Qi indicating the work machine <NUM> is extracted at the position of the target actual machine in the second work environment image coordinate system is determined (<FIG>/STEP127).

When the result of the determination is positive (<FIG>/STEP127. YES), a second label image indicating that the actual machine image Qi is the target actual machine (the own machine or the other machine) is generated by the second assisting processing element <NUM>, and data indicating the second label image is transmitted to the first client by the second assisting processing element <NUM> (<FIG>/STEP128). Then, a specifying notice indicating that the second label image is superimposed on the second work environment image is transmitted (<FIG>/STEP132). In the remote operating device <NUM>, when the second label image data is received through the remote radio communication device <NUM> (<FIG>/C22), the second label image which emphatically displays the presence of the target actual machine in the second work environment image is outputted instead of or in addition to the second work environment image on the image output device <NUM> constituting the output interface <NUM> (<FIG>/STEP216). Thus, as illustrated in <FIG>, for example, the result of superimposing a texture image T on the second work environment image in an area surrounded by a set EC of edge points in the actual machine image Qi is outputted as the second label image. The color of the texture image T is changeable according to preference of the operator. In addition, as illustrated in <FIG>, for example, the result of superimposing a mark image M on the second work environment image near the actual machine image Qi is outputted as the second label image. For the mark image M, a symbol (character) according to preference of the operator can be set.

When the target actual machine is not included in the shooting range of the second work environment image in the real space coordinate system (<FIG>/STEP126. NO) or when the actual machine image Qi indicating the work machine <NUM> is not extracted at the position of the target actual machine in the second work environment image coordinate system (<FIG>/STEP127. NO), whether or not the target actual machine is included in the shooting range of a third work environment image in the real space coordinate system is determined (<FIG>/STEP129). Here, the third work environment image is a work environment image not provided for the first client and the second client intercommunicating with the first client. The third work environment image is a work environment image provided for the second client not intercommunicating with the first client or a work environment image that can be provided for the first client and the second client intercommunicating with the first client.

When the result of the determination is positive (<FIG>/STEP129. YES), the position of the target actual machine in the real space coordinate system is transformed to the position in a third work environment image coordinate system. Then, the image analysis processing is executed on the third work environment image and whether or not the actual machine image Qi indicating the work machine <NUM> is extracted at the position of the target actual machine in the third work environment image coordinate system is determined (<FIG>/STEP130). When the result of the determination is positive (<FIG>/STEP130. YES), a third label image indicating that the actual machine image Qi is the target actual machine (the own machine or the other machine) is generated by the second assisting processing element <NUM>, and data indicating the third label image is transmitted to the first client by the second assisting processing element <NUM> (<FIG>/STEP131). Then, a specifying notice indicating that the third label image is superimposed on the third work environment image is transmitted (<FIG>/STEP132). In the remote operating device <NUM>, when third label image data is received through the remote radio communication device <NUM> (<FIG>/C23), the third label image which emphatically displays the presence of the target actual machine in the third work environment image is outputted instead of or in addition to the third work environment image on the image output device <NUM> constituting the output interface <NUM> (<FIG>/STEP218). Thus, as illustrated in <FIG>, for example, the result of superimposing a texture image T on the second work environment image in an area surrounded by a set EC of edge points in the actual machine image Qi is outputted as the second label image. The color of the texture image T is changeable according to preference of the operator. In addition, as illustrated in <FIG>, for example, the result of superimposing a mark image M on the third work environment image near the actual machine image Qi is outputted as the third label image. For the mark image M, a symbol (character) according to preference of the operator can be set.

When the target actual machine is not included in the shooting range of the third work environment image in the real space coordinate system (<FIG>/STEP129. NO) or when the actual machine image Qi indicating the work machine <NUM> is not extracted at the position of the target actual machine in the third work environment image coordinate system (<FIG>/STEP130. NO), a specifying notice indicating that the actual machine image corresponding to the target actual machine is not present in the first to third work environment images is transmitted to the remote operating device <NUM> (<FIG>/STEP132).

In the remote operating device <NUM>, when the specifying notice is received through the remote radio communication device <NUM> (<FIG>/C24), the specifying notice is displayed on the image output device <NUM> constituting the output interface <NUM>. The specifying notice notifies information on which of the first to third work environment images includes the target actual machine or information on that the target actual machine is not included in any of the first to third work environment images.

In the remote operating device <NUM>, the image output device <NUM> may have a function of switching between screens of the first to third work environment images whether the target actual machine is included in each of the shooting range or not.

The texture image T and the mark image M are in common between the first client and the second client performing intercommunication. For example, when a red texture image T is outputted as the first label image for the actual machine image Qi, on the image output device <NUM> of the first client, the texture image T of the actual machine image Qi is also red on the image output device <NUM> of the second client.

According to the work assisting system of the configuration and the work assisting server <NUM> constituting it, the first work environment image indicating the situation of the work site acquired through the image pickup device (for example, the image pickup device <NUM> loaded on the work machine <NUM>) is outputted on the output interface <NUM> constituting each of the plurality of remote operating devices <NUM> (see <FIG>/STEP204 and <FIG>). According to the label image request accompanied with the identifier from the first remote operating device <NUM>, whether or not the actual machine image Qi corresponding to the linked work machine identified by the identifier is included in the first work environment image is determined (see <FIG>/STEP123, <FIG>/STEP124). When the actual machine image Qi corresponds to the linked work machine, the first label image is outputted on the output interface <NUM> of the remote operating device <NUM> which is at least one of the first and second remote operating devices (see <FIG>/STEP214, <FIG>). The "first label image" indicates that the actual machine image Qi is the linked work machine.

Thus, by the operator of the first client (first remote operating device <NUM>) transmitting the label image request accompanied with the first identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, whether or not any of the actual machine images Qi included in the first work environment image outputted on the output interface <NUM> of the first client corresponds to the own machine which is the operation target by the first client or the operator identified by the first identifier can be checked. Further, by the operator of the first client transmitting the label image request accompanied with the second identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, whether or not any of the actual machine images Qi included in the first work environment image outputted on the output interface <NUM> of the first client corresponds to the other machine which is the operation target by the second client or its operator identified by the second identifier can be checked.

Similarly, by the operator of the first client (first remote operating device <NUM>) transmitting the label image request accompanied with the first identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, the operator of the second client can be enabled to check whether or not any of the actual machine images Qi included in the first work environment image outputted on the output interface <NUM> of the second client (second remote operating device <NUM>) corresponds to the other machine operated by the first client or the operator thereof identified by the first identifier. Further, by the operator of the first client transmitting the label image request accompanied with the second identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, the operator of the second client can be enabled to check whether or not any of the actual machine images Qi included in the first work environment image outputted on the output interface <NUM> of the second client corresponds to the own machine which is the operation target by the second client or the operator thereof identified by the second identifier.

Thus, the plurality of operators share the recognition of which remote operating device <NUM> remotely operates a work machine <NUM> shown in the first work environment image outputted on the output interface constituting the remote operating device.

Further, when none of one or more actual machine images Qi included in the first work environment image correspond to the linked work machine, the second work environment image including an actual machine image Qj corresponding to the linked work machine is outputted on the output interface <NUM> of the remote operating device <NUM> together with the second label image (see <FIG>/STEP126-STEP128, STEP216). The "second label image" indicates that the actual machine image Qj is the linked work machine.

Thus, by the operator of the first client (first remote operating device <NUM>) transmitting the label image request accompanied with the first identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, whether or not any of the actual machine images Qj included in the second work environment image outputted on the output interface <NUM> of the first client corresponds to the own machine can be checked. Further, by the operator of the first client transmitting the label image request accompanied with the second identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, whether or not any of the actual machine images Qj included in the second work environment image outputted on the output interface <NUM> of the first client corresponds to the other machine can be checked.

Similarly, by the operator of the first client transmitting the label image request accompanied with the first identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, the operator of the second client can be enabled to check whether or not any of the actual machine images Qj included in the second work environment image outputted on the output interface <NUM> of the second client corresponds to the other machine. Further, by the operator of the first client transmitting the label image request accompanied with the second identifier to the work assisting server <NUM> through the operation of the input interface <NUM>, the operator of the second client can be enabled to check whether or not any of the actual machine images Qj included in the second work environment image outputted on the output interface <NUM> of the second client corresponds to the own machine.

Thus, the plurality of operators share the recognition of which remote operating device <NUM> remotely operates a work machine <NUM> shown in the second work environment image outputted on the output interface constituting the remote operating device.

In the embodiment described above, the work assisting server <NUM> is constituted of one or more servers separate from each of the remote operating device <NUM> and the work machine <NUM> (see <FIG>); however, as another embodiment, the work assisting server <NUM> may be a component of the remote operating device <NUM> or the work machine <NUM>. Each of the individual components <NUM> and <NUM> of the work assisting server <NUM> may be components of two or more intercommunicable devices of the remote operating devices <NUM> and the work machines <NUM>.

When the second assisting processing element <NUM> receives the label image request not accompanied with the identifier based on the communication with the first client (first remote operating device <NUM>) (see <FIG>/C13), the second assisting processing element <NUM> may recognize which work machine <NUM> of the plurality of work machines <NUM> corresponds to the actual machine image Qi, based on the position of the actual machine image Qi in the first work environment image and the respective real space positions of the plurality of work machines <NUM> stored and held in the database <NUM>. Then, the second assisting processing element <NUM> causes the label image indicating which work machine <NUM> corresponds to the actual machine image Qi to be outputted on the output interface <NUM> of at least one remote operating device <NUM> of the first and second remote operating devices based on the communication with the at least one remote operating device <NUM>.

According to the work assisting server <NUM> or the like of the configuration, when the work machine <NUM> is shown in the work environment image outputted on the output interface constituting the first remote operating device, which remote operating device <NUM> or which operator remotely operates the work machine <NUM> is recognized. Then, the label image indicating the result of the recognition is outputted on the output interface of at least one remote operating device of the first and second remote operating devices. As a result, each of the operators can be enabled to easily recognize which remote operating device the actual machine image outputted on the output interface constituting the remote operating device corresponds to the remote operation target of.

In the embodiment described above, whether or not the target actual machine is included in the first to third work environment images is determined based on the real space position of the work machine <NUM> measured in the positioning device <NUM>; however, as another embodiment, the image analysis processing may be executed on each of the work environment images and thus the actual machine image Qi indicating the work machine <NUM> may be extracted in the first work environment image.

For example, when the image analysis processing is executed on the first work environment image, the actual machine images Qi (i=<NUM>, <NUM>,. ) indicating the work machines <NUM> are extracted in the first work environment image by the image analysis processing (<FIG>).

Subsequently, based on the position of the actual machine image Qi in the first work environment image coordinate system, the real space position of the work machine <NUM> corresponding to the actual machine image Qi is obtained. At the time, the real space distance from the image pickup device (image pickup device <NUM>, for example) to the work machine <NUM> is estimated based on the size of the actual machine image Qi in the first work environment coordinates. When the distance to an object an image of which is picked up acquired by a ranging sensor such as a TOF sensor is included as a pixel value of the first work environment image, the real space distance from the image pickup device (image pickup device <NUM>, for example) to the work machine <NUM> may be estimated based on the pixel value. Further, by coordinate transformation of the position of the work machine <NUM> in an image pickup device coordinate system to the real space coordinate system, the position of the target actual machine (work machine <NUM>) in the real space coordinate system is recognized or calculated. At the time of the coordinate transformation, the position of the target actual machine in the image pickup device coordinate system is recognized and a coordinate transformation factor (matrix or quaternion) indicating the position and posture of the image pickup device in the real space coordinate system is used. The coordinate transformation factor is registered in association with the identifier of the image pickup device (consequently, an image identifier) in the database <NUM>.

Then, the presence/absence in the first work environment image of the actual machine image Qi indicating the work machine <NUM> having the real space position that matches with or corresponds to the real space position of the target actual machine registered in the database <NUM> is determined.

In the embodiment described above, one work machine is the target of the label image request; however, a plurality of work machines may be the target of the label image request. For example, by imparting red to the texture image T of the label image of the first work machine and white or the like to the texture image T of the label image of the second work machine, the plurality of operators share the recognition.

In the embodiment described above, the label image cannot be set when the specifying notice indicating that the actual machine image corresponding to the target actual machine is not present in the first to third work environment images is transmitted to the remote operating device <NUM> (<FIG>/STEP132); however, as another embodiment, the label image may be set even when the actual machine image corresponding to the target actual machine is not present in the first to third work environment images. For example, it is the case where there are few usable image pickup devices at the work site or the case where the work machine <NUM> is being transported to the work site by a carrier and is to arrive at the work site henceforward. In this case, by setting the label image beforehand, the need of performing recognition sharing work again is eliminated and thus workability improves.

When the result of determining whether or not the real space position of the linked work machine is included in the real space image pickup range of the image pickup device is negative, the second assisting processing element <NUM> executes selective output processing of identifying, as a specified image pickup device, the image pickup device capable of including the real space position of the linked work machine in the real space image pickup range of the image pickup device, and causing the label image indicating the presence of the linked work machine at the specified position in the work environment image acquired by the specified image pickup device to be outputted on the remote output interface <NUM> of each of the plurality of remote operating devices <NUM>.

In this case, when the real space position of the linked work machine is not included in the real space image pickup range of one image pickup device, an alternative work environment image acquired by another image pickup device capable of including the real space position of the linked work machine in the real space image pickup range thereof can be outputted on the remote output interface <NUM> of each of the remote operating devices <NUM>. Thus, the plurality of operators can share the recognition about the presence of the linked work machine.

The second assisting processing element <NUM> identifies at least one actual machine image pickup device <NUM> loaded on at least one work machine of the plurality of work machines <NUM> as the specified image pickup device.

In this case, when the linked work machine which is the target of the label image request is not included in the work environment image individually outputted on the remote output interface <NUM> of each of the plurality of remote operating devices <NUM>, the work environment image outputted on the remote output interface <NUM> of another remote operating device <NUM> including the linked work machine can be outputted. Thus, the plurality of operators of the plurality of remote operating devices <NUM> can share the recognition about the presence of the linked work machine which is the target of the label image request.

When the second assisting processing element <NUM> cannot identify the at least one actual machine image pickup device <NUM> loaded on the at least one work machine <NUM> as the specified image pickup device, the second assisting processing element <NUM> identifies the image pickup device not loaded on any of the plurality of work machines <NUM> as the specified image pickup device.

In this case, when at least one image pickup device loaded on at least one work machine is not identified as the specified image pickup device, the work environment image acquired by the specified image pickup device which is the image pickup device (for example, the image pickup device installed at the work site) not loaded on any of the plurality of work machines can be outputted on the output interface of each of the plurality of remote operating devices. Thus, the plurality of operators of the plurality of remote operating devices can share the recognition about the presence of the linked work machine which is the target of the label image request.

When the work environment image outputted on the remote output interface <NUM> of the one remote operating device <NUM> and the work environment image outputted on the remote output interface <NUM> of another remote operating device <NUM> is the same, the second assisting processing element <NUM> causes the information indicating the other remote operating device <NUM> to be outputted on the remote output interface <NUM> of the one remote operating device <NUM>.

In this case, when the work environment image outputted on the remote output interface <NUM> of the one remote operating device <NUM> is outputted also on the remote output interface <NUM> of the other remote operating device <NUM>, the information indicating the other remote operating device <NUM> is outputted on the remote output interface <NUM> of the one remote operating device <NUM>.

When the label image outputted on the remote output interface <NUM> of one remote operating device <NUM> and the label image outputted on the remote output interface <NUM> of another remote operating device <NUM> indicate the presence of the same linked work machine, the second assisting processing element <NUM> causes the label image to be outputted on the remote output interfaces <NUM> of the one remote operating device <NUM> and the other remote operating device <NUM> in a common form.

In this case, since the label image indicating the presence of the same linked work machine is outputted in the common form on the remote output interfaces <NUM> of the plurality of remote operating devices <NUM>, the plurality of operators can share the recognition about the presence of the linked work machine which is the target of the label image request.

The second assisting processing element <NUM> causes the plurality of label images to be outputted in a mutually identifiable form on the remote output interface <NUM> of each of the plurality of remote operating devices <NUM>.

In this case, since the plurality of label images are outputted in the mutually identifiable form on the remote output interface of each of the plurality of remote operating devices <NUM>, the plurality of operators can share the recognition about the identification and presence of each of the plurality of linked work machines which are the targets of the label image request.

The second assisting processing element <NUM> determines whether or not the actual machine image corresponds to the linked work machine, based on the position of the actual machine image in the work environment image corresponding to the work machine and the real space position of the linked work machine stored and held in the database, and, on the condition that the result of the determination is positive, causes the label image indicating the presence of the linked work machine to be outputted on the remote output interface <NUM> of each of the plurality of remote operating devices <NUM>.

In this case, by the presence/absence of the output of the label image, the operator can be enabled to recognize whether or not the actual machine image included in the work environment image outputted on the output interface of one remote operating device corresponds to the linked work machine which is the remote operation target by the one remote operating device.

The second assisting processing element <NUM> recognizes which work machine <NUM> corresponds to the actual machine image, based on the position of the actual machine image in the work environment image corresponding to the work machine <NUM> and the respective real space positions of the plurality of work machines <NUM> stored and held in the database <NUM>, and causes the label image indicating which work machine <NUM> corresponds to the actual machine image to be outputted on at least one remote output interface <NUM> of at least one remote operating device <NUM> based on the communication with the plurality of remote operating devices <NUM>.

In this case, when the work machine <NUM> is shown in the work environment image outputted on the remote output interface <NUM> constituting the remote operating device <NUM>, which remote operating device <NUM> or which operator remotely operates the work machine <NUM> is recognized. Then, the label image indicating the result of the recognition is outputted on the output interface of at least one remote operating device of the first remote operating device and the second remote operating device. As a result, each of the operators can be enabled to easily recognize which remote operating device the actual machine image outputted on the output interface constituting the remote operating device corresponds to the remote operation target of.

Claim 1:
A work assisting server for assisting a remote operation of each of a plurality of work machines using corresponding each of a plurality of remote operating devices, the work assisting server comprising:
a database configured to store and hold, in association with one another, an identifier of a remote operating device, an identifier of a linked work machine, which is the work machine linked with the remote operating device, and a real space position of the linked work machine;
a first assisting processing element configured to cause one or more work environment images according to one or more picked-up images of a work site acquired through image pickup devices to be outputted on an output interface of each of the plurality of remote operating devices based on communication with the corresponding each of the plurality of remote operating devices; and
a second assisting processing element configured to recognize, in a case of receiving a label image request transmitted based on an operation of any one of the plurality of remote operation devices, wherein the label image request is accompanied with the identifier of the linked work machine based on communication with each of the plurality of remote operating devices, the real space position of the linked work machine stored and held in the database in association with the identifier, determine whether or not the real space position of the linked work machine is included in a real space image pickup range of the image pickup device, and, when a result of the determination is positive, cause a label image indicating presence of the linked work machine at a specified position in the work environment image corresponding to the real space position of the linked work machine to be outputted on the output interface of each of the plurality of remote operating devices,
wherein the label includes a texture image in which a color can be set by an operator or a mark image in which a symbol can be set by an operator.