Patent Description:
A terminal device for a remote monitoring assist system has been proposed for sharing information with sufficient accuracy between workers who are patrolling and inspecting a plant and those who are waiting outside the work site (see, for example,.

This terminal device comprises a video input unit configured to input video data from the site, a pen or mouse type input operation selection unit, a detection unit configured to detect whether a new video has been acquired or not, a communication control unit configured to wirelessly transmit and receive data to and from external devices, and an input/output screen display unit configured to display an entry screen for entering predetermined data.

<CIT> discloses a remote operator of a shovel being presented with an image of a space captured by a drone camera that is unable to be captured by a camera mounted on an upper turning body of the shovel. The operator remotely controls the drone position by means of a remote control displaying the drone's image.

However, when an image captured through an image pickup device is displayed on an output interface (image display device) that constitutes a remote operation device, and multiple work machines are in the image, the operator may mistakenly believe that another work machine different from the actual one in the displayed image is to be operated. This is particularly noticeable when multiple work machines of the same model of the same manufacturer are simultaneously displayed on the output interface. In this case, the motion mode of the work machine using the remote operation device based on the operator's misidentification may become inappropriate to execute the target work.

An object of the present invention is to provide a server and a system that can assist an operator in recognizing whether or not a work machine in a work-site bird's-eye image output on an output interface constituting a remote operation device is subject to remote operation by the remote operation device.

The present invention relates to a work assist server for assisting work machine remote operation by operators using remote operation devices.

A work assist server of the present invention comprises: a first assist processing element that causes an output interface of the remote operation device to output, based on communication with the remote operation device, a work-site bird's-eye image according to an image captured at a worksite acquired through an image pickup device, recognizes a real spatial position of a connection-target work machine as the work machine to be subject to operation through the remote operation device, and when a work machine is in the work-site bird's-eye image, determines whether or not the work machine is the connection-target work machine based on the position of the work machine in an image coordinate system and the real spatial position of the connection-target work machine; and a second assist processing element that causes the output interface of the remote operation device to output, based on communication with the remote operation device, a first sign image showing that the work machine in the work-site bird's-eye image is the connection-target work machine when the first assist processing element determines that the work machine is the connection-target work machine.

The work assist system of the present invention includes a work assist server of the present invention and the remote operation device.

In a work assist server and work assist system with this configuration (hereinafter referred to as "work assist server and the like" as appropriate), when a work machine is in a work-site bird's-eye image that is output on an output interface constituting a remote operation device according to the image captured by an image pickup device, whether or not the work machine is a connection-target work machine is determined. The "connection-target work machine" refers to the work machine to be actually remotely operated by the remote operation device. The "work-site bird's-eye image" is a concept that includes at least a part of the captured image itself as well as a pseudo-image generated based on the captured image to represent the work site in a pseudo manner. When the determination result is positive, the work machine in the work-site bird's-eye image is associated with the connection-target work machine, and a first sign image is displayed on the output interface of the remote operation device.

As a result, the operator can easily recognize whether or not the work machine in the work-site bird's-eye image is subject to remote operation by the remote operation device. Thus, for example, even when multiple work machines are in the work-site bird's-eye image, the operator can easily recognize which of the multiple work machines is the connection-target work machine or which of the multiple work machines is not the connection-target work machine.

The work assist system as one embodiment of the present invention shown in <FIG> includes a work assist server <NUM>, multiple remote operation devices <NUM> for remotely operate multiple work machines <NUM>. The work assist server <NUM>, remote operation devices <NUM>, and work machines <NUM> are configured to be able to communicate with each other through a network. The multiple remote operation devices <NUM> may be configured to be able to communicate with each other through a network different from the network for communication with the work assist server <NUM>.

The work assist server <NUM> comprises a database <NUM>, a first assist processing element <NUM>, and a second assist processing element <NUM>. The database <NUM> stores and maintains the position trajectories of the multiple work machines <NUM>, as well as captured images, work-site bird's-eye images, route guide images and the like. The database <NUM> may be a database server separate from the work assist server <NUM>. Each assist processing element is composed of an arithmetic processing device (single-core processor or multi-core processor, or processor core constituting it), reads necessary data and software from a memory or other storage devices, and performs the arithmetic computations described below according to the software on the data.

The remote operation device <NUM>, which constitutes a client, comprises a remote control device <NUM>, a remote input interface <NUM>, and a remote output interface <NUM>. The remote control device <NUM> is an arithmetic processing device (single-core processor or multi-core processor, or processor core constituting it), reads necessary data and software from a memory or other storage devices, and performs the arithmetic computations according to the software on the data. The remote input interface <NUM> comprises a remote operation mechanism <NUM>. The remote output interface <NUM> comprises an image output device <NUM> and a remote wireless communication unit <NUM>.

The client may be a mobile device such as a smartphone or tablet computer, or a wearable device such as VR goggles, that can connect to or has the function of intercommunicating with the remote operation device <NUM>. The mobile device or wearable device may have the function of communicating with the work assist server <NUM>.

The remote operation mechanism <NUM> includes a traveling operation device, a turning operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has operation levers that are subject to rotary operation. The operation levers of the traveling operation device (traveling levers) are operated to move a lower traveling body <NUM> of the work machine <NUM>. The traveling levers may also serve as traveling pedals. For example, traveling pedals may be fixed to the base or lower end of the traveling levers. The operation lever of the turning operation device (turning lever) is operated to move the hydraulic turning motor that constitutes a turning mechanism <NUM> of the work machine <NUM>. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder <NUM> of the work machine <NUM>. The operation lever of the arm operation device (arm lever) is operated to move the arm cylinder <NUM> of the work machine <NUM>. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder <NUM> of the work machine <NUM>.

Each of the operation levers constituting the remote operation mechanism <NUM> is located, for example, around a seat St for the operator to sit on, as shown in <FIG>. The seat St is in the form of a high-back chair with armrests, but can also be a low-back chair without a headrest, a chair without a backrest, or any other forms that a remote operator OP2 can sit on.

A pair of left and right traveling levers <NUM> corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the right operation lever <NUM> in front of the right frame of the seat St shown in <FIG> may function as a boom lever when operated in the forward/backward direction and as a bucket lever when operated in the left/right direction. Similarly, the left operation lever <NUM>, located in front of the left frame of the seat St shown in <FIG> may function as an arm lever when operated in the forward/backward direction and as a turning lever when operated in the left/right direction. The lever pattern may be changed as desired by the operator's operational instructions.

The image output device <NUM> consists of a right diagonally front image output device <NUM>, a front image output device <NUM>, and a left diagonally front image output device <NUM> which are located, for example, diagonally in front and to the right of, in front of, and diagonally in front and to the left of the seat St, respectively, as shown in <FIG>. The image output devices <NUM> to <NUM> may further comprise a speaker (audio output device).

The work machine <NUM> comprises an actual machine control device <NUM>, an actual machine input interface <NUM>, an actual machine output interface <NUM>, and a working mechanism <NUM>. The actual machine control device <NUM> is an arithmetic processing device (single-core processor or multi-core processor, or processor core constituting it), reads necessary data and software from a storage device, such as a memory, and performs arithmetic computations on the data according to the software.

The work machine <NUM> is, for example, a crawler excavator (heavy construction equipment) and, as shown in <FIG>, comprises a crawler-type lower traveling body <NUM>, and an upper turning body <NUM> mounted on the lower traveling body <NUM> via the turning mechanism <NUM> so that it can turn. A cab (operator's cab) <NUM> is provided in front to the left of the upper turning body <NUM>. A work attachment <NUM> is provided in front center of the upper turning body <NUM>.

The actual machine input interface <NUM> comprises an actual machine operation mechanism <NUM> and an actual machine image pickup device <NUM>. The actual machine operation mechanism <NUM> comprises multiple operation levers located around the seat inside the cab <NUM> in the same manner as in the remote operation mechanism <NUM>. A drive mechanism or robot is provided in the cab <NUM> to receive signals in response to the manner of operation of the remote operation levers and move the actual machine operation levers according to the received signals. The actual machine image pickup device <NUM> is installed, for example, inside the cab <NUM>, and captures an environment, including at least a part of the working mechanism <NUM>, through the front window of the cab <NUM>.

The actual machine output interface <NUM> comprises an actual machine wireless communication device <NUM>.

The work attachment <NUM> as a working mechanism comprises a boom <NUM> mounted on the upper turning body <NUM> so as to be movable up and down, an arm <NUM> pivotably coupled to the end of the boom <NUM>, and a bucket <NUM> pivotably coupled to the end of the arm <NUM>. The work attachment <NUM> has a boom cylinder <NUM>, an arm cylinder <NUM>, and a bucket cylinder <NUM>, which is an extendable hydraulic cylinder.

The boom cylinder <NUM> is interposed between the boom <NUM> and the upper turning body <NUM> so as to extend and contract with supplied hydraulic fluid to rotate the boom <NUM> in the up-and-down direction. The arm cylinder <NUM> is interposed between the arm <NUM> and the boom <NUM> so as to extend and contract with supplied hydraulic fluid to rotate the arm <NUM> about a horizontal shaft with respect to the boom <NUM>. The bucket cylinder <NUM> is interposed between the bucket <NUM> and the arm <NUM> so as to extend and contract with supplied hydraulic fluid to rotate the bucket <NUM> about a horizontal shaft with respect to the arm <NUM>.

The functions of the work assist system with the aforementioned configuration will be explained using the flowcharts shown in <FIG> and <FIG>. In the flowcharts, the block "Cn" is used to simplify description, means transmission and/or reception of data and means a conditional branch in which processing in a branch direction is executed on condition of transmission and/or reception of the data.

"Recognizing" information by any of the components (computing and processing resources or hardware resources) is a concept that includes processes for preparing the information in all forms that can be used in succeeding processes, such as receiving the information, reading out or searching for the information from or in a storage device or the like, writing the information into (causing the information to be stored and held in) a storage device or the like or registering the information with the storage device or the like, and performing estimation, judgment, identification, measurement, prediction and the like of the information by executing arithmetic processing of an output signal from a sensor and/or received or retrieved basic information according to a predetermined algorithm.

With the remote operation device <NUM>, it is determined whether or not a first designated operation has been made through the remote input interface <NUM> by the operator (<FIG>/STEP <NUM>). The first designated operation is an operation to select the work machine <NUM> to be connected to the remote operation device <NUM>, for example, an operation such as tapping, swiping, or clicking on the work-site bird's-eye image (see <FIG>) on the touch panel that constitutes the remote input interface <NUM> made by the operator. In <FIG>, actual machine images Q1 to Q4 of multiple (four) work machines <NUM> that can be subject to remote operation are displayed, and whether or not an operation has been performed on any of the positions of the actual machine images displayed on the touch panel is determined. The remote operation device <NUM> may acquire candidate heavy construction equipment that can be subject to remote operation by the remote operation device <NUM> via the communication network, and display various types of information such as the position, model, and operating status of the heavy construction equipment with the identifier of the heavy construction equipment on the remote output interface <NUM>. If the determination result is negative, such as when the first designated operation is interrupted (<FIG>/STEP <NUM>, NO), the series of processing ends. In contrast, if the determination result is positive (<FIG>/STEP <NUM>, YES), a first work environment confirmation request is sent to the work assist server <NUM> through the remote wireless communication unit <NUM> to request for the captured images needed to operate the work machine <NUM> to be connected to the remote operation device <NUM> (<FIG>/STEP <NUM>). The first work environment image request includes at least one of the identifier of the remote operation device <NUM> and the identifier of the operator.

When the work assist server <NUM> receives the first work environment confirmation request, the first assist processing element <NUM> sends the first work environment image request to the corresponding work machine <NUM> (<FIG>/C10).

When the work machine <NUM> receives the first work environment confirmation request through the actual machine wireless communication device <NUM> (<FIG>/C41), the actual machine control device <NUM> obtains the image captured through the actual machine image pickup device <NUM> (<FIG>/STEP <NUM>). The actual machine control device <NUM> sends captured image data representing the captured image, through the actual machine wireless communication device <NUM> to the work assist server <NUM> (<FIG>/STEP <NUM>).

When the work assist server <NUM> receives the captured image data (<FIG>/C11), first work environment image data (data representing all or part of the captured image itself or a simulated first work environment image generated based on it) related to the captured image data is sent to the remote operation device <NUM> (<FIG>/STEP <NUM>). Data representing the work-site bird's-eye image (see <FIG>), which will be described below, may be sent to the remote operation device <NUM> as the first work environment image data.

When the remote operation device <NUM> receives the first work environment image data through the remote wireless communication unit <NUM> (<FIG>/C21), a first work environment image related to the first work environment image data is output to the image output device <NUM> (<FIG>/STEP <NUM>).

Hence, as shown in <FIG>, for example, the work-site bird's-eye image that includes the boom <NUM>, arm <NUM>, bucket <NUM>, and arm cylinder <NUM> which are a part of the work attachment <NUM> as the working mechanism captured by the actual machine image pickup device <NUM> mounted on the work machine <NUM> to be connected to the remote operation device <NUM> is displayed on the image output device <NUM>.

In the remote operation device <NUM>, the remote control device <NUM> recognizes an operation mode of the remote operation mechanism <NUM> (<FIG>/STEP <NUM>), and a remote operation command suitable for the operation mode is sent to the work assist server <NUM> through the remote wireless communication unit <NUM> (<FIG>/STEP <NUM>).

When the work assist server <NUM> receives the remote operation command, the first assist processing element <NUM> sends the remote operation command to the work machine <NUM> (<FIG>/C12).

When, in the work machine <NUM>, the actual machine control device <NUM> receives an operation command through the actual machine wireless communication device <NUM> (<FIG>/C42), the motion of the work attachment <NUM> and the like is controlled (<FIG>/STEP <NUM>). For example, the bucket <NUM> is used to scoop soil in front of the work machine <NUM>, and the upper turning body <NUM> is turned and the soil is dropped from the bucket <NUM>.

In the work assist server, the bird's-eye view work environment representing the bird's-eye view of the work site (or data representing the same) is acquired based on the captured image acquired through an image pickup device (not shown in the drawing) installed at the work site or the actual machine image pickup device <NUM> mounted on a work machine <NUM> that is not connected to the remote operation device <NUM> (<FIG>/STEP <NUM>). Accordingly, based on the captured image acquired through the image pickup device C (see <FIG>) installed at the work site, for example, as shown in <FIG>, data representing an image of the work site including the actual machine images Q1 to Q4 of the multiple (four) work machines <NUM> is acquired as data representing a work-site bird's-eye image.

In the remote operation device <NUM> (first client), whether or not the second designated operation has been made through the remote input interface <NUM> by the operator is determined (<FIG>/STEP <NUM>). The second designated operation is an operation to allow a work-site bird's-eye image that is different from the first work environment image displayed on the image output device <NUM> upon the first designated operation to be displayed on the image output device <NUM>, and is, for example, an operation such as tapping through the remote input interface <NUM> or operating the remote operation mechanism <NUM>. Not a first work environment image but the work-site bird's-eye image may be output to the image output device <NUM> in response to the first designated operation (see <FIG>/STEP <NUM>), and a first work environment image may be output to the image output device <NUM> in response to the second designated operation (see <FIG>/STEP <NUM>). Different second work environment images may be output to the image output device <NUM> in response to the first designated operation and the second designated operation, respectively.

If the determination result is negative, such as when the second designated operation is interrupted (<FIG>/STEP <NUM>, NO), the series of processing ends. If the determination result is positive (<FIG>/STEP <NUM>, YES), a second work environment request is sent to the work assist server <NUM> through the remote wireless communication unit <NUM> (<FIG>/STEP <NUM>). The second work environment request includes the identifier of the corresponding remote operation device <NUM> or the work machine <NUM> connected to it.

When the work assist server <NUM> receives the second work environment confirmation request (<FIG>/C13), the first assist processing element <NUM> determines whether or not any work machine <NUM> is in the work-site bird's-eye image on the output interface <NUM> of the remote operation device <NUM> (<FIG>/STEP <NUM>). To be specific, the first assist processing element <NUM> performs image analysis processing on the work-site bird's-eye image to determine whether or not there is the actual machine image Qi (i = <NUM>, <NUM>,. , N) representing the work machine <NUM> in the work-site bird's-eye image.

If the determination result is positive (<FIG>/STEP <NUM>, YES), the index i for distinguishing the actual machine image Qi is set to "<NUM>" (<FIG>/STEP <NUM>).

After that, the first assist processing element <NUM> determines whether or not the actual machine image Qi corresponds to a connection-target work machine (<FIG>/STEP <NUM>). To be specific, whether or not the real spatial position of the work machine <NUM> corresponding to the actual machine image Qi matches or corresponds to the real spatial position of the connection-target work machine registered to the database <NUM> is determined.

For example, the position (e.g., the position of the center of gravity) of the actual machine image Qi in the image coordinate system is transformed into a real spatial coordinate system to determine the real spatial position of the work machine <NUM> corresponding to the actual machine image Qi. In this case, based on the size of the actual machine image Qi in the work-site bird's-eye image, the real spatial distance from the image pickup device (e.g., the image pickup device C (see <FIG>)) to the work machine <NUM> is estimated. When the distance to the object to be imaged, which is acquired by a distance measurement sensor such as TOF sensor, is included as a pixel value in the work-site bird's-eye image, the real spatial distance from the image pickup device (e.g., the mage pickup device C) to the work machine <NUM> may be estimated based on the pixel value. In the coordinate transformation, the position of each work machine <NUM> in the coordinate system of the image pickup device is recognized, and coordinate transformation factors (matrix or quaternion) representing the position and posture of the image pickup device in the real spatial coordinate system are used. The coordinate transformation factors are registered in the database <NUM>, being associated with the identifiers of the image pickup devices (and thus the image identifiers).

When intercommunication is established between the remote operation device <NUM> (client) and the work machine <NUM>, the identifiers of the remote operation device <NUM> (or its operator) and the work machine <NUM>, and the real spatial position of the work machine <NUM> are associated and registered to the database <NUM>. The real spatial position of each work machine <NUM> is determined using the GPS on the work machine <NUM> and, as needed, using a position determination device using an acceleration sensor. When the work machine <NUM> transmits the real spatial position or its time series to the work assist server <NUM>, the real spatial position of the work machine <NUM> registered in the database <NUM> is updated. Hence, based on the identifier, the real spatial position of the connection-target work machine can be searched for or recognized from the database <NUM>.

If the determination result is positive (<FIG>/STEP <NUM>, YES), the second assist processing element <NUM> sets a flag f to "<NUM>" (<FIG>/STEP <NUM>). If the determination result is negative (<FIG>/STEP <NUM>, NO), the second assist processing element <NUM> sets the flag f to "<NUM>" (<FIG>/STEP <NUM>).

Furthermore, whether or not the index i is equal to the total number N of the actual machine image Qi included in the work-site bird's-eye image is determined (<FIG>/STEP <NUM>). If the determination result is negative (<FIG>/STEP <NUM>, NO), the index i is increased by one (<FIG>/STEP <NUM>), and processing for determining whether or not the actual machine image Qi corresponds to the connection-target work machine is repeated (see <FIG>/STEP <NUM>).

If the determination result is positive (<FIG>/STEP <NUM>, YES), the second assist processing element <NUM> generates a second work environment image based on the work-site bird's-eye image and the flag f (<FIG>/STEP <NUM>).

To be specific, if there is an actual machine image Qi with the flag f set to "<NUM>" in the work-site bird's-eye image, a second work environment image containing a first sign image ID1 is generated. Accordingly, as shown in <FIG>, for example, the second work environment image containing the first sign image ID1 in the form of a speech balloon that shows that the work machine <NUM> corresponding to the actual machine image Q2 in the work-site bird's-eye image matches the connection-target work machine is generated. The speech balloon originates from the actual machine image Q2 or its vicinity. The combination of shape, color, and pattern of the first sign image ID1 may be changed as desired.

If there is an actual machine image Qi with the flag f set to "<NUM>" in the work-site bird's-eye image, a second work environment image containing second sign images ID2 is generated. Accordingly, as shown in <FIG>, for example, the second work environment image containing the second sign images ID2 in the form of speech balloons that show each work machines <NUM> corresponding to the actual machine images Q1, Q3, and Q4 in the work-site bird's-eye image does not match the connection-target work machine is generated. The speech balloons originate from the actual machine images Q1, Q3, and Q4 or their vicinities. The combination of shape, color, and pattern of each second sign image ID2 may be changed as desired as long as it can be distinguished from the first sign image ID1. A second work environment image containing only the first sign image ID1 but no second sign images ID2 may also be generated. If it is determined that a work machine i matches a connection-target work machine (<FIG>/STEP <NUM>, YES), the determination processing in STEP <NUM> may be omitted and a second work environment image may be generated immediately (<FIG>/STEP <NUM>).

The second assist processing element <NUM> then sends data representing the second work environment image to the remote operation device <NUM> (<FIG>/STEP <NUM>).

When the remote operation device <NUM> receives the second work environment image data through the remote wireless communication unit <NUM> (<FIG>/C22), the second work environment image is output through the image output device <NUM> that constitutes the remote output interface <NUM> (<FIG>/STEP <NUM>). Accordingly, as shown in <FIG>, for example, a second work environment image containing a first sign image ID1 showing that the work machine <NUM> corresponding to the actual machine image Q2 is the connection-target work machine, and second sign images ID2 showing that the work machines <NUM> corresponding to the actual machine images Q1, Q3, and Q4 are the other work machines is output to the image output device <NUM>.

In the work assist system and the work assist server <NUM> constituting the same with the aforementioned configuration, according to the captured image acquired through the image pickup device C, when a work machine <NUM> is in the work-site bird's-eye image (see <FIG>) that is output on the remote output interface <NUM> constituting the remote operation device <NUM>, whether or not the work machine <NUM> is a connection-target work machine is determined (see <FIG>/STEP <NUM>). If the determination result is positive, a first sign image ID1 associated with an actual machine image Q2 that represents the work machine <NUM> in the work-site bird's-eye image is displayed on the remote output interface <NUM> of the remote operation device <NUM> (see <FIG>/STEP <NUM>, YES ->. -> STEP <NUM> -> STEP <NUM> -> C22 -> STEP <NUM> and <FIG>).

If it is determined that work machines <NUM> in the work-site bird's-eye image are not connection-target work machines, the second sign images ID2 associated with the actual machine images Q1, Q3, and Q4 representing these work machines <NUM> are output on the remote output interface <NUM> of the remote operation device <NUM> (see <FIG>/STEP <NUM>, NO ->. -> STEP <NUM> -> STEP <NUM> -> C22 -> STEP <NUM> and <FIG>).

Consequently, the operator can easily recognize whether or not the work machines <NUM> or the actual machine images Qi in the work-site bird's-eye image are subject to remote operation by the remote operation device <NUM>. Thus, even if multiple work machines <NUM> are in the work-site bird's-eye image, for example (see <FIG>), the operator can easily recognize which of the multiple work machines <NUM> is a connection-target work machine or whether none of them is a connection-target work machine.

If the first assist processing element <NUM> determines that some work machines <NUM> are not connection-target work machines, the second assist processing element <NUM> causes the remote output interface <NUM> of the remote operation device <NUM> to output, based on communication with the remote operation device <NUM>, the second sign images ID2 showing that the work machines <NUM> are not connection-target work machines in such a manner that they can be distinguished from the first sign image ID1 (see <FIG>).

Thus, if it is determined that the work machines <NUM> in the work-site bird's-eye image are not connection-target work machines, the second sign images ID2 associated with these work machines <NUM> are output on the remote output interface <NUM> of the remote operation device <NUM>. Consequently, the operator can easily recognize that the work machines <NUM> displayed on the remote output interface <NUM> constituting the remote operation device <NUM> are not subject to remote operation by the remote operation device <NUM>.

In the aforementioned embodiment, the work assist server <NUM> is one or more servers separate from each of remote operation devices <NUM> and work machines <NUM> (see <FIG>); however, in another embodiment, the work assist server <NUM> may be a component of remote operation devices <NUM> or work machines <NUM>. Each of the components <NUM> and <NUM> of the work assist server <NUM> may be a component of each of two or more devices selected from the remote operation devices <NUM> and work machines <NUM> that can communicate with each other.

When one of the work machines <NUM> determined to be a connection-target work machine by the first assist processing element <NUM> is in the work-site bird's-eye image, the greater the number of work machines <NUM> in the work-site bird's-eye image, the more the second assist processing element <NUM> may emphasize the first sign image ID1 output on the remote output interface <NUM> of the remote operation device <NUM>.

In the work assist server <NUM> and the like with the aforementioned configuration, if one of the work machines <NUM> in the work-site bird's-eye image is a connection-target work machine, the greater the number of work machines <NUM> in the work-site bird's-eye image, the more the first sign image ID1 is emphasized on the remote output interface <NUM> of the remote operation device <NUM>. "Emphasize" is to make the output and display first sign image ID1 more eye-catching or visible by changing the shape, size, color (brightness, lightness, or saturation), or pattern, or any combination thereof, of the first sign image. This is because, in general, the greater the number of work machines <NUM> in the work-site bird's-eye image, the greater the likelihood that the operator will misidentify any of the other work machines <NUM> as a connection-target work machine. Accordingly, the operator can clearly recognize whether or not the work machines <NUM> contained in the work-site bird's-eye image are connection-target work machines.

When multiple work machines <NUM> including a connection-target work machine are shown in the work-site bird's-eye image, the smaller the distance between the connection-target work machine and other work machines in the work-site bird's-eye image, the more the second assist processing element <NUM> may emphasize the first sign image ID1 on the remote output interface <NUM> of the remote operation device <NUM>.

In the work assist server <NUM> and the like with the aforementioned configuration, when multiple work machines <NUM> including a connection-target work machine are shown in the work-site bird's-eye image, the smaller the distance between the connection-target work machine and other work machines in the work-site bird's-eye image, the more the first sign image ID1 is emphasized on the remote output interface <NUM> of the remote operation device <NUM>. This is because the smaller the distance, the greater the likelihood that the operator will misidentify any of the other work machines <NUM> as a connection-target work machine. The distance of "<NUM>" means that the connection-target work machine and the other work machines <NUM> are overlapped in the work-site bird's-eye image. When multiple other work machines <NUM> are shown in the work-site bird's-eye image, the distance means the average distance or the shortest distance between the connection-target work machine and each of the multiple other work machines <NUM>. Thus, the operator can clearly recognize whether or not a certain work machine <NUM> included in the work-site bird's-eye image is a connection-target work machine.

Even if the first assist processing element <NUM> determines that the work machine <NUM> is not a connection-target work machine (<FIG>/STEP <NUM>, NO), the second assist processing element <NUM> may generate a second work environment image that does not contain any second sign image ID2.

The first assist processing element <NUM> may recognize a time series of at least the real spatial position or real spatial posture of the connection-target work machine, and the second assist processing element <NUM> may generate a second work environment image containing, as a first sign image ID, a trajectory image representing a time series of at least the real spatial position or real spatial posture of the connection-target work machine. Accordingly, for example, as shown in <FIG>, a second work environment image containing, as a first sign image ID1, an arrow-shaped trajectory image showing the position trajectory or motion of the actual machine image Q2 or the corresponding work machine <NUM> is generated.

The second work environment image is then output to the remote output interface <NUM> of the remote operation device <NUM>, and the first sign image ID1 allows the operator of the remote operation device <NUM> to recognize whether or not a certain work machine <NUM> in the work-site bird's-eye image is a connection-target work machine, as well as a time series of at least the real spatial position or real spatial posture of the connection-target work machine. Consequently, for example, in consideration of consistency between the operator's operation mode of the remote operation device, and a time series of at least the real spatial position or real spatial posture of the connection-target work machine represented by the trajectory image serving as the first sign image ID1, the operator can recognize more clearly that the connection-target work machine is the work machine <NUM> that he or she is remotely operating.

The second assist processing element <NUM> may cause the remote output interface <NUM> of the remote operation device <NUM> to output, based on communication with the remote operation device <NUM>, a map or work-site bird's-eye image showing the real spatial position of the work machine <NUM> and the real spatial position and angle of the image pickup device C. Accordingly, as shown in <FIG>, for example, in addition to the actual machine image or icons Q1 to Q4 representing the work machines <NUM> existing at the work site, a captured bird's-eye image or bird's-eye map showing the real spatial position and angle R0 of the image pickup device C is output to the image output device <NUM>. The captured bird's eye image may be acquired, for example, through an image pickup device mounted on an unmanned aerial vehicle or an image pickup device installed on a pole or other structures at the work site. Each of the image capture location and angle of the captured image may be changed as desired. A bird's-eye map may be generated based on the captured bird's-eye image.

Accordingly, in consideration of the real spatial relative positions of the work machines shown in the work-site bird's-eye image and the image pickup device that has acquired that work-site bird's-eye image, the operator can check whether or not the perspectives of one or more work machines <NUM> in the work-site bird's-eye image match (see <FIG>). This allows the operator to recognize whether or not a certain work machine <NUM> on the remote output interface <NUM> is a connection-target work machine, as well as whether or not that work machine <NUM> is the work machine <NUM> shown on the map or the work-site bird's-eye image.

The image output to the image output device <NUM> by the first designated operation may be a captured image acquired through an image pickup device (not shown in the drawing) installed at the work site or the actual machine image pickup device <NUM> mounted on a work machine <NUM> that is not connected to the remote operation device <NUM>. In this case, if the determination result of the first designated operation through the remote input interface <NUM> by the operator is positive (<FIG>/STEP <NUM>, YES), a second work environment request is sent to the work assist server <NUM> through the remote wireless communication unit <NUM> (<FIG>/STEP <NUM>), so that the second work environment image data is received through the remote wireless communication unit <NUM> (<FIG>/C22), and the second work environment image is output through the image output device <NUM> constituting the output interface <NUM> (<FIG>/STEP <NUM>).

The image output device <NUM> may output multiple first work environment images and multiple second work environment images through right diagonally front image output device <NUM>, the front image output device <NUM>, and the left diagonally front image output device <NUM>, respectively, or in a split manner in each image output device. In this case, multiple captured images acquired through image pickup devices including the actual machine image pickup device <NUM> mounted on the work machine <NUM> connected to the remote operation device <NUM> are output to the image output device <NUM> by multiple first designated operations, respectively. In addition, multiple captured images acquired through image pickup devices (not shown in the drawing) installed at the work site or image pickup devices including the actual machine image pickup device <NUM> mounted on a work machine <NUM> that is not connected to the remote operation device <NUM> are output to the image output device <NUM> respectively by multiple second designated operations.

For the second work environment image output to the image output device <NUM>, when there is the actual machine image Qi with the flag f set to "<NUM>" in the work-site bird's-eye image, only a first sign image ID1 is generated and second sign images ID2 do not have to be generated.

Claim 1:
A work assist server (<NUM>) for assisting remote operation of a work machine (<NUM>) by an operator using a remote operation device (<NUM>), the work assist server (<NUM>) comprising:
a first assist processing element (<NUM>) that causes an output interface (<NUM>) of the remote operation device (<NUM>) to output, based on communication with the remote operation device (<NUM>), a work-site bird's-eye image according to an image captured at a worksite acquired through an image pickup device (C), determines a real spatial position of a connection-target work machine, which is the work machine to be subject to operation through the remote operation device (<NUM>), and when a work machine is in the work-site bird's-eye image, determines whether or not the work machine is the connection-target work machine based on the position of the work machine in an image coordinate system and the real spatial position of the connection-target work machine; and
a second assist processing element (<NUM>) that causes the output interface (<NUM>) of the remote operation device (<NUM>) to output, based on communication with the remote operation device (<NUM>), a first sign image showing that the work machine in the work-site bird's-eye image is the connection-target work machine when the first assist processing element (<NUM>) determines that the work machine is the connection-target work machine.