Patent Description:
Working machines, in the form of e.g. excavators, dozers, loaders, and the like, may be used to perform various tasks in construction, mining and/or landscaping operations. Such working machines may generally be equipped with one or more work implements that can engage various objects to assist in performing the tasks. The work implements may be adjustable or manipulated in an intended manner by an operator, who e.g. may be positioned on-board the working machine. There exist various control systems and technologies to be incorporated into such working machines for assist its operator in controlling the working machine and/or work implements. These systems and technologies can interact with various a multitude of sensors for providing information and data to the operator regarding the working machine and/or work implements to further assist in control of the same.

Recent advances in computer science now allows the above mentioned control systems providing assistance to the operator to, in real-time, correlate data acquired by the sensors with a dynamic structural environment in which the working machine is operating. Based on the correlation, a so-called augmented reality (AR) may be displayed to the operator for assisting the operator in manipulating the working machine.

Specifically, AR may provide the operator with a view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, text, graphics, or video. In a typical prior-art AR application, a data processor reviews a camera-captured image for cues that trigger the display of additional information and images along with a display of the captured image.

The AR information may for example be presented to the operator using a display screen arranged within an operator compartment comprised with the working machine. However, such an implementation makes in necessary for the operator to split his attention between the display and e.g. the work implement, making the operation precarious and tiresome.

A more dynamic approach may as an alternative be taken, such as disclosed in <CIT>, where the operator is provided with a head mounted display (HMD) that is adapted for augmented reality capabilities. According to <CIT>, AR information may additionally be allowed to assist the operator to "see" the work implement even in cases where the work implement is obstructed from the direct view of the operator.

Even though the solution presented in <CIT> provides great improvements to the safety operation of a working machine, the present inventors have identified further possibilities to assisting the operator, without having to resort to forcing the operator to wear e.g. a HMD, allowing further risk reductions when operating a working machine at e.g. a construction site or the like.

According to an aspect of the present disclosure, the above is at least partly alleviated by working machine, as defined in claim <NUM>, comprising a working machine structure, working machine means pivotally connected to the working machine structure, and an operator compartment provided at the working machine structure and provided with control means adapted to allow an operator to control a position of the working machine means, wherein the working machine further comprises an image projection arrangement for projecting an image at a surface of the working machine means facing the operator compartment and the projected image comprises information for the operator to control the working machine.

In accordance to the present disclosure, it is thus possible to allow for any projection of information for assisting the operator to take place at an area externally of said working machine, i.e. without having to resort to presenting the information at a fixed or a mobile (e.g. HMD) display as proposed in accordance to prior-art. Rather, the working machine means comprised with the working machine is instead used as a "canvas" for presenting the assisting information.

An advantage following this implementation is that the operator of the working machine may be allowed to place his focus at the "right place", which typically will be outside of the operator compartment rather than within the operator compartment. In addition, the projected information may typically be projected where and when "needed" and without the need to provide the operator with e.g. a dedicated HDM or similar. In addition, the solution proposed by the present disclosure solves prior-art problems of focal depth, where the operator has to re-focus when switching his gaze between the working machine means and the display device arranged within the operator compartment.

The image projection arrangement should be understood to mean an arrangement or a device that is capable of providing a visible illumination, including for example markings, lines, symbols, icons, text, images, etc., at the surface of the working machine means facing the operator compartment. The image projection arrangement may as such comprise different type of light sources, such as e.g. light emitting diodes (LED), or an array of LEDs, etc. Other alternatives are of course conceivable, such as e.g. lasers, image/video projectors, etc. Using a laser projector may in some embodiments be preferred due to the possible to provide a high focused monochrome light beam for projecting the image at the surface of the working machine means, as compared to what is some implementations may be achieved using a less focused light beam from e.g. an image projector. It should however be understood that the use of an image projector may be suitable in some implementations of the present disclosure.

Furthermore, the working machine may for example be at least one of an excavator, a wheel loader, a dozer, a grader and a backhoe loader, or any form of corresponding construction equipment comprising suitable working machine means arranged externally and at the same time visible for the operator and suitable for allowing the image to be projected in a manner as discussed above. Correspondingly, the working machine means may of course depend in the type of working machine. For example, the working machine means may in some embodiments comprises at least one of an implement and linkage. The implement may for example be a bucket or similar.

The working machine may, in some embodiments, comprise an internal combustion engine, or be at least one of a pure electrical vehicle (PEV) and a hybrid electric vehicle (HEV). Furthermore, in some embodiments of the present disclosure it may be desirable to equip the working machine with a geolocation arrangement, such as for example a GLONASS or GPS receiver comprised with the working machine, for determining a present location of the working machine such as at a work or construction site. Information relating to the position of the working machine may also comprise information relating to a relative orientation of the working machine.

In the present invention the image projection arrangement further comprises a control unit adapted to adjust a position of the image projected at the surface of the working machine means based on an expected position of the user when the user is positioned at the operator compartment. The estimation is performed using an eye and/or head tracking mechanism connected to the control unit. With the estimated position available, it is possible to allow e.g. perspective views of the image to be correctly projected when seen from the operator's present position.

The control unit may in one embodiment be an electronic control unit (ECU) is typically provided as an onboard component of the working machine. Furthermore, the control unit is preferably adapted to receive information indicative of the current position of the working machine means (relative to e.g. the operator compartment or a ground surface), and to adapt the image projected at the working machine means based on the current position of the working machine means and/or the position of the operator.

The image projected at the surface of the working machine means may comprise information of a current or future task to be performed by the operator of the working machine, such as for example how to control the working machine means at e.g. the present location of the working machine. Such information may for example comprise a specific task using the implement for the excavator, such as digging a hole in ground at a specific location. Thus, the image formed and projected may include information for the operator of where the digging is to take place ("digging instructions"). The digging instruction, or similar, may also comprise a warning to the operator as to e.g. tubes and or pipes arranged underground. Thus, with such information at hand in may be possible to reduce possible digging accidents. The information may also be used informing the operator of e.g. at least one of a desired and a maximum filling level of the bucket.

It should however be understood that the information provided to the operator also, or instead, may include e.g. driving instructions to the operator for maneuvering the working machine. Alternatively, the information may be adapted to provide the operator with feedback on operation of the working machine, such as for example for operating the working machine in a more fuel-efficient manner, etc..

In addition to the above, the control unit may be further adapted to receive geographical data relating to a surrounding of the working machine. Such geographical data may possibly be acquired using one or a plurality of sensors arranged at or in the vicinity of the working machine. Such a sensor or sensors may for example include at least one of camera, a laser scanner, a radar arrangement, an IR based arrangement and a Lidar arrangement. The geographical data may provide information relating to e.g. how the ground surface is arranged, relate to possible structures or obstacles in the surrounding of the working machine, etc. The geographical data may, in one embodiment, be included when forming the image to be projected at the surface of the working machine means.

According to another aspect of the present disclosure there is provided a method for assisting an operator of a working machine, wherein the working machine comprises a working machine structure, working machine means pivotally connected to the working machine structure, an operator compartment provided at the working machine structure and provided with control means adapted to allow the operator to control a position of the working machine means, an image projection arrangement for projecting an image, and a control unit for controlling the image projection arrangement, wherein the method comprises the steps of determining, using the control unit, a position of the working machine means, and projecting the image at a surface of the working machine means facing the operator compartment based on the determined position of the working machine means. This aspect of the present disclosure provides similar advantages as discussed above in relation to the previous aspect of the present disclosure.

This disclosure may, however, be embodied in many different forms within the scope as defined in the appended claims.

With particular reference to <FIG>, there is provided a working machine <NUM> in the form of an excavator <NUM>. The excavator <NUM> comprises movable load unit arrangement <NUM>, which movable load unit arrangement <NUM> comprises a working machine means, exemplified as boom arm <NUM> and an implement <NUM>. In the embodiment depicted in <FIG>, the implement <NUM> is a bucket, although other implements are conceivable as well, such as e.g. a gripping tool, etc. The excavator <NUM> comprises a working machine structure including an upper structure <NUM> and a lower structure <NUM>, wherein the upper <NUM> and lower <NUM> structures are movable relative to each other. In detail, the upper structure <NUM> can rotate relative the lower structure <NUM> around a substantially vertical geometric axis (not shown).

The load unit arrangement <NUM> is connected to the upper structure <NUM> of the excavator <NUM>, which makes it rotatably movable relative to the lower structure <NUM> as well as relative to the ground surface <NUM> thereof. The load unit arrangement <NUM> can also be lifted and lowered relative to the ground surface <NUM> such that the distance between the implement <NUM> and the ground surface <NUM> increases/reduces. In addition, the implement <NUM> can be tilted relative to the boom arm <NUM>. As further illustrated in <FIG>, the upper structure <NUM> comprises a compartment <NUM> for housing an operator of the excavator <NUM>, while the lower structure <NUM> comprises a pair of ground engaging members <NUM> in the form of caterpillar treads <NUM>. Other ground engaging members <NUM> are also conceivable, such as e.g. wheels, etc..

As described, the load unit arrangement <NUM> is movable relative to the lower structure <NUM>, as well as relative to the ground surface <NUM>. <FIG> depicts the movement from a first, present point in time at which the load unit arrangement <NUM> is depicted with solid lines, to a future point in time. The load unit arrangement <NUM> is depicted with dashed lines <NUM> at the future point in time. The movement from the present position to the future position is depicted by arrow <NUM>.

The working machine <NUM> further comprises image projection arrangement <NUM>. A detailed description of the image projection arrangement <NUM> and its implementation and operation in relation to the working machine is given in relation to the description below.

Turning now to <FIG>, there is conceptually shown a possible implementation of the image projection arrangement <NUM>. As exemplified, the image projection arrangement <NUM> is arranged to project an image onto a surface <NUM> externally of said working machine <NUM> e.g. at the boom arm <NUM> and an implement <NUM>. Further details of various image projection at the working machine means, such as the boom arm <NUM> and an implement <NUM> are given below in relation to the description of <FIG>.

The image projection arrangement <NUM> can as discussed above comprise e.g. a laser projector <NUM> for presenting information at the surface <NUM>. The image projection arrangement <NUM> further comprises a control unit <NUM>, such as an electronic control unit (ECU), adapted to form the image to be projected by the laser projector <NUM>. The control unit <NUM> is further adapted to control the laser projector <NUM>. The control unit <NUM> may for example be manifested as a general-purpose processor, an application specific processor, a circuit containing processing components, a group of distributed processing components, a group of distributed computers configured for processing, a field programmable gate array (FPGA), etc..

The processor may be or include any number of hardware components for conducting data or signal processing or for executing computer code stored in memory. The memory may be one or more devices for storing data and/or computer code for completing or facilitating the various methods described in the present description. The memory may include volatile memory or non-volatile memory. The memory may include database components, object code components, script components, or any other type of information structure for supporting the various activities of the present description. According to an exemplary embodiment, any distributed or local memory device may be utilized with the systems and methods of this description. According to an exemplary embodiment the memory is communicably connected to the processor (e.g., via a circuit or any other wired, wireless, or network connection) and includes computer code for executing one or more processes described herein.

The control unit <NUM> is further adapted to receive information indicative of a present location of the working machine <NUM>, such as from a geolocation arrangement <NUM> comprised with the working machine <NUM>. The geolocation arrangement <NUM> may for example comprise a GPS receiver <NUM> or a local positioning arrangement at a construction site, such as for example a Wi-Fi positioning system. It may, as understood by the skilled addressee, be necessary to use an alternative to satellite navigation in case the working machine is operating underground, such as for example in a mine or similar.

In addition, the control unit <NUM> may be adapted to receive information relating to an orientation of the working machine <NUM>, relative to e.g. the work site, thus allowing the working machine <NUM> both positioned and oriented in a desired manner. Such orientation information may for example be provided by the GPS receiver <NUM> or using a separate compass functionality provided with the working machine <NUM>. In the exemplary implementation shown in <FIG>, the control unit <NUM> is further adapted to receive construction data that has been previously stored in a database <NUM> arranged in communication with the control unit <NUM>.

The image projection arrangement <NUM> may further comprise a database <NUM> arranged onboard the working machine <NUM> or remotely from the working machine <NUM>. The database <NUM> is arranged in communication with the control unit <NUM>. In case the database <NUM> is arranged remotely from the working machine <NUM> the image projection arrangement <NUM> may comprise a transceiver (not shown) used for establishing a network connection with the database <NUM>. The database <NUM> may for example comprise digging instructions, driving instructions or similar relating to a specific location (e.g. working site) where the working machine <NUM> is to be operated.

In addition to the above, the control unit <NUM> is preferably adapted to receive sensor data relating a surrounding of the working machine <NUM>, such as from a camera arrangement <NUM>. The camera arrangement <NUM> may, in some embodiments, be a three-dimensional (3D) camera adapted to be used for forming a 3D representation of the surrounding of the working machine <NUM>. Accordingly, the control unit <NUM> is adapted to implement a feedback functionality where e.g. sensor data may be correlated with e.g. the digging instructions, driving instructions or similar stored within the database <NUM>. Thus, it may be possible to determine a matching between a current state at e.g. the work/construction site (i.e. collected using the camera arrangement <NUM>) and the "desired" end result provided by the construction data. The correlation will further allow the projected image to be updated in a corresponding manner, i.e. to be continuously dependent on the current status of the work/construction site and the operation of the working machine means performed by the operator. The control unit <NUM> may also be adapted to be connected to navigation equipment (not shown) comprised with the working machine <NUM> for providing navigation instructions to the control unit <NUM> as will be further discussed below.

The control unit <NUM> may additionally be adapted to (continuously) receive information relating to a current positioning of the working machine means. Such information may for example be available at a CAN bus comprised with the working machine <NUM> or alternatively received from further sensors (not shown) arranged at the working machine means.

With further reference to <FIG> in conjunction with <FIG>, there is provided four examples of assisting the operator of the working machine <NUM> using the image projection arrangement <NUM>.

In <FIG> the operator is provided with a basic instruction at the above discussed surface <NUM> externally of said working machine <NUM> at e.g. the boom arm <NUM> and the implement <NUM>. In <FIG>, the surface <NUM> is defined to include a surface pointing towards the operator compartment <NUM> of at least one of boom arm <NUM> and the implement <NUM>.

The instruction to be provided to the operator in accordance to <FIG> is provided for controlling a position of the implement <NUM> in relation to the ground surface <NUM>. This is according to the presented embodiment achieved by determining, S1, a location (or position) of the implement <NUM>, e.g. using data received from the CAN bus or using the camera arrangement <NUM>. Based on the location/position of the implement <NUM>, the control unit <NUM> forms, possibly based on e.g. the digging instruction received from the database <NUM>, an image <NUM> to be projected, S2, at the surface <NUM>. In the illustration provided in <FIG>, the image is projected at the boom arm <NUM>.

As a result of the projected image <NUM>, showing an arrow pointing upward, the operator is to adjust the position/location of the implement <NUM> such that the implement <NUM> is raised, and the control unit will <NUM> will as a consequence determine, S3, an adjusted position of the implement <NUM>. In case the implement <NUM> has been "raised more" than what was desired (e.g. raised "too much"), an adjusted, S4, image <NUM> may be formed and subsequently projected, S5, at the surface <NUM>. In <FIG>, the adjusted image <NUM> is provided as an arrow pointing downward.

As an alternative and as shown in <FIG>, an alternative image <NUM> may be provided for positioning of the implement <NUM>. Such an image <NUM> may for example comprise lines/markings easily understandable for the operator. In <FIG>, the idea is for the operator to level a centerline <NUM> and the same height as two outer lines <NUM> provided adjacently to the centerline <NUM>, for arranging the implement <NUM> at e.g. a specific height in relation to the ground surface <NUM>. The illustration provided in <FIG> will consequently be dynamically updated based on continuous operation of the implement <NUM>. Thus, the current position/location of the implement <NUM> will be monitored and compared to a desired location/position of the implement <NUM>, such as for example provided with mentioned the digging instructions. The implementation shown in <FIG> will thus be somewhat analogue by showing a continuous adjustment of the centerline <NUM>, whereas the implementation shown in <FIG> will be somewhat "digital" switching between the upward <NUM> and downward <NUM> pointing arrow based on a threshold comparison.

In <FIG> there is provided a still further example of providing the operator with an instruction for operating the implement <NUM>. Specifically, in <FIG> there is a desire to keep the implement "leveled". A determination is made as to the present location/position of the implement <NUM> and it is indicated that the implement is slightly rotated. As a result, a rotational image <NUM> is presented to the operator at the surface <NUM>, as exemplified at the boom arm <NUM>. Once the operator has leveled the implement <NUM> an updated image (not shown) may be provided to inform the operator that the implement <NUM> now is leveled.

Furthermore and as indicated above, the image projected at the surface <NUM> may not necessarily be limited to show information relating to e.g. a specific handling of the implement <NUM>. Rather, other types of information may be provided. As exemplified in <FIG>, an image <NUM> may be projected at the surface <NUM> (exemplified as at the implement <NUM>) for providing the operator with a driving instruction when moving the working machine <NUM> around e.g. a work site. The driving instruction may for example be based on the present location of the working machine <NUM> and a provided task to move the working machine <NUM> from a current to a future location at the work site, for example received from a supervisor at the work site. Again, the position/location of the implement <NUM> is determined, in <FIG> shown as slightly moved to the side to give the operator a proper line of sight when moving the working machine <NUM>. In <FIG>, the image <NUM> is instructing the operator to turn to the right within <NUM> meters from the present location of the working machine <NUM>.

Claim 1:
A working machine (<NUM>), comprising:
- a working machine structure (<NUM>),
- working machine means (<NUM>, <NUM>) pivotally connected to the working machine structure (<NUM>), and
- an operator compartment (<NUM>) provided at the working machine structure (<NUM>) and provided with control means adapted to allow an operator to control a position of the working machine means (<NUM>, <NUM>),
wherein the working machine (<NUM>) further comprises:
- an image projection arrangement (<NUM>) for projecting an image (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) at a surface (<NUM>) of the working machine means (<NUM>, <NUM>) facing the operator compartment (<NUM>), wherein the projected image (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) comprises information for the operator to control the working machine (<NUM>),
characterized in that the image projection arrangement (<NUM>) further comprises a control unit (<NUM>) adapted to adjust a position of the image projected at the surface of the working machine means (<NUM>, <NUM>) based on an expected position of the operator within the operator compartment (<NUM>) and a current position of the working machine means (<NUM>, <NUM>) outside the operator compartment, wherein the expected position of the operator is determined using an eye and/or head tracking mechanism connected to the control unit.