Image Display System

An image display system includes a visual image system for generating image data from a plurality of points of view, an object detection system for detecting objects, and a machine sensor for sensing a state of the machine. A controller receives the image data, generates a unified image by combining the image data, and detects any objects in proximity to the machine. The controller further senses a state of the machine, determines an image to be rendered based upon the state of the machine and any detected objects, and renders the image on a display device.

TECHNICAL FIELD

This disclosure relates generally to an image display system, and more particularly, to a system and method for selecting and rendering an image based upon objects detected adjacent to a movable machine and a state of the machine.

BACKGROUND

Movable machines such as haul trucks, dozers, motor graders, excavators, wheel loaders, and other types of equipment are used to perform a variety of tasks. For example, these machines may be used to move material and/or alter work surfaces at a work site. The machines may perform operations such as digging, loosening, carrying, etc., different materials at the work site.

Due to the size and configuration of these machines, an operator may have a limited field of view with respect to the environment in which a machine is operating. Accordingly, some machines may be equipped with image processing systems including cameras. The cameras capture images of the environment around the machine, and the image processing system renders the images on a display within an operator station of the machine to increase the visibility around the machine.

While improving visibility, such image processing systems may not identify obstacles in the operating environment adjacent the machines. As a result, while an operator may monitor an image from the image processing system, the operator may not appreciate that obstacles are in proximity to the machine and, in particular, within a blind spot of the machine.

Some machines further include an object detection system having a plurality of sensors to sense objects that are adjacent the machine. Such an object detection system will typically provide a signal or an alarm if an object is detected that is within a predetermined distance from the machine. However, while operating a machine to perform a desired task, operators process a significant amount of information and, as a result, alarms and visual indicators of Obstacles are sometimes missed or ignored.

A system that may be used to improve visibility is disclosed in U.S. Patent Application Publication 2012/0262580, The system of the '580 Publication provides a surround view from a vehicle by way of cameras positioned at various locations on the vehicle. The cameras can generate image data corresponding to the surround view, and a processing device can process the image data and generate the surround view on a simulated predetermined shape that can be viewed from a display. The simulated predetermined shape can have a flat bottom with a rectangular shape and a rim with a parabolic shape. Although the system of the '580 Publication may increase visibility, it does not necessarily increase safety as the entire surround view is displayed.

The foregoing background discussion is intended solely to aid the reader, It is not intended to limit the innovations described herein, nor to limit or expand the prior art discussed. Thus, the foregoing discussion should not be taken to indicate that any particular element of a prior system is unsuitable for use with the innovations described herein, nor is it intended to indicate that any element is essential in implementing the innovations described herein, The implementations and application of the innovations described herein are defined by the appended claims.

SUMMARY

In one aspect, an image display system includes a visual image system mounted on a machine for generating image data from a plurality of points of view relative to the machine, an object detection system associated with the machine for detecting objects in proximity to the machine, and a machine sensor associated with the machine for sensing a state of the machine, A controller is configured to receive the image data from the visual image system, generate a unified image by combining the image data from the plurality of points of view, and detect any objects in proximity to the machine. The controller is further configured to sense a state of the machine, determine an image to be rendered based upon the state of the machine and any objects detected in proximity to the machine, and render the image on a visual image display device.

In another aspect, a method of operating an image display system includes receiving image data from a visual image system mounted on a machine for generating image data from a plurality of points of view relative to the machine, generating a unified image by combining the image data from the plurality of points of view, and detecting any objects in proximity to the machine. The method further includes sensing a state of the machine based upon a machine sensor associated with the machine, determining an image to be rendered based upon the state of the machine and any Objects detected in proximity to the machine, and rendering the image on a visual image display device.

In still another aspect, a machine includes a propulsion system, a visual image system mounted on the machine for generating image data from a plurality of points of view relative to the machine, an object detection system associated with the machine for detecting objects in proximity to the machine, and a machine sensor associated with the machine for sensing a state of the machine. A controller is configured to receive the image data from the visual image system, generate a unified image by combining the image data from the plurality of points of view, and detect any objects in proximity to the machine. The controller is further configured to sense a state of the machine, determine an image to be rendered based upon the state of the machine and any objects detected in proximity to the machine, and render the image on a visual image display device.

DETAILED DESCRIPTION

FIG. 1illustrates an exemplary work site100with a machine10operating at the work site. Work site100may include, for example, a mine site, a landfill, a quarry, a construction site, a road work site, or any other type of work site. Machine10may perform any of a plurality of desired operations or tasks at work site100, and such operations or tasks may require the machine to generally traverse work site100. Any number of machines10may simultaneously and cooperatively operate at work site100, as desired. Machine10may embody any type of machine. For example, machine10may embody a mobile machine such as the haul truck depicted inFIG. 1, a service truck, a wheel loader, a dozer, or another type of mobile machine known in the art.

Machine10may include, among other things, a body11supported by one or more traction devices12and a propulsion system for propelling the traction devices. The propulsion system may include a prime mover13, as shown generally by an arrow inFIG. 1indicating association with the machine10, and a transmission14, as shown generally by an arrow inFIG. 1indicating association with the machine10, operatively connected to the prime mover. Machine may include a cab or operator station that an operator may physically occupy and provide input to operate the machine, Referring toFIG. 2, operator station15may include an operator seat16, one or more input devices17through which the operator may issue commands to control the operation of the machine10such as the propulsion and steering as well as operate various implements associated with the machine. Operator station15may further include a visual image display device18such as fiat screen display.

Machine10may include a control system20, as shown generally by an arrow in Fig,1indicating association with the machine10. The control system20may utilize one or more sensors to provide data and input signals representative of various operating parameters of the machine10and the environment of the work site100at which the machine is operating, The control system20may include an electronic control module or controller21and a plurality of sensors associated with the machine10.

The controller21may be an electronic controller that operates in a logical fashion to perform operations, execute control algorithms, store and retrieve data and other desired operations. The controller21may include or access memory, secondary storage devices, processors, and any other components for running an application. The memory and secondary storage devices may be in the form of read-only memory (ROM) or random access memory (RAM) or integrated circuitry that is accessible by the controller, Various other circuits may be associated with the controller21such as power supply circuitry, signal conditioning circuitry, driver circuitry, and other types of circuitry.

The controller21may be a single controller or may include more than one controller disposed to control various functions and/or features of the machine10. The term “controller” is meant to be used in its broadest sense to include one or more controllers and/or microprocessors that may be associated with the machine10and that may cooperate in controlling various functions and operations of the machine. The functionality of the controller21may be implemented in hardware and/or software without regard to the functionality. The controller21may rely on one or more data maps relating to the operating conditions and the operating environment of the machine10and the work site100that may be stored in the memory of controller. Each of these data maps may include a collection of data in the form of tables, graphs, and/or equations.

The control system20may be located on the machine10and may also include components located remotely from the machine such as at a command center not shown). The functionality of control system20may be distributed so that certain functions are performed at machine10and other functions are performed remotely. In such case, the control system20may include a communications system such as wireless network system (not shown) for transmitting signals between the machine10and a system located remote from the machine.

Machine10may be equipped with a plurality of machine sensors22, as shown generally by an arrow inFIG. 1indicating association with the machine10, that provide data indicative (directly or indirectly) of various operating parameters of the machine and/or the operating environment in which the machine is operating. The term “sensor” is meant to be used in its broadest sense to include one or more sensors and related components that may be associated with the machine10and that may cooperate to sense various functions, operations, and operating characteristics of the machine and/or aspects attic environment in which the machine is operating.

A position sensing system23, as shown generally by an arrow inFIG. 1indicating association with the machine10, may include a position sensor24to sense a position of the machine relative to the work site100. The position sensor24may include a plurality of individual sensors that cooperate to provide signals to controller21to indicate the position of the machine10. In one example, the position sensor24may include one or more sensors that interact with a positioning system such as a global navigation satellite system or a global positioning system to operate as a position sensor. The controller21may determine the position of the machine10within work site100as well as the orientation of the machine such as its heading, pitch and roll. In other examples, the position sensor24may be an odometer or another wheel rotation-sensing sensor, a perception based system, or may use other systems such as lasers, sonar, or radar to determine the position of machine10.

In some instances, the operator station15may be positioned to minimize blind spots of machine10(i.e., maximize the unobstructed area viewable by an operator or operators of machine10). However, because of the size and configuration of some machines10, the blind spots may be relatively large. As a result, obstacles or objects may sometimes be located within a blind spot and thus not directly visible to an operator.

To increase the operator's field of view of the area surrounding the machine, machine10may include a visual image system25mounted on or associated with the machine, as shown generally by an arrow inFIG. 3indicating association with the machine10. The visual image system25may include a plurality of visual image sensors such as cameras26for generating image data. from a plurality of points of view relative to the machine10. The visual image system25may be used to display views of the environment around machine0on a visual image display device18within the operator station15of machine10.

Each camera26may be mounted on the machine10at a relatively high vantage point such as at the top of the frame of the machine or the roof. As depicted schematically inFIG. 3, four cameras26are provided that record or sense images in the forward and rearward directions as well as to each side of machine10. In the embodiment depicted inFIG. 1, the cameras26may be positioned in other locations but may face in the same directions as depicted inFIG. 3, Controller21may receive image data from the cameras26and generate video or still images based upon such images.

In some embodiments, controller21may combine the image data captured by the cameras26into a unified image120of a portion of the work site100adjacent and surrounding the machine10depicted.FIG. 4is a pictorial illustration of one example of controller21combining image data from each of the cameras26to generate the unified image120. The unified image120may represent all image data available for the environment of machine10. In one example, the unified image120represents a 360-degree view or model of the environment of machine10, with machine10at the center of the 360-degree view. According to some embodiments, the unified image120may be a non-rectangular shape. For example, the unified image120may be hemispherical and machine10may be conceptually located at the pole, and in the interior, of the hemisphere.

Controller21may generate the unified image120by mapping pixels of the image data captured by the cameras26to a pixel map. The pixel map may be divided into sections, with each section corresponding to one set of image data, For example, as shown inFIG. 3, front or first camera26acaptures image data that is mapped to section121, right or second camera26bcaptures image data that is mapped to section122, rear or third camera26ccaptures image data that is mapped to section123, and left or fourth camera26dcaptures image data that is mapped to section124. Pixels may be mapped directly using a one-to-one or one-to-many correspondence, and the mapping may correlate a two dimensional point from the image data to a three dimensional point on the map used to generate the unified image120, For example, a pixel of the image data located at (1,1) may be mapped to location (500, 500, 1) of the unified image. The mapping may be accomplished using a look-up table that may be stored within controller21, The look-up table may be configured based on the position and orientation of each camera26on machine10. Although a look-up table is one method by which controller21may map the image data to the unified image120, those skilled in the art will appreciate that other methods for mapping image data may be used to achieve the same effect.

Controller21may also use parameters associated with cameras26to map pixels from the image data to the unified image120. The parameters may be included metadata of the image data. For example, the parameters may include the position of each camera26with respect to machine10. Controller21may correlate sections121-124of the unified image120with machine10, and controller21may use the correlations to determine which of the image data to map to each section. For example, controller21may correlate section121with the front of machine10, When the controller receives image data from front or first camera26a,the parameters included in the metadata associated with such image data may indicate that it was captured by first camera26a.The parameters may also indicate that first camera26ais positioned on the front of machine10. Controller21may analyze the parameters and determine that certain image data should be mapped to section121. Thus, as controller21accesses the image data, it can correctly map it to sections121-124of the unified image120, Other manners of generating a unified image are contemplated.

Controller21may be configured to select a portion of the unified image120for rendering on visual image display device18within operator station15and/or another display (not shown). The portion may be selected using a designated viewpoint, The viewpoint125depicted inFIG. 3represents a plane from which the unified image120may be viewed, and the pixels located under the plane form the portion of the unified image120that controller21renders on visual image display device18. For example, as shown inFIG. 3, viewpoint125is positioned above the entire unified image120, and all of the pixels of the unified image are located under viewpoint125. With this designated viewpoint, the unified image is configured as a birds-eye or overhead view with the machine10centered therein and such image may be rendered on visual image display device18,

Other viewpoints may be used to generate an image to be displayed. For example, the viewpoint125may be shifted laterally relative to the unified image120to provide a larger field of view of one portion or side of the operating environment around the machine10. In such case, the controller21may render a shifted bird's eye view which is based upon the bird's eye view, but with the machine shifted relative to the unified image120. This may be desirable to emphasize the existence or details of objects detected on one or two sides of machine10. In another example, controller21may generate images from a single point of view or direction such as by displaying an image indicative of image data from only one camera26, Such viewpoint may be referred to as a directional view as it may correspond to a direction relative to the machine10. In some circumstances, a directional view may be generated by data from a. combination of two or more cameras26. In some instances, a directional view may correspond to a state of the machine (e.g., correspond to a direction that the machine is moving or a state of the transmission such as neutral, drive, or reverse).

While operating at work site100, machine10may encounter one or more obstacles101. Obstacle101may embody any type of object including those that are fixed or stationary as well as those that are movable or that are moving. Examples of fixed obstacles may include infrastructure, storage, and processing facilities, buildings, and other structures and fixtures found at a work site. Examples of movable obstacles may include machines, light duty vehicles (such as pick-up trucks and cars), personnel, and other items that may move about work site100.

To reduce the likelihood of a collision between machine10and an obstacle101, an object detection system30may be mounted on or associated with the machine, as shown generally by an arrow inFIG. 3indicating association with the machine10. The object detection system30may include a radar system, a SONAR system, a LIDAR system, and/or any other desired system together with associated Object detection sensors31. Object detection sensors31may generate data that is received by the controller21and used b the controller to determine the presence and position of obstacles101within the range of the sensors. The range of each object detection sensor31is depicted schematically inFIG. 3by reference number37.

An object identification system33may be mounted on or associated with the machine in addition to the object detection system30, as shown generally by an arrow inFIG. 3indicating association with the machine10. In some instances, the object detection system30and the object identification system33may be integrated together. Object identification sensors34may generate data that is received by the controller21and used by the controller to determine the type of obstacles detected by the object detection system30. The object identification sensors34may be part of or replace the object detection sensors and thus are depicted schematically as the same components inFIG. 3. In an alternate embodiment, the object identification sensors may be separate components from the object detection sensors31.

The object identification system33may operate to differentiate categories of object detected such as machines, light duty vehicles, personnel, or fixed objects.

in sonic instances, the object identification system33may operate to further identify the specific object or type of object detected.

Object identification system33may be any type of system that determines the type of object that is detected. In one embodiment, the object identification system33may embody a computer vision system that uses edge detection technology to identify the edges of a detected object and then matches the detected edges with known edges contained within a. data map or database to identify the object detected. Other types of object identification systems and methods of object identification are contemplated.

In an alternate or supplemental embodiment, controller21may include or access an electronic map of the work site100including the position of machine10and the positions of various known obstacles101at the work site. The object detection system30may utilize the electronic map of the work site100together with the position data of the machine from the position sensing system23to determine the proximity of the machine to any obstacles101. The electronic map of the work site100may also include the type of object in addition to its location and the object identification system33may use this information to determine the type of obstacle101at the work site.

Still further, the object identification sensors34may comprise RFID sensors and certain objects or obstacles101at the work site100may be equipped with MD chips or tags (not shown). The object identification system33may be configured to read RFID chips of any obstacles101that are within a predetermined range to identify such obstacles.

Visual image system25and object detection system30may operate together to define an image display system35, as shown generally by an arrow inFIG. 3indicating association with the machine10. Object identification system33, if present, may also operate as a part of the image display system35.

Referring toFIG. 5, a flowchart of the operation of the image display system35is depicted. During the operation of machine10, cameras26generate image data at stage40and controller21receives at stage41the image data from the cameras. Inasmuch as the cameras26face in a multiple directions, image data may be generated depicting the operating environment surrounding the machine10. The image data may include images captured by cameras26, as well metadata including parameters associated with each of cameras26. The parameters may describe the orientation of each camera26, the position of each camera with respect to machine10, and the range of each camera's field of view.

At stage42, controller21may use the image data to generate a unified image120of the operating environment of machine10by combining the image data generated by the cameras26as described in more detail above. Controller21may receive at stage43state data from various machine sensors associated with the machine10. For example, the state data may include the direction of travel and the speed of movement of the machine as well as the gear or setting of the transmission14. The controller21may determine at stage44, the state of the transmission14.

Once controller21has generated the unified image120, the controller may select and generate at stage45a view based upon a portion of the unified image120, a directional view from one or more of the cameras26, or some other image to be rendered on the visual image display device18. The controller21may select the image to be rendered based upon a plurality of factors including the number of and proximity to any objects detected adjacent the machine10, the state of the transmission14, and the identity of any objects detected. At stage46, controller21may render the image on the visual image display device18.

FIGS. 6-8depict examples of processes used by the image display system35to select the views at stage45based upon the state of the transmission14.FIG. 6depicts an example of a process while the transmission is in neutral or park,FIG. 7depicts an example of a process while the transmission is in drive or a forward gear, andFIG. 8depicts an example of a process while the transmission is in reverse.

Referring toFIG. 6, with the transmission14in neutral or park, object detection sensors31generate data and controller21receives at stage50the data from the object detection sensors. At stage51, the controller51determines whether any objects are detected within the range of object detection sensors31. If no objects are detected, the controller may generate at stage52a bird's eye or overhead view of the area surrounding the machine10that depicts the machine centered within the bird's eye view. The bird's eye view with the machine10centered therein is referred to herein as a standard bird's eye view.

If objects are detected at stage51, the controller21may at stage53determine the distance from any detected objects to the machine10. If the controller21determines at stage54that two or more objects are within a predetermined range from the machine10, the controller may generate at stage52a standard bird's eye view of the area surrounding the machine10. Such bird's eye view will permit an operator within operator station15to see all of the Obstacles within the predetermined range and their proximity to machine10. In some instances, it may be desirable to zoom the standard bird's eye view to some extent while still maintaining all of the objects within the image.

If the controller21determines at stage54that two or more objects are not within a predetermined range from the machine10, the controller may determine at stage55whether a single object is within the predetermined range from the machine. no objects are detected within the predetermined range, the controller21may generate at stage52a standard bird's eye view of the area surrounding the machine10. If only one object is detected within the predetermined range from the machine10, the controller21may generate at stage56a modified image such as a shifted bird's eye view in which the bird's eye view is shifted towards the object and the machine10is no longer centered within the view. In an alternate embodiment, the controller21may generate a directional view in which the images from one or more cameras26are rendered on visual image display device18.

Once the controller21determines the image to be displayed and generates such image at stages52or56, the controller21may render the image on visual image display device18at stage46,

FIG. 7depicts a process for selecting a view to be generated while the transmission14is in drive or a forward gear. When moving the machine10forward, an operator typically has some range of view from the operator station15such that objects in front of the machine that are relatively far away are visible. In other words, while objects that are very close to the machine10may be within a blind spot, objects that are in front of the machine but farther away are likely to be visible to an operator within the operator station15. As the machine10moves forward, the operator will likely be aware of objects in front of the machine based upon the operator's memory even if such objects are in a blind spot. However, if the machine has been stationary or moving in reverse, it is possible that one or more movable objects may have moved into a blind spot in front of the machine without the knowledge of the operator. Accordingly, when the transmission14is initially shifted into drive or a forward gear, it may be desirable to provide additional assistance to the operator by displaying any objects that are in proximity to or adjacent the machine10.

At stage60, the controller21determines whether the transmission14was recently shifted into a drive or a forward gear. As used herein, recently may refer to a predetermined period of time or a predetermined distance the machine10has traveled since being shifted into drive or a forward gear. In one example, the predetermined distance may be two to five meters, If the transmission14was recently shifted into a drive or a forward gear, a movable object may have moved into a blind spot of the machine10while the machine was stationary or in a reverse gear. In either case, the operator may not be aware of the object in the blind spot. Accordingly, if the transmission14was recently shifted into a drive or a forward gear, the controller21may generate at stage61a bird's eye view and display or render the bird's eye view on visual image display device18at stage62.

If the transmission14was not recently shifted into drive or a forward gear, the object detection sensors31generate data and controller21receives at stage63the data from the object detection sensors. At stage64, the controller51determines whether any Objects are within the range of object detection sensors31. If no objects are detected, the controller21may generate at stage65a front directional view including an image from the front or first camera26a.If desired, images from the right or second camera26band the left or fourth camera26dmay be combined with the image from the first camera26ato expand the field of

If one or more objects are detected at stage64, the controller21may generate at stage66a standard bird's eye view of the area surrounding the machine10. Such bird's eye view will permit an operator within operator station15to see all of the obstacles and provide some degree of spatial relationship between the detected object or objects and the machine10. In some instances, it may be desirable to zoom the standard bird's eye view to some extent while still maintaining all of the detected objects within the image.

Once the controller21determines the image to be displayed and generates such image at stages65or66, the controller21may render the image on visual image display device18at stage46.

FIG. 8depicts the process for selecting a view to be generated while the transmission14is in reverse. Object detection sensors31generate data and controller21receives at stage70the data from the object detection sensors. At stage71, the controller51determines whether any objects are detected within the range of the object detection sensors31. If no objects are detected, the controller may generate at stage72a rear directional view including an image from the rear or third camera26c.If desired, images from the right or second camera26band the left or fourth camera26dmay be combined with the image from the rear or third camera26cto expand the field of vision.

If objects are detected at stage71, the controller21may at s age73determine the distance from any detected objects to the machine10.

If the controller21determines at stage74that two or more objects are within a predetermined range from the machine10, the controller may generate at stage75a standard bird's eye view of the area surrounding the machine10. Such bird's eye view will permit an operator within operator station15to see all of the obstacles within the predetermined range and their proximity to machine10. In some instances, it may be desirable to zoom the standard bird's eye view to some extent while still maintaining all of the objects within the view.

If the controller21determines at stage74that two or more objects are not within a predetermined range from the machine10, the controller may determine at stage76whether a single object is within the predetermined range from the machine. If no objects are detected within the predetermined range, the controller21may generate at stage72a rear directional view from the rear or third camera26c.if desired, images from the right or second camera Nib and the left or fourth camera26dmay be combined with the image from the rear or third camera26cto expand the field of vision.

If one object is detected within the predetermined range from the machine10, the controller21may determine at stage77whether the object is closer than a predetermined threshold. If the object is not closer than the predetermined threshold, the controller21may continue to generate at stage72the rear directional view. If the machine10continues to move rearwardly towards the object or the relative distance between the machine and the object otherwise is decreased to less than the predetermined threshold, the controller21may generate a modified image such as a shifted bird's eye view in which the bird's eye view is shifted towards the detected object.

Once the controller21determines the image to be displayed and generates such image at stages72,75, or78, the controller21may render the image on visual image display device18at stage46.

In some instances, even if the controller21determines that more than one object has been detected, the processes depicted inFIGS. 6-8may be performed as if only a single object was detected. More specifically, the controller21may analyze the positions of the plurality of detected objects to determine whether the detected objects are within a predetermined field of view or a predetermined distance from each other. If all of the objects detected are within a predetermined field of view or close enough together, the view selection process may operate as if only a single object were detected. This may occur, for example, if the objects are close enough together that they are all visible with a directional view from cameras26.

The image to be selected may also be dependent on the state of the machine10and/or the state of the detected objects. More specifically, the controller21may monitor the speed and direction of movement of the machine10as well as the speed and direction of movement of any detected objects and use such information to determine which views to select. In one example, if relative movement of the machine10is away from a detected object, the controller21may be configured to disregard the detected object and the view selection process proceeds as if no objects were detected. This may occur is the machine10is moving and the detected object is stationary, the machine is stationary and the detected object is moving, or both are moving in such a manner that that results in relative movement away from each other. In an example in which two Objects are detected, the controller21may disregard the detected object that is moving relatively away from the machine10so that the view selection process operates as if only one object were detected. In still another example, the relative speeds between a detected object and machine10may be monitored so that the view selection process may disregard a detected object if it is passing by machine10relatively quickly and the object is at least a predetermined distance away.

If the image display system35includes an object identification system33, the view selection process may also use the identification of the detected objects to determine the view to be selected. For example, the controller21may select different views depending on whether the detected objects are fixed or movable obstacles and whether any movable obstacles are machines, light duty vehicles or personnel.

In addition, controller21may be configured to add additional detail to a rendered image such as an overlay based upon the type of Object detected and the distance to such object. For example, different color overlays may be used depending on the type of object detected and the color may change depending on the distance to such object, If desired, aspects of the overlay may also flash or change to provide an additional visual warning to an operator.

Overlays may also be task-based to assist in operating machine10such as by rendering a target position and a target path to assist an operator in completing a desired task, In one example, an overlay may be used to assist in positioning a haul truck for loading by a wheel loader. In such case, the object detection system30and the object identification system33may detect and identify the wheel loader. The image display system35may render a rear directional view on visual image display device18that includes images from the rearwardly facing third camera26cas well as the second camera26band the fourth camera26d.An overlay may be depicted or rendered on the visual image display device18highlighting certain components of the wheel loader and a target position for the haul truck as well projecting a desired path of the haul truck. If desired, once the haul truck is within a predetermined distance from the wheel loader, the depicted view may change to a shifted bird's eye view to assist in aligning the haul truck and the wheel loader along multiple axes.

INDUSTRIAL APPLICABILITY

The industrial applicability of the system described herein will be readily appreciated from the foregoing discussion. The foregoing discussion is applicable to machines10that are operated at a work site100and include an image display system35. The image display system35may be used at a mining site, a landfill, a quarry, a construction site, a roadwork site, a forest, a farm, or any other area in which it is desired to improve the visibility of a machine operator.

The image display system35may include a visual image system25mounted on a machine10for generating image data from a plurality of points of view relative to the machine and an object detection system30associated with the machine for detecting Objects in proximity to the machine. In addition, a plurality of machine sensors22may be associated with the machine10for sensing a state of the machine. The controller21may be configured to receive image data from the visual image system25and generate a unified image120by combining the image data from the plurality of points of view, determine an image to be displayed based upon the state of the machine10and any objects detected in proximity to the machine, and render the image on a visual image display device18.

Image display system35provides a system to enhance the awareness of an operator of machine10to objects adjacent the machine. A unified image120is generated and an image to be rendered is determined base upon based upon the state of the machine10and any objects detected in proximity to the machine. In one example, the image to be rendered is based upon the state of the transmission14of the machine10.