Machine control system utilizing stereo disparity density

A control system for a mobile machine is disclosed. The control system may have a first sensor mounted on the mobile machine and configured to capture a first image of a region near the mobile machine, a second sensor mounted on the mobile machine and configured to capture a second image of the region, and a controller in communication with the first and second sensors. The controller may be configured to generate a stereo image from the first and second images, compute a disparity map of the stereo image, and generate an output to affect operation of the machine when a density of the disparity map is less than a threshold density.

TECHNICAL FIELD

The present disclosure relates generally to a machine control system and, more particularly, to a system that utilizes stereo disparity density to control operations of a mobile machine.

BACKGROUND

Machines such as off-highway haul trucks, motor graders, snow plows, and other types of heavy equipment are used to perform a variety of tasks. Some of these tasks involve carrying or pushing large, awkward, loose, and/or heavy loads up steep inclines or along rough or poorly marked haul roads. Because of the size and momentum of the machines and/or because of poor visibility, these tasks can be difficult for a human operator alone to complete effectively.

To help guide the machines safely and efficiently along the haul roads, some machines are equipped with sensors, for example cameras, located on a front end of each machine. These sensors are often connected to a visual display and/or a guidance system of the machine such that control over machine maneuvering may be enhanced or even automated by two-dimensional images provided by the sensors.

When multiple two-dimensional sensors scan the same region from different positions onboard the machine, differences in the resulting images can be used to determine three-dimensional aspects of the region. That is, objects at different distances from the sensors project images to the sensors that differ in their positions and/or size, giving the depth cue known as disparity. By matching particular features (e.g., pixels, boundary lines, etc.) from the images produced by each sensor, and then comparing the disparity between the matched features, the size, location, and orientation of the matched features in the scanned region can be determined, processed, and used to simulate a three-dimensional environment.

A quality of the simulated three-dimensional environment can be represented by a number of features that are matched between the two images and subsequently used for disparity calculations. This quality parameter is known as a disparity density. When the disparity density is high (i.e., when many of the features from each sensor's image are matched), it can be concluded that both sensors are producing accurate images of the same object or region. When the disparity density is low, it can be concluded that one or both of the sensors are experiencing some kind of impairment. The impairments can include, among other things, rain, snow, dust, fog, debris, etc. When one or both of the sensors are impaired, reliance on the simulated environment for machine control may not be appropriate.

U.S. Patent Publication No. 2009/0180682 (the '682 publication) of Camus published on Jun. 16, 2009 discloses a system and method for ensuring that only good stereo images are processed and used for machine control based on disparity calculations. Specifically, the '682 publication describes capturing images from a left camera and a right camera, and producing a single stereo disparity image from the two captured images. The stereo disparity image is then divided into three parts, including a left third, a center third, and a right third. Each third of the stereo disparity image is then scrutinized to determine a disparity measure representing a quality of the stereo disparity image. To compute the disparity measure, a number of edge discontinuities between adjacent regions in each image third are summed and subtracted from a number of valid image pixels, then divided by a total number of image pixels in the image third. Based on the disparity measure, a disparity algorithm defines the image as valid or invalid. A small number of large cohesive disparity regions will increase the disparity measure, while a larger number of small, fragmented regions will decrease the disparity measure. If the stereo disparity image is determined to be valid (i.e., if the disparity measure falls within a specific threshold), the disparity image is further processed for object and collision detection. However, if the disparity image is determined to be invalid (i.e., if the disparity measure falls outside the threshold value), the disparity image is ignored and new left and right images are obtained from each of the cameras to repeat the process.

Although the method of the '682 publication may help ensure that machine control is not implemented based on images from an impaired camera, the method may do little to improve the image produced by the impaired camera or to affect machine control differently when the camera is impaired. Instead, the system of the '682 publication may simply slow down or stop working altogether when one or both cameras becomes impaired.

The disclosed control system is directed to overcoming one or more of the problems set forth above and/or other problems of the prior art.

SUMMARY OF THE INVENTION

In one aspect, the present disclosure is directed to a control system for a mobile machine. The control system may include a first sensor mounted on the mobile machine and configured to capture a first image of a region near the mobile machine, a second sensor mounted on the mobile machine and configured to capture a second image of the region, and a controller in communication with the first and second sensors. The controller may be configured to generate a stereo image from the first and second images, compute a disparity map of the stereo image, and generate an output to affect operation of the machine when a density of the disparity map is less than a threshold density.

In yet another aspect, the present disclosure is directed to a method of controlling a mobile machine. The method may include capturing a first image of a region near the mobile machine from a first location onboard the mobile machine, capturing a second image of the region from a second location onboard the mobile machine, and generating a stereo image from the first and second images. The method may also include computing a disparity map of the stereo image, and generating an output to affect operation of the mobile machine when a density of the disparity map is less than a threshold density.

DETAILED DESCRIPTION

FIG. 1illustrates a worksite10and an exemplary machine12performing a specialized task at worksite10. Worksite10may include, for example, a mine site, a landfill, a quarry, a construction site, or any other type of worksite having a roadway14traversable by machine12. Although shown inFIG. 1as a single lane roadway, it is contemplated that roadway14may alternatively include multiple lanes or an open excavation surface, if desired.

The task being performed by machine12may be associated with altering the geography at worksite10and may include, for example, a hauling operation, a grading operation, a leveling operation, a plowing operation, a bulk material removal operation, or any other type of operation. As such, machine12may embody a mobile machine, for example a haul truck, a motor grader, a loader, or a snow plow. Machine12may include, among other things, a power source20, one or more traction devices22, and a control system24. Power source20may generate and provide power to traction devices22to propel machine12, while control system24may selectively affect operations of machine12in response to various input.

Power source20may embody an internal combustion engine such as, for example, a diesel engine, a gasoline engine, a gaseous fuel powered engine, or any other type of engine apparent to one skilled in the art. Power source20may alternatively or additionally include a non-combustion source of power such as a fuel cell, a power storage device, an electric motor, or other similar mechanism. Power source20may be connected to drive traction devices22via a direct mechanical coupling, a hydraulic circuit, an electrical circuit, or in any other suitable manner.

Traction device22may be a wheel, a belt, a track or any other driven traction device known in the art. Traction device22may be driven by power source20to rotate and propel machine12in accordance with an output rotation of power source20. A steering device26, for example a hydraulic cylinder, a hydraulic motor, an electric motor, and/or a rack-and-pinion configuration may be associated with one or more traction device22to affect steering thereof. In addition, a braking mechanism28, for example a compression disk brake, an internal fluid brake, an engine retarder, an exhaust brake, and/or a transmission brake may be associated with one or more traction device22and/or power source20to affect braking of machine12.

Control system24may include multiple components that interact to affect operations of machine12. Specifically, control system24may include two or more sensors30, a display32, a warning device33, a communications device35, and a controller34. Controller34may be in communication with power source20, sensors30, display32, warning device33, communication device35, steering device26, and braking mechanism28, and be configured to automatically control maneuvering (i.e., steering, fueling, and/or braking) of machine12and/or provide warnings and recommendations to an operator of machine12or to operators of other machines12at worksite10based on input received from sensors30and/or from an operator of machine12.

Sensors30may be attached to a body36of machine12to capture an image of a region near machine12in a travel direction (e.g., a region substantially forward of machine12). Each sensor30may embody an image scanning device, for example a camera.FIG. 1illustrates a left camera30L being spaced horizontally apart from a right camera30R. It is contemplated, however, that sensors30may alternatively or additionally be spaced apart from each in a vertical direction, if desired, and that any number of sensors30may be attached to machine12. Each of left and right cameras30L,30R may be configured to capture an image that includes at least a portion of the region in front of machine12from a different location onboard machine12, and send the images to controller34for further processing. Because left and right cameras30L,30R may be positioned at different locations onboard machine12, disparities may exist between overlapping portions of the images captured by left and right cameras30L,30R. As will be described in more detail below, these disparities may be used by controller34to generate a stereo (i.e., 3-D) image from the images captured by left and right cameras30L,30R.

Display32may be any appropriate type of device that provides a graphics user interface (GUI) for presentation of results and information to operators and other users of machine12. For example, display32may be a computer console or cab-mounted monitor.

Warning device33, when activated by controller34, may provide a warning and/or recommendation to an operator of machine12. The warning and recommendation may be audible, visual, or a combination of both audible and visual stimulation.

Communications device35may embody any mechanism that facilitates the exchange of data between machines12and/or between machine12and an offboard device, for example a site controller or database. Communications device35may include hardware and/or software that enables each machine12to send and/or receive data messages through a direct data link (not shown) or a wireless communication link. The wireless communications may include, for example, satellite, cellular, infrared, and any other type of wireless communications that enable machines12to wirelessly exchange information.

Controller34may embody a single or multiple microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), etc. that include components for affecting an operation of machine12in response to images received from sensors30. Numerous commercially available microprocessors can be configured to perform the functions of controller34. It should be appreciated that controller34could readily embody a microprocessor separate from that controlling other machine functions, or that controller34could be integral with a general machine microprocessor and be capable of controlling numerous machine functions and modes of operation. If separate from the general machine microprocessor, controller34may communicate with the general machine microprocessor via datalinks or other methods. Various other known circuits may be associated with controller34, including power supply circuitry, signal-conditioning circuitry, actuator driver circuitry (i.e., circuitry powering solenoids, motors, or piezo actuators), and communication circuitry.

As described above, controller34may be configured to generate a stereo image from the overlapping portions of the separate images captured by left and right cameras30L,30R. The stereo image may include three-dimensional depth information that can be reconstructed from the two captured images using one or more predetermined algorithms. The pre-determined algorithms may function by mathematically relating a known spacing on machine12between left and right cameras30L,30R with disparities in common pixels (e.g., pixels from each image that correspond with the same real world object) that have been matched from the left and right images. This process of generating the stereo image is well known in the art and will not be described in detail in this disclosure.

After controller34generates the stereo image, a quality of the stereo image may be determined. In one embodiment, the quality of the stereo image may be determined by computing a disparity map of the stereo image (i.e., a map of disparities associated with each common pixel location that has been matched between the captured images), determining a density of the disparities within the map, and comparing the disparity density to a threshold density. For the purposes of this disclosure, disparity density may be considered a measure of the number of common pixels that have been matched between the images captured by left and right cameras30L,30R. When the disparity density of the disparity map (or particular regions of interest within the map) is less than the threshold density, it may be concluded that the stereo image (or the corresponding particular region within the image) is of poor quality and may not be trustworthy. When the computed disparity density is equal to or greater than the threshold density, it may be concluded that the stereo image (or the particular region) is of acceptable quality and can be trusted.

In one embodiment, the threshold density used to determine the quality of the stereo image may be adjustable. In particular, the threshold density may be adjusted based on a travel speed of machine12, a type of machine12, a condition of machine12and/or worksite10, a characteristic or preference of an owner/operator of machine12, or based on any other condition known in the art. For example, as a speed, a size, or a load of machine12increases, the threshold density may likewise increase. Similarly, as road conditions at worksite10degrade, the threshold density may again increase. Additionally, the threshold density may increase or decrease by an amount related to a condition of machine12and/or an operator's experience or ability with regard to machine12. In general, any change in the conditions of machine12, worksite10, and/or the operator of machine12that makes control of machine12more difficult, may result in an increase in the threshold density. The threshold density may be manually adjusted or automatically adjusted by controller34in response to various input. In this way, potential risk associated with operation of machine12under difficult conditions may be reduced.

It is contemplated that the stereo image produced by controller34may be divided into multiple zones. For example,FIG. 2illustrates a stereo image generated by controller34that has been divided by a 3×3 grid into nine different zones. It should be noted that any number of zones may be utilized. Controller34may compute a disparity map and associated disparity density of each individual zone to determine a quality of the stereo image in that zone. Controller34may then compare the disparity density of each zone to the threshold density.

In one embodiment, particular zones of the stereo image may be more important than other zones. For example, the zone shown inFIG. 2as corresponding with grid position2-1(i.e., column2, row1) may represent an area directly in front of machine12that will immediately be traversed by machine12. If this zone contained an obstacle, immediate maneuvers may be required for machine12to avoid the obstacle and, accordingly, this zone may be classified by controller34as a critical zone. In contrast, however, the zone corresponding with grid position1-3may represent an area above machine12and to the left of roadway14. This zone may never be entered by any portion of machine12during travel of machine12and, accordingly, may be classified by controller34as non-critical. The number, size, and criticality of each zone may be selected by an operator of machine12or by controller34based on known geometry and operating conditions of machine12(e.g., based on a travel speed or direction of machine12), and changed at any time.

The threshold density for each zone of the stereo image may have a value corresponding to the importance of the zone. For example, the zone corresponding with grid position2-1, which was classified by controller34as being a critical zone, may have a higher threshold density than the non-critical zone corresponding with grid position1-3. It is contemplated that the threshold density value may be manually assigned to each zone or, alternatively, automatically assigned based on a criticality of each zone.

When it is determined by controller34that the stereo image (or a region of interest within the image) is of sufficient quality (i.e., when the disparity density at a corresponding location in the disparity map is equal to or greater than the threshold density), controller34may regulate operations of machine12and/or display the stereo image to an operator of machine12according to preprogrammed instructions. These operations may include, among other things, autonomous or semi-autonomous control over fueling, steering, braking, etc., of machine12.

When, however, it is determined by controller34that the stereo image is of poor quality (i.e., when the disparity density at the corresponding location in the disparity map is less than the threshold density), controller34may generate an output to affect operation of machine12in a manner different than when the stereo image is of acceptable quality. A poor quality image can be caused by a number of different impairments to one or both of left and right cameras30L,30R. When one or both of left and right cameras30L,30R are impaired, the number of common pixels that are matched between the images captured by left and right cameras30L,30R will be low. Typical impairments can include, for example, obstructions blocking one or both of cameras30L,30R such as air-borne dust (represented by element38inFIG. 2), smoke, precipitation (represented by element40inFIG. 2), fog (represented by element42inFIG. 2), debris, etc. These impairments may correspond with low visibility from an operator's perspective on machine12. Other impairments may alternatively correspond with sensor malfunction.

Controller34may generate an output to affect operation of machine12when the disparity density of the disparity map or the disparity density of a particular zone within the disparity map is less than the threshold density. In one example, the output may be associated with machine control. Specifically, controller34, in response to a low disparity density value that could potentially correspond with a low visibility situation, may automatically activate headlights of machine12, reduce a speed of machine12via braking mechanism28, or automatically steer machine12away from the region of low visibility via steering mechanism26. Alternatively or additionally, controller34may provide a message to an operator of machine12or to operators of other machines12at worksite10warning of the potential for low visibility or regarding recommended actions to change a control parameter such as braking or steering. In addition, controller34may display representations of the stereo image, the disparity map, and/or corresponding visibility measures on display33within machine12.

Controller34may be further configured to receive information from other machines12operating at worksite10. In particular, controller34may receive via communications device35messages from other machine controllers34regarding the potential for low visibility or recommend actions to change control parameters. In addition, it is contemplated that a centrally-located site controller (not shown) may receive messages from the controller34of each machine12operating at worksite10, and generate a general site map, if desired, the site map showing all locations of potential low visibility. This site controller could then communicate the site map to each of the different machines12at worksite10for use in control of individual machines12.

It is further contemplated that portions of a stereo image or the disparity map generated by the controller34of one machine12operating at worksite10may be selectively communicated to the controller34of another machine12also operating at worksite10. For example, if multiple similar machines12are traveling in a column along the same roadway14, and a sensor(s)30of a trailing machine12experiences an impairment at a particular location on roadway14that results in low disparity density within a particular zone of the disparity map, it may be possible for a leading machine12having already passed through that particular location without any sensor impairment to provide a previously-recorded replacement stereo image or disparity map corresponding with the zone having a low disparity density (i.e., a zone having a disparity density less than a corresponding threshold density). In this manner, the trailing machine12may be provided with or be able to create a high quality stereo image that can be used to effectively control operations of machine12, even though that particular machine12may be currently experiencing a sensor impairment. This situation may occur, for example, when dust blows across roadway14behind the leading machine12but in front of the trailing machine12, thereby causing low visibility for only the trailing machine12. Similar information may be communicated between machines12that are at different locations within or are approaching from a different direction a storm, fog, or smoke-filled pass.

FIG. 3illustrates an exemplary method performed by controller34during operation of machine12.FIG. 3will be described in more detail in the follow section to better illustrate the disclosed system and its operation

INDUSTRIAL APPLICABILITY

The disclosed control system may be applicable to any mobile machine where quality of a provided three-dimensional representation of machine environment is important. The disclosed control system may generate the three-dimensional representation and check the representation for quality. When a low quality representation is detected, the disclosed control system may provide warnings, control recommendations, and autonomous machine maneuvering. In addition, the disclosed control system may communicate with other co-located machines to improve the representation. Operation of control system24will now be described.

As machine12is traveling along roadway14, control system24may continuously receive images captured by left and right cameras30L,30R (Step100). From the captured images, and more specifically from disparities between overlapping portions of the captured images, and based on the known geometrical relationship between the locations of left and right cameras30L,30R, controller34may generate a three-dimensional stereo image. Controller34may also compute a corresponding map of the disparities (Step110), and then determine a density of the disparities within the map (Step120).

Controller34may compare the disparity density of the disparities in the map with a threshold density (Step130). As described above, the stereo image may be divided into zones and, as such, controller34may be configured to compare the density of disparities within a particular zone, with a threshold density defined for that zone. If the density of disparities within the particular zone of the disparity map is sufficiently high (e.g., greater than the threshold density) (Step130: No), controller34may control machine12according to one or more pre-programmed instructions (Step140). The stereo image may be presented on display32(Step150), and control may return to step100to repeat the process.

If, however, at step130, it is determined that the density of disparities within the particular zone of the disparity map is low (i.e., less than the threshold density) (Step130: Yes), controller34may conclude that sensor impairment exists, and responsively affect machine operations (Step160). For example, in the stereo image ofFIG. 2, dust38is shown in the lower right zone corresponding to grid location3-1, precipitation40is shown in the upper left zone corresponding to grid location1-3, and fog42is shown in the middle zone corresponding to grid location2-2. These impairments may negatively affect the image(s) captured by one or both of left and right cameras30L,30R, causing a loss in the number of matches between common pixels of the two images. Because of the impairments and corresponding low number of pixel matches, the stereo image in these areas may be low quality and a confidence that the stereo image accurately represents the real world environment may also be low.

Controller34may implement a number of different actions based on the low disparity density and based on the particular zone in which the low disparity density is detected. For example, because the illustrated impairments may be affecting only non-critical zones inFIG. 2, for example areas that are inaccessible by machine12or a long distance from machine12, controller34may only provide a warning to the operator of machine12regarding the potential for low visibility. Additionally or alternatively, controller may provide a recommendation to the operator, for example to slow down or to turn on headlights. If however, the impairment was detected within a critical zone such as the zone corresponding to grid location2-1, controller34may be configured to activate braking mechanism28and slow machine12to an acceptable speed corresponding to the visibility with the critical zone or to activate steering mechanism26to avoid the critical zone altogether. In addition, controller34may communicate information to other machines12or to a site controller regarding the potential for low visibility in zones3-1,1-3, and2-2. This information may be used to control machines12, update a site map, and/or implement actions to improve visibility (e.g., by arranging for a water truck to spray roadway14and thereby reduce the dust in particular areas of worksite10).

In addition to affecting operations of machine12based on the disparity density of the disparity map, controller34may also or alternatively attempt to improve the stereo image. In particular, controller34may communicate with other machines12co-located at worksite10and/or with a site controller to receive high quality portions of the stereo image generated by those other machines12and that correspond with the zones of low disparity density. For example, machine12, as it moves through a particular location at worksite10may encounter blowing dust that impairs one or both of left and right cameras30L,30R. At this same location during an earlier trip, the same or a different machine may have passed through without any impairment and produced and recorded a high quality stereo image. Accordingly, controller34of machine12may determine which zones of the current stereo image have low disparity density, and replace those portions of the stereo image with portions of a previously-generated stereo image having high disparity density (Step170). In this manner, controller34may improve the current stereo image by replacing low-quality portions of the image with higher-quality data from a older stereo image. It is contemplated that, in some situations, only non-critical zones having low disparity density may be replaced. Once the stereo image has been improved with replacement zones, the improved stereo image may be presented on display33(Step150), and control may return to step100to repeat the process.

Several benefits may be associated with the disclosed control system. For example, the disclosed control system may allow operation of machine12during impairment of sensors30. In addition, the disclosed control system may still be able to generate a high quality stereo image, even when sensors30are impaired. Further, the disclosed control system may facilitate communications between machines12and/or between a site controller and machines12that improve site coordination and conditions.

It will be apparent to those skilled in the art that various modifications and variations can be made to the control system of the present disclosure. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the control system disclosed herein. For example, although this disclosure describes creation of a stereo image before or simultaneous with creation of a disparity map, it is contemplated that the disparity map may alternatively be created before the stereo image, if desired. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.