SYSTEM AND METHOD FOR ENVIRONMENT RECOGNITION

An environment recognition system for a machine operating at a worksite is provided. A processing device of the environment recognition system receives a plurality of data points from at least one perception sensor. The processing device generates an environment map and detects a plurality of objects. Further, the processing device extracts a geometry and computes an expected shadow of each of the plurality of detected objects. The processing device detects one or more missing data points indicative of a casted shadow of the respective detected object. The processing device computes a geometry of the casted shadow and compares the casted shadow with the expected shadow of the respective detected object. The processing device determines whether the geometry of any of the plurality of detected objects has been misestimated based on the comparison of the casted shadow with the expected shadow of the respective detected object.

TECHNICAL FIELD

The present disclosure relates to an environment recognition system. More particularly, the present disclosure relates to the environment recognition system associated with a machine operating at a worksite.

BACKGROUND

Movable machines such as rotary drills, haul trucks, dozers, motor graders, excavators, wheel loaders, and other types of equipment are used to perform a variety of tasks. For example, these machines may be used to move material and/or alter work surfaces at a work site. The machines may perform operations such as drilling, digging, loosening, carrying, etc., different materials at the worksite.

Some of the machines such as autonomous blast holes drills may need to be able to navigate efficiently on benches. To achieve such capabilities a map of an environment may be built based on inputs from a perception system. However, the perception system may have a limited vantage point of the environment so the map may contains areas having missing data due to occlusions, poor data density and accuracy at range. Furthermore, sensors associated with the perception system may have limited and discrete resolution. Due to limited sensor range, object sizes within the map may be underestimated. The objects having less reflective surfaces, such as flat black planes, may also produce gaps in the data. Further, due to such missing data path planning algorithms, motion prediction of moving objects, and the completeness of the environment model may be constrained.

U.S. Pat. No. 8,842,036 describes a method, a radar image registration manager, and a set of instructions. A primary sensor interface receives a primary sensor image and a camera model of the primary sensor image. A data storage stores a digital elevation model. A processor automatically aligns the primary sensor image with the digital elevation model.

However, known systems using perception sensors to estimate the environment continue to contain missing data due to sensor limitations and object occlusion. Hence there is a need for an improved system for environment recognition.

SUMMARY OF THE DISCLOSURE

In an aspect of the present disclosure, an environment recognition system for a machine operating at a worksite is provided. The environment recognition system includes at least one perception sensor associated with the machine. The at least one perception sensor is configured to output a plurality of data points corresponding to an environment around the machine. A processing device is communicably coupled to the at least one perception sensor. The processing device is configured to receive the plurality of data points from the at least one perception sensor. The processing device is configured to generate an environment map based on the received plurality of data points. The processing device is configured to detect a plurality of objects within the generated environment map. Further, the processing device is configured to extract a geometry of each of the plurality of detected objects. The processing device is configured to compute an expected shadow of each of the plurality of detected objects based on the extracted geometry. The processing device is configured to detect one or more missing data points in the generated environment map. The one or more missing data points are indicative of a casted shadow of the respective detected object. The processing device is configured to compute a geometry of the casted shadow of the respective detected object. The processing device is configured to compare the casted shadow with the expected shadow of the respective detected object. The processing device is configured to determine whether the geometry of any of the plurality of detected objects has been misestimated based on the comparison of the casted shadow with the expected shadow of the respective detected object.

In another aspect of the present disclosure, a method for environment recognition associated with a machine operating on a worksite is provided. The method includes receiving a plurality of one or more data points from at least one perception sensor. The method includes generating an environment map based on the received plurality of one or more data points. The method includes detecting a plurality of objects within the generated environment map. The method includes extracting a geometry of each of the plurality of detected objects. The method includes computing an expected shadow of each of the plurality of detected objects based on the extracted geometry. The method includes detecting one or more missing data points in the generated environment map. The one or more missing data points are indicative of a casted shadow of the respective detected object. The method includes computing a geometry of the casted shadow of the respective detected object. The method includes comparing the casted shadow with the expected shadow of the respective detected object. The method includes determining whether the geometry of any of the plurality of detected objects has been misestimated based on the comparison of the casted shadow with the expected shadow of the respective detected object.

In yet another aspect of the present disclosure, a computer program product is provided. The computer program product is embodied in a computer readable medium. The computer program product is useable with a programmable processing device for environment recognition at a worksite. The computer program product is configured to execute a set of instructions comprising receiving a plurality of one or more data points from at least one perception sensor. The computer program product is configured to execute a set of instructions comprising generating an environment map based on the received plurality of one or more data points. The computer program product is configured to execute a set of instructions comprising detecting a plurality of objects within the generated environment map. The computer program product is configured to execute a set of instructions comprising extracting a geometry of each of the plurality of detected objects. The computer program product is configured to execute a set of instructions comprising computing an expected shadow of each of the plurality of detected objects based on the extracted geometry. The computer program product is configured to execute a set of instructions comprising detecting one or more missing data points in the generated environment map. The one or more missing data points are indicative of a casted shadow of the respective detected object. The computer program product is configured to execute a set of instructions comprising computing a geometry of the casted shadow of the respective detected object. The computer program product is configured to execute a set of instructions comprising comparing the casted shadow with the expected shadow of the respective detected object. The computer program product is configured to execute a set of instructions comprising determining whether the geometry of any of the plurality of detected objects has been misestimated based on the comparison of the casted shadow with the expected shadow of the respective detected object.

DETAILED DESCRIPTION

Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or the like parts.FIG. 1illustrates an exemplary worksite100with machines102operating at the worksite100. The worksite100may include, for example, a mine site, a landfill, a quarry, a construction site, a road work site, or any other type of worksite. The machines102may perform any of a plurality of desired operations or tasks at the worksite100, and such operations or tasks may require the machine102to generally traverse the worksite100. Any number of the machines102may simultaneously and cooperatively operate at the worksite100, as desired. As depicted inFIG. 1, two machines102are depicted as rotary drill machines. Alternatively, the machines102may embody any type of machine including dozers, excavators, haul trucks, and any other machine capable of moving about a worksite100.

The machines102may be configured to be operated autonomously, semi-autonomously, or manually. When operating semi-autonomously or manually, the machines102may be operated by remote control and/or by an operator physically located within an operator station202(seeFIG. 2) of the machine102. As used herein, the machine102operating in an autonomous manner operates automatically based upon information received from various sensors without the need for human operator input. The machine102operating semi-autonomously includes an operator, either within the machine102or remotely, who performs some tasks or provides some input and other tasks are performed automatically and may be based upon information received from various sensors. The machine102being operated manually is one in which an operator is controlling all or essentially all of the functions of the machine102. The machine102may be operated remotely by an operator (i.e., remote control) in either a manual or semiautonomous manner.

In addition to the machines102operating at worksite100, various types of obstacles may be located at the worksite100. The obstacles may embody any type of object including those that are fixed or stationary as well as those that are movable or that are moving. Examples of fixed obstacles may include mounds of material104, infrastructure, storage, and processing facilities, buildings such as a command center106, trees, and other structures and fixtures found at the worksite100. Examples of movable obstacles include other machines such as a skid steer loader108, a light duty vehicles110, personnel112, and other objects that may move about the worksite100.

Referring toFIG. 2, an exemplary machine102is illustrated. A rotary drill machine200may include a frame204supported on a ground engaging drive mechanism such as tracks206that are operatively connected to a propulsion system (not shown) associated with the machine102for propelling the machine102about the worksite100. The rotary drill machine200further includes a mast208pivotably mounted on the frame204and movable between a vertical drilling position, as depicted inFIG. 2, and a horizontal transport position (not shown). During a drilling operation, the rotary drill machine200may be raised above the work surface210and supported on jacks212. The rotary drill machine200may include the cab or operator station202that an operator may physically occupy and provide input to operate the machine102.

The machine102may include a control system (not shown). The control system may utilize one or more sensors to provide data and input signals representative of various operating parameters of the machine102and the environment of the worksite100at which the machine102is operating. The control system may include an electronic control module associated with the machine102.

The machine102may be equipped with a plurality of machine sensors that provide data indicative (directly or indirectly) of various operating parameters of the machine and/or the operating environment in which the machine is operating. The term “sensor” is meant to be used in its broadest sense to include one or more sensors and related components that may be associated with the machine102and that may cooperate to sense various functions, operations, and operating characteristics of the machine and/or aspects of the environment in which the machine102is operating.

Referring toFIG. 3, a perception system300may be mounted on or associated with the machine102. The perception system300may include one or more systems such as a Light Detection And Ranging (LADAR) system, a radar system, a Sound Navigation And Ranging (SONAR) system, and/or any other desired system that operate with associated with one or more perception sensors302. The perception sensors302may generate data points that are received by a processing device402(seeFIGS. 4 and 5) and used by the processing device402to determine the position of the work surface101and the presence and position of obstacles within the range of the perception sensors302. The field of view of each of the perception sensors302is depicted schematically inFIG. 3by reference number304.

The perception system300may include a plurality of perception sensors302mounted on the machine102for generating perception data from a plurality of points of view relative to the machine102. Each of the perception sensor302may be mounted on the machine102at a relatively high vantage point. As depicted schematically inFIG. 3, eight perception sensors302are provided that record or sense images in the forward and rearward directions as well as to each side of the machine102. The perception sensor302may be positioned in other locations as desired. The number and location of the perception sensors302described herein is exemplary and does not limit the scope of the present disclosure.

The present disclosure relates to an environment recognition system400(seeFIGS. 4 and 5) associated with the machines102operating at the worksite100. The environment recognition system400includes the processing device402. Referring toFIGS. 4 and 5, the processing device402is communicably coupled to the perception sensors302. The processing device402may receive the data points from the perception sensors402and generate point cloud data associated with the worksite100. In some embodiments, the processing device402may combine the raw data points captured by the perception sensors302into a unified environment map404of at least a portion of the worksite100adjacent and surrounding the machine102. The generated environment map404may represent all the point cloud data available for the environment adjacent machine102.

In one example, the generated environment map404represents a 360-degree view or model of the environment of the machine102, with the machine102at the center of the 360-degree view. According to some embodiments, the generated environment map404may be a non-rectangular shape. For example, the generated environment map404may be hemispherical and the machine102may be conceptually located at the pole, and in the interior, of the hemisphere. The generated environment map404shown in the accompanying figures is exemplary and does not limit the scope of the present disclosure.

The processing device402may generate environment map404by mapping raw data points captured by the perception sensors302to an electronic or data map. The mapping may correlate a two dimensional point from a perception sensor302to a three dimensional point on the generated environment map404. For example, a raw data point of the data point located at (1, 1) may be mapped to location (500, 500, 1) of the generated environment map404. The mapping may be accomplished using a look-up table that may be stored within the processing device402. The look-up table may be configured based on the position and orientation of each of the perception sensors302on the machine102. Alternatively, other methods for transforming the data points from the perception sensors302into the point cloud data may be utilized without any limitation.

The processing device402is configured to detect a plurality of objects (seeFIG. 5) within the generated environment map404. The processing device402may utilize any known object segmentation technique to detect the objects within the generated environment map404. In one embodiment, the environment recognition system400may also include an object identification system (not system). The object identification system may operate to differentiate and store within the generated environment map404categories of objects detected such as machines, light duty vehicles, personnel, or fixed objects.

In some instances, the object identification system may operate to further identify and store the specific object or type of object detected. The object identification system may be any type of system that determines the type of object that is detected. In one embodiment, the object identification system may embody a computer based system that uses edge detection technology to identify the edges of the detected object and then matches the detected edges with known edges contained within a data map or database to identify the object detected. Other types of object identification systems and methods of object identification are contemplated. Further, the processing device402is configured to extract a geometry of the objects based on the detection. This extracted geometry may be indicative of an estimated geometry of the object.

Based on the extracted geometry of the object, the processing device402may compute an expected shadow that the object should cast on the generated environment map404. Further, the processing device402may compute a polyhedron indicative of a region on the generated environment map404that the respective object may occlude. In one embodiment, the polyhedron may be computed using ray tracing technique. In order to compute the polyhedron, the processing device402may consider the position and orientation of the perception sensor302and the geometry of the rays coming from it. The rays that fall on the boundary of the object form points on the polyhedron at their intersection point on the object, and form edges of the polyhedron as they continue beyond the object. In one embodiment, the expected shadow of the object may be computed by projecting a front face of the object onto a detected ground surface following these rays.

In addition, the generated environment map404corresponding to the environment around the machine102may include missing data points. The processing device402is configured to detect the one or more missing data points within the generated environment map404. These missing data points are indicative of holes in the point cloud data. Further, the missing data points represent a casted shadow of the respective object within the generated environment map404.

A person of ordinary skill in the art will appreciate that the missing data points are blind zones or areas of limited information or visibility. For example, in some instances, the fields of some or all of the perception sensors302may be limited so as not to cover or extend fully about the machine102or only extend a limited distance in one or more directions. This may be due to limitations in the range or capabilities of the perception sensors302, the software associated with the perception sensors302, and/or the positioning of the perception sensors302. Such limitations of the perception sensors302are typically known to the processing device402. Further, the processing device402is able to determine an amount of the missing data points caused by such limitations of the perception sensors302and an amount of the missing data points caused by object occlusion. The processing device402computes a geometry of the casted shadow of the respective objects.

The processing device402is further configured to compare the expected shadow with the casted shadow of the respective object. If the expected shadow matches the casted shadow of the respective object, the processing device402determines that a true geometry of the respective object is same as that of the extracted geometry of the object obtained from the environment map404. However, if the expected shadow does not match the casted shadow of the respective object, the processing device402determines that the true geometry of the respective object is different from the extracted or estimated geometry. Moreover, if there is a mismatch between the expected shadow and the casted shadow, the processing device402determines that the geometry of the object has been misestimated. Misestimating the geometry of the object may be indicative of incorrect estimation of at least one dimension of the object. In one embodiment, the mismatch between the expected shadow and the casted shadow of the object may be indicative that the processing device402underestimated the geometry of the object.

FIG. 6is an exemplary low-level implementation of the environment recognition system400ofFIGS. 4 and 5. The present disclosure has been described herein in terms of functional block components, modules, and various processing steps. It should be appreciated that such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, a computer based system, hereinafter referred as system600may employ various integrated circuit components, e.g., memory elements, processing elements, logic elements, look-up tables, and/or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the system600may be implemented with any programming or scripting language such as, but not limited to, C, C++, Java, COBOL, assembler, PERL, Visual Basic, SQL Stored Procedures, extensible markup language (XML), with the various algorithms being implemented with any combination of data structures, objects, processes, routines or other programming elements.

Further, it should be noted that the system600may employ any number of conventional techniques for data transmission, signaling, data processing, network control, and/or the like. Still further, the system600could be configured to detect or prevent security issues with a user-side scripting language, such as JavaScript, VBScript or the like. In an embodiment of the present disclosure, the networking architecture between components of the system600may be implemented by way of a client-server architecture. In an additional embodiment of this disclosure, the client-server architecture may be built on a customizable.Net (dot-Net) platform. However, it may be apparent to a person ordinarily skilled in the art that various other software frameworks may be utilized to build the client-server architecture between components of the system600without departing from the spirit and scope of the disclosure.

The present disclosure (i.e., system400, system600, method700, any part(s) or function(s) thereof) may be implemented using hardware, software or a combination thereof, and may be implemented in one or more computer systems or other processing systems. However, the manipulations performed by the present disclosure were often referred to in terms such as detecting, determining, and the like, which are commonly associated with mental operations performed by a human operator. No such capability of a human operator is necessary, or desirable in most cases, in any of the operations described herein, which form a part of the present disclosure. Rather, the operations are machine operations. Useful machines for performing the operations in the present disclosure may include general-purpose digital computers or similar devices. In accordance with an embodiment of the present disclosure, the present disclosure is directed towards one or more computer systems capable of carrying out the functionality described herein. An example of the computer based system includes the system600, which is shown by way of a block diagram inFIG. 6.

The system600includes at least one processor, such as a processor602. The processor602may be connected to a communication infrastructure604, for example, a communications bus, a cross-over bar, a network, and the like. Various software embodiments are described in terms of this exemplary system600. Upon perusal of the present description, it will become apparent to a person skilled in the relevant art(s) how to implement the present disclosure using other computer systems and/or architectures. The system600includes a display interface606that forwards graphics, text, and other data from the communication infrastructure604for display on a display unit608.

The system600further includes a main memory610, such as random access memory (RAM), and may also include a secondary memory612. The secondary memory612may further include, for example, a hard disk drive614and/or a removable storage drive616, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. Removable storage drive616reads from and/or writes to a removable storage unit618in a well-known manner. The removable storage unit618may represent a floppy disk, magnetic tape or an optical disk, and may be read by and written to by the removable storage drive616. As will be appreciated, the removable storage unit618includes a computer usable storage medium having stored therein, computer software and/or data.

In accordance with various embodiments of the present disclosure, the secondary memory612may include other similar devices for allowing computer programs or other instructions to be loaded into the system600. Such devices may include, for example, a removable storage unit620, and an interface622. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units and interfaces, which allow software and data to be transferred from the removable storage unit620to system600.

The system600may further include a communication interface624. The communication interface624allows software and data to be transferred between the system600and external devices630. Examples of the communication interface624include, but may not be limited to a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, and the like. Software and data transferred via the communication interface624may be in the form of a plurality of signals, hereinafter referred to as signals626, which may be electronic, electromagnetic, optical or other signals capable of being received by the communication interface624. The signals626may be provided to the communication interface624via a communication path (e.g., channel)628. The communication path628carries the signals626and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link and other communication channels.

In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such as the removable storage drive616, a hard disk installed in the hard disk drive614, the signals626, and the like. These computer program products provide software to the system600. The present disclosure is also directed to such computer program products.

The computer programs (also referred to as computer control logic) may be stored in the main memory610and/or the secondary memory612. The computer programs may also be received via the communication interface604. Such computer programs, when executed, enable the system600to perform the functions consistent with the present disclosure, as discussed herein. In particular, the computer programs, when executed, enable the processor602to perform the features of the present disclosure. Accordingly, such computer programs represent controllers of the system600.

In accordance with an embodiment of the present disclosure, where the disclosure is implemented using a software, the software may be stored in a computer program product and loaded into the system600using the removable storage drive616, the hard disk drive614or the communication interface624. The control logic (software), when executed by the processor602, causes the processor602to perform the functions of the present disclosure as described herein.

In another embodiment, the present disclosure is implemented primarily in hardware using, for example, hardware components such as application specific integrated circuits (ASIC). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s). In yet another embodiment, the present disclosure is implemented using a combination of both the hardware and the software.

Various embodiments disclosed herein are to be taken in the illustrative and explanatory sense, and should in no way be construed as limiting of the present disclosure. All numerical terms, such as, but not limited to, “first”, “second”, “third”, or any other ordinary and/or numerical terms, should also be taken only as identifiers, to assist the reader's understanding of the various embodiments, variations, components, and/or modifications of the present disclosure, and may not create any limitations, particularly as to the order, or preference, of any embodiment, variation, component and/or modification relative to, or over, another embodiment, variation, component and/or modification.

It is to be understood that individual features shown or described for one embodiment may be combined with individual features shown or described for another embodiment. The above described implementation does not in any way limit the scope of the present disclosure. Therefore, it is to be understood although some features are shown or described to illustrate the use of the present disclosure in the context of functional segments, such features may be omitted from the scope of the present disclosure without departing from the spirit of the present disclosure as defined in the appended claims.

INDUSTRIAL APPLICABILITY

The present disclosure relates to the system and method for environment recognition associated with the worksite100.FIG. 7is a flowchart of the method700of operation of the environment recognition system400. At step702, the processing device402of the environment recognition system400receives the one or more data points from the perception sensors302. At step704, the processing device402generates the environment map404based on the received data points. At step706, the processing device402detects the objects within the generated environment map404. At step708, the processing device402extracts the geometry of the detected objects. At step710, the processing device402computes the expected shadow of the detected objects based on the extracted geometry. At step712, the processing device402detects one or more missing data points in the generated environment map404. The one or more missing data points are indicative of the casted shadow of the respective detected object. At step713, the processing device402computes a geometry of the casted shadow of the respective detected object. At step714, the processing device402compares the casted shadow with the expected shadow of the respective detected object. At step716, the processing device402determines whether the geometry of any of the plurality of detected objects has been misestimated based on the comparison of the casted shadow with the expected shadow of the respective detected object.

The environment recognition system400is capable of determining if the missing data in the environment map404are due to object occlusion, obscurant (dust/rain) occlusions or sensor blockage. The processing device402extracts the shape of the object causing the occlusion based on the casted shadow within the generated map404. Further, the processing device402may judge whether the dimensions of the object have been misestimated. Further, by using already available information about sensor limitations, the processing device402may determine what percent of the mismatch is due to occlusion rather than sensor limitations. This may help to minimize the impacts of sensor resolution, viewpoint limitations, and environmental constraints.

While aspects of the present disclosure have been particularly shown and described with reference to the embodiments above, it will be understood by those skilled in the art that various additional embodiments may be contemplated by the modification of the disclosed machines, systems and methods without departing from the spirit and scope of the disclosure. Such embodiments should be understood to fall within the scope of the present disclosure as determined based upon the claims and any equivalents thereof.