Patent Description:
It is the object of the present invention to provide an improved method and system for collision avoidance.

The subject disclosure is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. It may be evident, however, that the subject disclosure can be practiced without these specific details.

As used in this application, the terms "component," "system," "platform," "layer," "controller," "terminal," "station," "node," "interface" are intended to refer to a computer-related entity or an entity related to, or that is part of, an operational apparatus with one or more specific functionalities, wherein such entities can be either hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical or magnetic storage medium) including affixed (e.g., screwed or bolted) or removable affixed solid-state storage drives; an object; an executable; a thread of execution; a computer-executable program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. Also, components as described herein can execute from various computer readable storage media having various data structures stored thereon. The components may communicate via local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network such as the Internet with other systems via the signal). As another example, a component can be an apparatus with specific functionality provided by mechanical parts operated by electric or electronic circuitry which is operated by a software or a firmware application executed by a processor, wherein the processor can be internal or external to the apparatus and executes at least a part of the software or firmware application. As yet another example, a component can be an apparatus that provides specific functionality through electronic components without mechanical parts, the electronic components can include a processor therein to execute software or firmware that provides at least in part the functionality of the electronic components. As further yet another example, interface(s) can include input/output (I/O) components as well as associated processor, application, or Application Programming Interface (API) components. While the foregoing examples are directed to aspects of a component, the exemplified aspects or features also apply to a system, platform, interface, layer, controller, terminal, and the like.

As used herein, the terms "to infer" and "inference" refer generally to the process of reasoning about or inferring states of the system, environment, and/or user from a set of observations as captured via events and/or data. Inference can be employed to identify a specific context or action, or can generate a probability distribution over states, for example. The inference can be probabilistic-that is, the computation of a probability distribution over states of interest based on a consideration of data and events. Inference can also refer to techniques employed for composing higher-level events from a set of events and/or data. Such inference results in the construction of new events or actions from a set of observed events and/or stored event data, whether or not the events are correlated in close temporal proximity, and whether the events and data come from one or several event and data sources.

Furthermore, the term "set" as employed herein excludes the empty set; e.g., the set with no elements therein. Thus, a "set" in the subject disclosure includes one or more elements or entities. As an illustration, a set of controllers includes one or more controllers; a set of data resources includes one or more data resources; etc. Likewise, the term "group" as utilized herein refers to a collection of one or more entities; e.g., a group of nodes refers to one or more nodes.

Various aspects or features will be presented in terms of systems that may include a number of devices, components, modules, and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, etc. and/or may not include all of the devices, components, modules etc. discussed in connection with the figures. A combination of these approaches also can be used.

Many technologies exist for detecting distances of objects or surfaces within a monitored space. These include, but are not limited to, time of flight (TOF) optical sensors or other types of three-dimensional sensors - such as photo detectors or multi-pixel image sensors - which are used to detect distances of objects or surfaces within a viewing range of the sensor. These optical sensors can include, for example, photo detectors that measure and generate a single distance data point for an object within range of the detector, as well as multi-pixel image sensors comprising an array of photo-detectors that are each capable of generating a distance data point for a corresponding image pixel. Some three-dimensional optical sensors, such as stereo vision technology (for a passive sensor) or structured light technology (for active sensor) measure distances using triangulation.

Some types of TOF sensors that employ pulsed light illumination measure the elapsed time between emission of a light pulse to the viewing field (or viewing space) and receipt of a reflected light pulse at the sensor's photo-receiver. Since this time-of-flight information is a function of the distance of the object or surface from the sensor, the sensor is able to leverage the TOF information to determine the distance of the object or surface point from the sensor.

<FIG> is a generalized block diagram of a TOF sensor <NUM> illustrating pulsed light time of flight principles. The TOF sensor <NUM> includes an illumination light source - represented by emitter <NUM> and associated illumination optics <NUM> in <FIG> - that projects light <NUM> modulated with pulses <NUM>. The space encompassed by the projected modulated light <NUM> represents the field of illumination (FOI). Objects and surfaces within the viewing field, such as object <NUM>, reflect part of the pulses' radiation back to the TOF sensor's imaging lens <NUM>, which directs the received pulses <NUM> to pixels of receiver <NUM> (e.g., a photo-detector or a photo-sensor such as a photo-diode). A subset of the modulated light <NUM> - including reflected pulses <NUM> - is scattered by objects and surfaces in the scene, and a portion of this scattered pulsed illumination is received at the TOF sensor's imaging lens <NUM> and associated receiver <NUM> (e.g., light photo-detectors). Receiver <NUM> can include, for example, a dedicated multi-pixel CMOS (complementary metal oxide semiconductor) application-specific integrated circuit (ASIC) imager that integrates specialized means for measuring the position in time of received pulses <NUM>. In general, the sensing technology used by some TOF sensors measures the time taken by a light pulse to travel from the sensor's illumination light source to an object <NUM> or surface within the viewing field and back to the sensor's light photo-detectors. Distance measurement components can measure the distance d to the object <NUM> as <MAT> where c is the speed of light, and t is the measured time of the round trip for the pulse from the emitter <NUM> to the object <NUM> and back to the receiver <NUM>.

Since the speed of light c is a known constant and the time t elapsed between emission and reception of the pulse <NUM> can be measured, the TOF sensor's distance measuring components can determine, for each pixel of the receiver <NUM>, the distance between the object <NUM> and the sensor by calculating half of the round-trip distance, as given by equation (<NUM>) above. Collectively, the distance information obtained for all pixels of the viewing space yields depth map or point cloud data for the viewing space. In some implementations, the TOF sensor's distance measurement components can include a timer that measures the arrival time of a received pulse relative to the time at which emitter <NUM> emitted the pulse, or may include demodulation circuitry that determines the measured time of the pulse's round trip by comparing the phase of the received pulse with its corresponding emitted pulse. In general, the TOF sensor <NUM> generates information that is representative of the position in time of the received pulse.

Each pixel of the captured image has an associated photo-receiver or photo-detector. When radiation of a reflected pulse <NUM> is incident on the photo-receivers or photo-detectors that make up sensor <NUM>, the incident light is converted into an electrical output proportional to the intensity of the incident light. The distance measurement components can then recover and analyze the electrical output in order to identify the pulse, thereby determining that the reflected pulse has been received at the receiver <NUM>.

In some implementations, the sensor's emitter <NUM> may emit a burst of pulses into the scene for a given measuring sequence and perform the distance measurement based on an accumulation of multiple received pulses reflected back to the sensor. The photo-detectors of receiver <NUM> can accumulate electrical charges based on the exposure duration of the receiver <NUM> to the received light pulse radiation relative to a time reference. The accumulated charges on a given photo-detector translate into a voltage value that, evaluated over time, defines a demodulated pulse waveform that is recognizable by the distance measurement components. Once the pulse is identified in this manner, the TOF sensor <NUM> can estimate the time that the reflected pulse was received at the photo-detector relative to the time that the pulse was emitted (e.g., based on phase shift analysis or other types of analysis performed on the pulse waveform), and the distance associated with the corresponding pixel can be estimated based on this time using equation (<NUM>) (or another distance determination equation or algorithm that defined distance as a function of light pulse propagation time).

If a TOF sensor's distance calculation capabilities are sufficiently reliable, the sensor can serve as an industrial safety device for an industrial safety system. In an example implementation illustrated in <FIG>, TOF-based detection systems can be installed on automated guided vehicles (AGVs) <NUM> or other types of automated or autonomous mobile equipment to facilitate detection of people or obstacles <NUM> in the path of the vehicle, which can cause the AGV <NUM> to initiate a collision avoidance response (e.g., by deviating its planned path to avoid the obstacle <NUM>, by slowing its travel, or by stopping). Such collision avoidance systems can be implemented in diverse types of AGVs <NUM>, autonomous mobile robots (AMRs), or other types of autonomous or guided mobile assets or platforms, including but not limited to automated warehouse vehicles used to transport products or materials, automated cars, autonomous amusement park rides, or other such applications.

In some cases, TOF-based collision detection systems may use two-dimensional (2D) laser scanner devices or 2D light detecting and ranging (Lidar) systems to detect obstacles <NUM>. <FIG> is a diagram illustrating an AGV <NUM> equipped with a 2D laser scanner <NUM> that projects a wide, flat light beam <NUM> into the space in front of the AGV <NUM> (the wide flat beam <NUM> is depicted from the side in <FIG>, and so is seen as a line). Some 2D laser scanners <NUM> may create the wide flat beam <NUM> by sweeping a narrow beam across the area in front of the AGV <NUM> in an oscillatory manner to collect line-wise distance data. Alternatively, the 2D laser scanner may project a stationary, substantially planar beam <NUM> across the area in front of the AGV <NUM> and collect distance data for objects that pass through the beam <NUM>.

<FIG> is a diagram illustrating detection of an obstacle <NUM> in front of the AGV <NUM> using scanner <NUM>. When the projected beam is incident on an obstacle <NUM> in the path of AGV <NUM> the reflected modulated light returned to the scanner's sensing circuitry is analyzed to determine a distance of the obstacle <NUM> from the AGV <NUM>. If the safety system determines that this distance is found to be less than a defined safe distance, the system may initiate a collision avoidance action, such as causing the AGV <NUM> to stop or to divert its planned path of travel to avoid the obstacle. In the example depicted in <FIG>, the obstacle <NUM> is located in front of the scanner <NUM> with an approach (represented by the arrow) parallel to the scanning plane of beam <NUM>. When the measured distance between the obstacle <NUM> and the AGV <NUM> is smaller than a defined threshold, a signal is triggered indicating that the obstacle <NUM> is intruding into the protective field of the scanner <NUM>. In the case of 2D scanners, the protective field is defined as a plane parallel to the floor <NUM> and therefore parallel to the direction of approach between the AGV <NUM> and the obstacle <NUM>.

TOF-based collision detection systems that use 2D laser scanners <NUM> have a number of drawbacks that limit their utility in crucial safety applications. For example, the field of view - and thus the visibility - of a 2D scanner <NUM> is limited due to the flat planar shape of the beam <NUM>. Moreover, the safety integrity level (SIL) rating for detection systems using 2D laser scanners <NUM> typically cannot exceed SIL <NUM> with a performance level (PL) of PLd. This level of safety reliability is insufficient for many safety applications, which require a safety integrity of at least SIL <NUM> with a performance level of PLe.

As a related consideration, safety systems and their associated detection devices must remain in reliable working order and meet requirements that guarantee the safe operation of their associated protected systems (e.g., AGVs). In general, the system is considered unsafe if system failures result in hazards going undetected (i.e., failing to danger), or if the system is not capable of performing to requirements without going into a safe state. Safety system failures or measurement inaccuracies should be detected automatically, and corrective measures carried out, to ensure that such failures do not result in injury or damage.

To address these and other issues, one or more embodiments described herein provide a safety system and method capable of achieving safety-rated collision avoidance functionality for mobile equipment (e.g., AGVs or other types of mobile machines, including autonomous vehicles or driving assistance systems) by detecting objects located in the field of view of a three-dimensional (3D) TOF vision system or camera. Relative to a 2D scanner, incorporating a 3D TOF camera into a collision avoidance system allows a larger volume to be monitored for object intrusion, improving reliability of object detection. To ensure reliability of the safety system's obstacle detection capabilities, the collision avoidance system also includes self-diagnostic capabilities that verify the accuracy of the TOF camera's distance measurements even in the absence of a test object within the camera's field of view. These self-diagnostic capabilities can further improve the safety rating of the collision avoidance system.

<FIG> is a block diagram of an example TOF collision avoidance system <NUM> according to one or more embodiments of this disclosure. Although <FIG> depicts certain functional components as being integrated on a common platform with shared processor and memory, it is to be appreciated that one or more of the functional components illustrated in <FIG> may reside on a separate platform relative to system <NUM> in some embodiments. Also, although the subject collision detection techniques are described herein in connection with TOF sensor devices, it is to be appreciated that the collision detection techniques described herein can be implemented in other types of 3D sensors that are capable of generating a distance map or point cloud data. Aspects of the systems, apparatuses, or processes explained in this disclosure can constitute machine-executable components embodied within machine(s), e.g., embodied in one or more computer-readable mediums (or media) associated with one or more machines. Such components, when executed by one or more machines, e.g., computer(s), computing device(s), automation device(s), virtual machine(s), etc., can cause the machine(s) to perform the operations described.

TOF collision avoidance system <NUM> can include an emitter component <NUM>, a photo-sensor component <NUM>, a distance determination component <NUM>, a diagnostic component <NUM>, a level component <NUM>, a control output component <NUM>, a user interface component <NUM>, one or more processors <NUM>, and memory <NUM>. In various embodiments, one or more of the emitter component <NUM>, photo-sensor component <NUM>, distance determination component <NUM>, diagnostic component <NUM>, level component <NUM>, control output component <NUM>, user interface component <NUM>, the one or more processors <NUM>, and memory <NUM> can be electrically and/or communicatively coupled to one another to perform one or more of the functions of the TOF collision avoidance system <NUM>. In some embodiments, one or more of components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> can comprise software instructions stored on memory <NUM> and executed by processor(s) <NUM>. TOF collision avoidance <NUM> may also interact with other hardware and/or software components not depicted in <FIG>. For example, processor(s) <NUM> may interact with one or more external user interface devices, such as a keyboard, a mouse, a display monitor, a touchscreen, or other such interface devices. TOF collision avoidance system <NUM> may also include network communication components and associated networking ports for sending data generated by any of components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> over a network (either or both of a standard data network or a safety network), or over a backplane.

In some embodiments, collision avoidance system <NUM> may comprise a 3D TOF camera <NUM> comprising the emitter component <NUM>, photo-sensor component <NUM>, and distance determination component <NUM>. Emitter component <NUM> can be configured to control emission of light by the 3D TOF camera <NUM>. In this regard, TOF camera <NUM> may include a laser or light emitting diode (LED) light source under the control of emitter component <NUM>. Emitter component <NUM> can generate pulsed light emissions directed to the monitored space so that time-of-flight information for the reflected light pulses can be generated by the TOF camera <NUM> (e.g., by the distance determination component <NUM>).

Photo-sensor component <NUM> can be configured to convert light energy incident on a photo-receiver or photo-detector array to electrical energy for respective pixels of a monitored space, and measure this electrical energy for the purposes of pulse identification and distance analysis. In some embodiments, photo-sensor component <NUM> can selectively control the storage of the converted electrical energy in various electrical storage components (e.g., measuring capacitors) for subsequent pulse waveform identification and distance analysis. Distance determination component <NUM> can be configured to determine a propagation time (time of flight) of emitted light pulses received at respective pixels based on the stored electrical energy generated by the photo-sensor component <NUM>, and to further determine a distance value of an object or surface corresponding to a pixel within the viewing space based on the determined propagation time.

Diagnostic component <NUM> can be configured to execute a diagnostic sequence that verifies the accuracy of distance values measured and reported by the 3D TOF camera <NUM>. This diagnostic sequence can involve measuring test distances to points on the ground or floor within the camera's field of view and comparing these measured values with expected distances corresponding to these points.

Level component <NUM> is configured to measure a direction and degree of inclination of an AGV or other mobile equipment on which system <NUM> is mounted, and adjust the protective field of the TOF camera <NUM> to compensate for the measured inclination, ensuring that measurements of distances to obstacles in front of the AGV remain accurate even if the AGV is traversing uneven terrain.

Control output component <NUM> can be configured to control one or more sensor outputs based on results generated by the distance determination component <NUM> and diagnostic component <NUM>. This can include, for example, sending an analog or digital control signal to a control or supervisory device (e.g., an on-board computer mounted in the AGV or other mobile machinery, etc.) to perform a control action, initiating a safety action (e.g., stopping an AGV, altering a path of the AGV, removing power from the AGV, etc.), initiating a notification (e.g., an audible or visual signal), or other such actions.

User interface component <NUM> can be configured to receive user input and to render output to the user in any suitable format (e.g., visual, audio, tactile, etc.). In some embodiments, user interface component <NUM> can be configured to communicate with a graphical user interface (e.g., a programming or development platform) that executes on a separate hardware device (e.g., a laptop computer, tablet computer, smart phone, etc.) communicatively connected to system <NUM>. In such configurations, user interface component <NUM> can receive input parameter data entered by the user via the graphical user interface, and deliver output data to the interface. Input parameter data can include, for example, protective field definition data, normalized pulse shape data that can be used as reference data for identification of irregularly shaped pulses, light intensity settings, minimum safe distances or other distance threshold values to be compared with the measured distance values for the purposes of determining when to initiate a collision avoidance action, or other such parameters. Output data can comprise, for example, status information for the collision avoidance system <NUM> in general or the TOF camera <NUM> in particular, alarm or fault information, parameter settings, or other such information.

The one or more processors <NUM> can perform one or more of the functions described herein with reference to the systems and/or methods disclosed. Memory <NUM> can be a computer-readable storage medium storing computer-executable instructions and/or information for performing the functions described herein with reference to the systems and/or methods disclosed.

<FIG> is a diagram illustrating an AGV <NUM> equipped with a 3D TOF camera <NUM> for object detection purposes. <FIG> is a diagram depicting detection of an obstacle <NUM> by the TOF camera <NUM>. As an alternative to the 2D scanner <NUM> discussed above in connection with <FIG>, a 3D collision avoidance system <NUM> implemented by a multi-pixel TOF camera <NUM> can detect the presence of an obstacle <NUM> relative to a protective field <NUM> defined as a volume (as opposed to the plane of a 2D scanner <NUM>). The TOF camera's emitter component <NUM> emits pulsed light into the field of view of the camera <NUM>, and the obstacle <NUM> is based on measurement of light back-refracted from the obstacle <NUM> and received on multiple pixels of the camera's photo-sensor component <NUM>. In the example depicted in <FIG>, the protective field <NUM> to be monitored for intrusions is set to be narrower than the camera's total field of view encompassed by emitted beam <NUM>. In some embodiments, the dimensions or boundaries of the protective field <NUM> to be monitored for intrusions can be configured by a user (e.g., via user interface component <NUM>).

<FIG> is a block diagram illustrating components of TOF collision avoidance system <NUM> according to one or more embodiments. Although examples described herein assume that collision avoidance system <NUM> is mounted on an AGV, it is to be appreciated that the object detection and diagnostic features described herein are applicable for use with other types of automated or autonomous moving equipment, including but not limited to AMRs, automated warehouse vehicles, automated cars, autonomous amusement park rides, or other such mobile equipment. Embodiments of collision avoidance system <NUM> can also be implemented in non-autonomous vehicles as part of a driving assistance system. In such embodiments, the collision avoidance system <NUM> can be configured to generate alerts in response to determining that a driver is at risk of colliding with an obstacle, or to assume a degree of automated control from the driver in order to mitigate a risk of collision with an obstacle.

In the example implementation depicted in <FIG>, emitter component <NUM> emits a pulsed light beam <NUM> to a scene in front of the AGV <NUM> (e.g., via emitting lens element <NUM>). Light pulses are reflected from the objects and surfaces within the scene, and the reflected light is received at a receiving lens element <NUM> and directed to a photo-receiver array <NUM> associated with photo-sensor component <NUM>, which generates pixel data <NUM> for the scene. In some implementations, pixel data <NUM> can comprise measured voltage values - or sets of voltage values - measured for respective pixels of the camera's pixel array, where the voltage values are indicative of the presence and phase of reflected pulses received by the pixels.

The pixel data <NUM> is provided to distance determination component <NUM>, which computes distance values for each pixel <NUM> of the resulting image based on the pixel data <NUM> (e.g., based on equation (<NUM>) or a variation thereof), resulting in distance data <NUM>. This results in a point cloud <NUM> for the image, which represents the array of measured distance values corresponding to the array of pixels <NUM> that make up the image. The distance value of a given pixel <NUM> represents a distance of a point on an object or surface corresponding to that pixel from the TOF camera <NUM>. The distance values for all pixels <NUM> of the image yields the point cloud <NUM> for the image. The point cloud distances are provided to control output component <NUM> as distance data <NUM>. If the distance data <NUM> indicates that an obstacle <NUM> (e.g., a person in the example depicted in <FIG>) is located within a protective distance from the camera <NUM> (that is, within the protective field defined for the camera <NUM>), the control output component <NUM> generates a control output <NUM> directed to the control system of the AGV <NUM> that initiates a safety action (e.g. rerouting of the AGV's path, slowing or stopping of the AGV, etc.).

As noted above, in some embodiments the protective field <NUM> to be monitored for intrusions by camera <NUM> can be configured by a user via user interface component <NUM>. In one or more example embodiments, the user interface component <NUM> can generate and render configuration displays on a client device that allow a user to define the protective field <NUM> in terms of a selected section <NUM> of the total point cloud <NUM> or image whose distances are to be analyzed for possible obstacles, as well as minimum safe distances (i.e., the protective distance) for the pixels within this section <NUM>. In this regard, the selected section <NUM> of the point cloud <NUM> represents the x-y boundaries of the protective field <NUM>, while the minimum safe distances represent the z-axis limits of the protective field. Based on this configuration, control output component <NUM> will generate control output <NUM> based on a determination of whether a sufficient number of the subset of pixels <NUM> that make up the selected section <NUM> have distance values that are less than the defined minimum safe distance. As will be described in more detail below, some embodiments of system <NUM> can automatically adjust the location or direction of the protective field <NUM> in response to detected conditions.

<FIG> is a diagram illustrating detection of an example cubical obstacle <NUM> by embodiments of TOF camera <NUM>. As discussed above, the emitter component <NUM> of TOF camera <NUM> emits a series of light pulses into the field of view as beam <NUM> and the photo-sensor component <NUM> receives returned light pulses reflected from the field of view, including light pulses reflected from obstacle <NUM>. Distance determination component <NUM> analyzes the voltage values captured by the photo-sensor component <NUM> in response to the reflected light pulses incident on the subset of pixels corresponding to obstacle <NUM> to estimate total propagation times for of the pulses received at the pixels, and translates these propagation times to estimated distances of obstacle <NUM> from the respective pixels (e.g., based on equation (<NUM>) above or a variation thereof).

One pixel of the TOF camera's photo-detector observes a portion of obstacle <NUM> located in that pixel's field of view and captures the optical energy from a light pulse reflected from the obstacle <NUM> necessary to calculate the distance of the obstacle <NUM> from the camera <NUM> (and thus from the AGV <NUM>). The obstacle detection and self-diagnostic techniques described herein are not dependent upon any particular technique used by the TOF camera <NUM> to translate incident optical energy into pixel-level distance information for an obstacle in the field of view.

Control output component <NUM> performs intrusion detection by comparing a measured distance d of obstacle <NUM> with a defined threshold distance DP representing the limit of the protective field (the protected area) being monitored by the collision avoidance system <NUM>. Threshold distance DP - representing a minimum safe distance - can be configured by a user (e.g., via user interface component <NUM>) based on the needs of the safety application within which system <NUM> is being used (e.g., the relative level of hazard posed by the AGV <NUM>, the speed of the AGV <NUM> during normal operation, etc.). For example, threshold distance DP can be set to correspond to a determined minimum safe distance from the AGV <NUM>, taking into account the AGV's speed or stopping time. When control output component <NUM> determines that the distance d of obstacle <NUM> is less than the threshold distance DP (that is, when d < DP), the obstacle <NUM> is determined to be within the protective field. In some applications, in response to detection of an obstacle <NUM> within the protective field (that is, when the measured distance d is determined to be less than the distance DP of the protective field boundary), the control output component <NUM> can initiate a defined safety action to mitigate a potential collision between the AGV <NUM> and the obstacle <NUM> (e.g., by re-routing, slowing, or stopping the AGV <NUM>).

In <FIG>, variable d represents the measured distance of obstacle <NUM> determined by distance determination component <NUM>, while dT represents the true distance of obstacle <NUM> from the AGV <NUM>. The limit or boundary of the protective field <NUM> is defined as a plane <NUM> perpendicular to the central sensing axis z of the TOF camera <NUM>, and the distance of that boundary plane <NUM> to the sensor is DP. An obstacle <NUM> is inside the protective field <NUM> (see <FIG>) if its true distance dT is smaller than the threshold distance DP. In the example scenario depicted in <FIG>, the distance of a point to the TOF camera <NUM> is defined as a point coordinate on the z-axis, which is different than point-to-point distance, such as radial distance r.

The distance DP of the protective field <NUM> can be viewed as the sum of a detection zone distance DD and a tolerance zone distance DZ. With the intrusion detection criterion being defined as d < DP, the probability of detection must be guaranteed if the obstacle <NUM> is at the boundary of the detection zone - located at a distance DD from the TOF camera <NUM> - or intruding somewhat into the detection zone.

Also included in <FIG> is an example probability distribution curve <NUM> of the measured distance d of the obstacle <NUM>. The tolerance zone, represented by the distance Dz between the detection zone boundary <NUM> and the protective field boundary plane <NUM>, is set to guarantee high probability of detection, such that that the measured distance d is unlikely to be greater than DP = (Do + Dz) for an obstacle <NUM> located in the field (that is, when the true object distance dT is less than or equal to (DD - DI), the measured distance d should not indicate that the obstacle <NUM> is outside the protective field). As an example of a normal distribution of the measured distance, a value of the tolerance zone distance DZ greater than 5σ guarantees, for dT =< DD, a measured distance d smaller than DP with a probability above a required level of <NUM>-<NUM>. <NUM>-<NUM>.

A 3D TOF camera <NUM> can offer greater reliability of obstacle detection relative to a 2D scanner since the 3D TOF camera's total available field of view is often larger than the protective field <NUM> that must be monitored for a given collision avoidance system. In contrast to monitoring for presence of obstacles <NUM> within the flat plane of a 2D scanner, a 3D TOF camera <NUM> can be configured to monitor a volume of space in front of the AGV <NUM> capable of detecting obstacles <NUM> across a greater range of approach angles.

As noted above, the safety rating for an industrial safety system (including the collision avoidance systems described herein) can be improved if the system is implemented with diagnostics that verify faultless operation of crucial components of the safety system, and is capable of performing a safety action if a system failure or measurement inaccuracy is detected. Ideally, such diagnostics should be performed automatically at regular intervals, and should verify the integrity of both the illumination path and the imaging path. This includes verifying that the distance measurement components - e.g., photo-sensor component <NUM>, distance determination component <NUM>, photo-receiver array <NUM>, etc. - are producing accurate distance values for each pixel. According to one diagnostic approach for verifying distance measurement accuracy, a test measurement sequence can be performed using an object at a known distance from the 3D TOF camera <NUM>. The camera can measure distance values for the object, and these measured distance values can then be compared with expected distance value to determine whether the actual measurements align with the expected distances. This approach can confirm correct operation of detection and measurement components along the return path of reflected light received at the camera <NUM>, including optical elements, sensing elements, and computational components that make up the system <NUM>.

However, if the 3D TOF camera <NUM> is directed such that beam <NUM> is projected forward as shown in <FIG>, there is no fixed object within the camera's protective field <NUM> during default scenarios (e.g., the scenario depicted in <FIG>) that can be used as a test object for verifying correct operation of the collision avoidance system's distance measurement components. In the absence of such a test object, no light from the field in front of the AGV <NUM> is returned to the camera <NUM>, no diagnostic distance value can be generated, and accuracy of the system's distance measurement capabilities cannot be verified.

To address this issue, the 3D TOF camera <NUM> can be inclined toward the floor <NUM> so that a portion of the floor <NUM> is within the camera's protective field <NUM>, as illustrated in <FIG>. By directing the camera <NUM> in this manner, beam <NUM> is incident on the ground or floor <NUM>, which reflects a portion of the light back to the camera's sensing components. Thus, the ground or floor <NUM> provides a return light path that can be used during diagnostic sequences for verifying correct operation of the system's imaging components.

In the case of the multi-pixel 3D TOF camera <NUM> used in the present collision avoidance system <NUM>, each pixel having the ground or floor <NUM> in its field of view will generate a distance value corresponding to the point on the floor <NUM> seen by that pixel. This distance value represents the distance <NUM> from the TOF camera <NUM> to the floor <NUM> along the optical axis of the pixel.

<FIG> is a diagram illustrating detection of an obstacle <NUM> in the path of the AGV <NUM> when the TOF camera <NUM> is inclined downward as shown in <FIG>. If an obstacle <NUM> enters the protective field <NUM> of the 3D TOF camera <NUM>, distance determination component <NUM> can distinguish between the distance <NUM> from the camera <NUM> to the obstacle <NUM> (that is, the distance from a given camera pixel to the obstacle <NUM> along the pixel's optical axis) and the horizontal distance <NUM> from the AGV <NUM> to the obstacle <NUM>. Specifically, distance determination component <NUM> can derive the horizontal distance <NUM> between the AGV <NUM> and the obstacle <NUM> based on the measured distance <NUM> between the camera <NUM> and the obstacle <NUM> along the camera's optical path, and the control output component <NUM> can use this calculated distance <NUM> to determine whether a control output <NUM> should be generated.

<FIG> is a diagram illustrating an approach carried out by the distance determination component <NUM> for calculating the distance <NUM> when camera <NUM> is inclined as illustrated in <FIG>. In this example, the minimum safe distance (threshold distance DP of <FIG>) of the camera's protective field <NUM> is defined in terms of the horizontal distance <NUM> between the AGV <NUM> and a given obstacle <NUM>. This distance is given as dvo in <FIG> (although distance <NUM> is referred to as being a horizontal distance <NUM> in the case of a perfectly horizontal floor <NUM>, more generally distance <NUM> is considered a distance between the AGV <NUM> and the obstacle <NUM> along a line substantially parallel to a slope of the floor <NUM>). Control output component <NUM> will generate a control output <NUM> initiating a safe action in response to determining that distance dvo is less that this minimum safe distance. The measured distance <NUM> between a pixel of the 3D TOF camera <NUM> and the obstacle <NUM> along the optical axis (or field view axis) of the pixel is given as dso in <FIG>. The field of view angle β represents the angle between the pixel's optical axis and the upper boundary <NUM> of the camera's total field of view.

The difference between distance <NUM> and distance <NUM> depends on the rotation or inclination angle α of the TOF 3D camera <NUM> relative to the horizontal <NUM>. The rotation angle α also determines the maximum and minimum distance range within which an obstacle <NUM> can be detected.

During normal operation, camera <NUM> projects the pulsed light beam <NUM> ahead of AGV <NUM> (as shown in <FIG>), tilted downward at rotation or inclination angle α, and the camera's distance determination component <NUM> generates pixel-wise distance data <NUM> based on a portion of the light reflected from the floor <NUM> back to the camera's imaging components (e.g., photo-sensor component <NUM>, lens element <NUM>, photo-receiver array <NUM>, etc.). The distance data <NUM> can be generated as described above in connection with <FIG>. This distance data <NUM> comprises an array of distance values dso for respective pixels of the camera's pixel array. In some embodiments, distance determination component <NUM> may only compute distances dso for a subset of the camera's pixels that correspond to the defined protective field <NUM>. Distance determination component <NUM> updates the distance values dso substantially in real time during normal operation of the collision avoidance system <NUM>.

Distance determination component <NUM> also computes the horizontal distance dvo corresponding to each pixel (or each pixel corresponding to the protective field <NUM>) based on these measured values of dso. In an example embodiment, distance determination component <NUM> can compute values of dvo using a trigonometric approach based on the distance values dso as well as the inclination angle α of the camera <NUM> and the pixel field of view angle β, according to the following equation or a variation thereof: <MAT>.

That is, distance determination component <NUM> can determine the horizontal distance dvo for a given pixel - that is, the horizontal distance from the AGV <NUM> to a point on the obstacle <NUM> seen by the pixel - based on the pixel's measured distance value dso multiplied by the cosine of the sum of the inclination angle α and the pixel field of view angle β. In some embodiments, the pixel field of view angle β may be a fixed parameter stored on the camera's memory (e.g., memory <NUM>) for each pixel stored on the camera's memory. The inclination angle α may be measured internally by the collision avoidance system <NUM>, or may be a fixed parameter set manually by a user based on camera installation measurements.

Distance determination component <NUM> can update the calculated distance value dvo for each pixel substantially in real time as the measured distance dso is updated during normal operation. This calculated distance value dvo represents the horizontal distance from the AGV <NUM> to the point on the obstacle <NUM> corresponding to the pixel. These calculated values of dvo are provided to the control output component <NUM>, which analyzes the values of dvo to determine whether the obstacle <NUM> is within the defined minimum safe distance (the protective field's threshold distance Dp) from the AGV <NUM>. In response to determining that the obstacle <NUM> is within the minimum safe distance, the control output component <NUM> can generate control output <NUM>, thereby initiating a safety action intended to mitigate a collision between obstacle <NUM> and AGV <NUM> (e.g., by diverting the path of the AGV <NUM>, slowing or stopping the AGV <NUM>, or triggering a warning indication in the case of human obstacles).

Control output component <NUM> can apply any suitable analysis to the values of dvo to determine whether to initiate control output <NUM>. For example, in some embodiments control output component <NUM> may generate the control output <NUM> if any one of the array of values of dvo (corresponding to respective pixels of the protective field) are less than the defined minimum safe distance. In such embodiments, control output component <NUM> may generate the control output <NUM> only if the value of dvo remains less than the minimum safe distance for a duration of time in excess of a defined debounce duration. In other embodiments, control output component <NUM> may generate the control output <NUM> in response to determining that a number of pixels in excess of a defined threshold number N have corresponding distance values dvo that are less than the minimum safe distance. This approach may also be associated with a debouce duration in some embodiments, such that control output component <NUM> generates the control output <NUM> in response to determining that the total number of pixels having respective values of dvo that are less than the minimum safe distance has exceeded N pixels for a defined duration of time (a debounce duration). This debounce duration can be set to mitigate premature initiation of safety actions in response to momentary or glancing intersections between the protective field <NUM> and obstacles <NUM>, or in response to momentary loss of distance information for one or more pixels, while still ensuring that safety measures are reliably performed in response to legitimate collision hazards.

To verify proper operation of the collusion avoidance system's distance measurement components, the diagnostic component <NUM> of collision avoidance system <NUM> can initiate a diagnostic sequence at defined intervals (e.g., periodically, or in response to a defined conditions) during normal operation. This diagnostic sequence is intended to verify that all optical and sensing elements in the return light path are properly operating and collaborating to produce accurate distance values dso. <FIG> is a diagram illustrating a diagnostic approach that can be carried out by diagnostic component <NUM> to confirm that collision avoidance system <NUM> is correctly measuring distances. <FIG> is a block diagram illustrating example data flows carried out by the collision avoidance system <NUM> during the diagnostic sequence. Typically, diagnostic component <NUM> will initiate the diagnostic sequence only if there are no obstacles <NUM> within the protective field <NUM> and the 3D TOF camera <NUM> has an unobstructed view of the floor <NUM>. If an obstacle <NUM> is within the camera's protective field <NUM> at the time of a scheduled diagnostic test, the diagnostic component <NUM> will postpone the diagnostic test until such time as the camera's protective field is free of obstructions.

With reference to <FIG>, for a given height hc of the camera above the floor <NUM> (where height hc is the vertical distance of the camera <NUM> from the floor <NUM>), the distance <NUM> from each pixel to the ground or floor <NUM> along the optical axis of the pixel a known function of hc. This distance <NUM> is given as dsg in <FIG>, and can be derived as a function of the camera height hc according to the following equation: <MAT>.

That is, the distance dsg from a pixel of camera <NUM> to the floor <NUM> along the optical axis of the pixel is equal to the vertical height hc of the camera <NUM> above the floor <NUM> divided by the sine of the sum of the camera inclination angle α and the pixel field of view angle β.

Accordingly, collision avoidance system <NUM> can measure this distance dsg during the diagnostic sequence and compare the measured value of dsg with this expected value of dsg to determine whether the system's optical and imaging components along the return light path are generating correct distance values. In particular, during the diagnostic sequence, emitter component <NUM> emits a pulsed light beam <NUM> (e.g., via emitting lens element <NUM>) directed to the floor <NUM>, and photo-sensor component <NUM> generates pixel data <NUM> based on analysis of light pulses reflected back to the camera from the floor <NUM> (as described above in connection with <FIG>). The pixel data <NUM> is provided to distance determination component <NUM>, which computes distance values for each pixel <NUM> of the resulting image based on the pixel data <NUM> (e.g., based on equation (<NUM>) or a variation thereof), resulting in a measured distance value dsg for each pixel <NUM>.

For each pixel, diagnostic component <NUM> compares that pixel's measured value of dsg <NUM> with the pixel's expected value of dsg <NUM> (see <FIG>). Typically, since different pixels of the camera's pixel array will be associated with different pixel field of view angles β, each pixel will have its own expected value of dsg <NUM> that is a function of this angle β (given by equation (<NUM>)). In some configurations, the expected value of dsg <NUM> for a given pixel may depend on the row of the pixel array within which the pixel is located, which determines the pixel's field of view angle β relative to the upper boundary <NUM> of the camera's total field of view. The expected value of dsg <NUM> for each pixel of the array can be stored in memory (e.g., memory <NUM>) associated with the collision avoidance system <NUM>. In some embodiments, diagnostic component <NUM> can be configured to automatically update the expected values of dsg <NUM> in response to detected changes in the camera's height hc or inclination angle α. In such embodiments, camera <NUM> may be equipped with a height measurement component that measures its own distance hc from the floor <NUM>, and/or level sensing components that measure the camera's inclination angle α relative to the horizontal <NUM>. Based on measurements from these components, diagnostic component <NUM> can maintain accurate estimates of the expected distances dsg <NUM> that each pixel should be reporting during the diagnostic sequence.

Diagnostic component <NUM> can perform distance measurement validation for each pixel <NUM> in the pixel array during the diagnostic sequence. If the measured values of dsg <NUM> for a number of pixels deviate from their corresponding respective values in excess of a defined tolerance, diagnostic component <NUM> can notify control output component <NUM> that reliability of the collision avoidance system's distance measurements has fallen below an acceptable level, causing control output component <NUM> to issue a control output <NUM> that places the AGV <NUM> in a safe state (e.g., by stopping the AGV or otherwise placing the AGV in a state in which faulty distance measurements will not result in harm or damage). The AGV <NUM> can then be decommissioned until the distance measurement faults are corrected (e.g., by replacing the camera <NUM> or the entire collision avoidance system <NUM>, or by otherwise correcting the fault in the camera's distance measurement components). Alternatively, if diagnostic component <NUM> determines that the measured distance values dsg <NUM> align with their corresponding expected values <NUM>, normal operation of the AGV and its collision avoidance system <NUM> is permitted to continue.

Diagnostic component <NUM> can use substantially any criteria for determining when the camera's distance measurement reliability has fallen below acceptable levels and necessitates placement of the AGV <NUM> in a safe state. For example, diagnostic component <NUM> may initiate control output <NUM> in response to determining that a number of pixels above a defined threshold number report measured values of dsg <NUM> that deviate from their corresponding expected values <NUM> in excess of a defined tolerance. Such configurations may mitigate unnecessary decommissioning of the AGV <NUM> when only a small number of pixels have become unreliable, since the number of functioning pixels may remain sufficiently large in such scenarios to ensure a high level of safety reliability.

If the ground or floor <NUM> is relatively flat over all areas on which the AVG <NUM> will traverse, measured test distance values dsg from the camera <NUM> to the floor will remain somewhat consistent regardless of the AGV's location, and measured distances dso from the camera <NUM> to obstacles <NUM> (as well as corresponding calculated values dvo from the AGV to the obstacle <NUM> based on the measured distances dso) will be accurate as the AVG <NUM> moves around the area, barring faults in the system's distance measuring components. If the floor <NUM> is uneven - that is, the floor <NUM> changes in inclination at different areas - the test distance value dsg returned by a pixel may be inconsistent as the AGV <NUM> moves over this variable terrain. <FIG> is a diagram illustrating a scenario in which AGV <NUM> is moving over a bump <NUM> in the floor <NUM>. In this example, the front wheels of the AGV <NUM> have been elevated by the bump <NUM>, causing the AGV <NUM> and its associated camera <NUM> to change inclination. At the moment prior to the AGV's contact with the bump <NUM> (at time (t-<NUM>)), the calculated distance <NUM> from the AGV <NUM> to the obstacle <NUM> (based on measured distance <NUM> from the camera <NUM> to the obstacle <NUM>) for a given pixel was based on the distance dso measured to point <NUM> on the obstacle <NUM> (that is, the point <NUM> seen by the pixel). When the AGV <NUM> is elevated by bump <NUM> (at time t), the pixel's field of view is also elevated such that distance dso is measured to point <NUM>, which is considerably higher than the point that would be used as the basis for measured distance dso (and calculated distance dvo) if the AVG <NUM> remained on flat terrain. This measurement discontinuity as the AGV <NUM> is traversing uneven terrain can cause the calculated distance dvo to be inconsistent as the AGV <NUM> and obstacle <NUM> approach one another.

In general, the inclination of the AGV <NUM> and camera <NUM> will vary at each bump <NUM> relative to the average slope of the floor <NUM>. This causes the central axis of the camera <NUM> to move relative to the floor <NUM> and relative to obstacles <NUM> located on the floor <NUM>. This variation in inclination can render measurements of distance <NUM> from the AGV <NUM> to obstacles in the AGV's path inaccurate.

To address this issue, the collision avoidance system <NUM> includes level component <NUM> capable of measuring an inclination of the AGV <NUM> and correcting distance measurements based on this measured inclination. <FIG> is a diagram illustrating an example AGV <NUM> that includes a collision avoidance system <NUM> with an associated level component <NUM> used to compensate for variations of inclination. According to the invention, level component <NUM> is configured to measure and quantify a degree of inclination of the AGV <NUM> relative to the horizontal, or to otherwise measure and quantify a direction and degree of a change in the AGV's inclination relative to an average slope of the ground or floor <NUM>. Based on the measured degree of inclination, camera <NUM> automatically adjusts the protective field <NUM> to compensate for the measured inclination. In the example depicted in <FIG>, the AGV <NUM> is inclined upward due to rolling over bump <NUM>. Level component <NUM> measures the direction and magnitude of this change in the AVG's inclination or tilt, and camera <NUM> adjusts the protective field <NUM> downward by an amount calculated to compensate for the degree of inclination measured by the level component <NUM>. In particular, the protective field <NUM> is adjusted such that the protective field <NUM> intersects with a section of the obstacle <NUM> that would be covered by the protective field <NUM> if the AGV <NUM> was on level ground (or had an inclination that matched the average slope of the floor <NUM>) , thus ensuring reasonable measurement continuity from the moment just prior to contact with the bump <NUM>.

<FIG> is a block diagram illustrating components of TOF collision avoidance system <NUM> including a level component <NUM> according to one or more embodiments. As described above, level component <NUM> measures the direction and degree of inclination of the AGV <NUM> (or the collision avoidance system <NUM> itself) and generates compensation instructions <NUM> directed to the control output component <NUM> for adjusting the protective field <NUM> to be monitored for intrusions. This adjustment can comprise, for example, shifting the defined section <NUM> of point cloud <NUM> - which represents the x-y boundaries of the protective field <NUM> - upward or downward within the boundaries of the point cloud <NUM> to maintain measurement continuity relative to the obstacle <NUM>.

Although <FIG> depicts protective field adjustment as being performed by control output component <NUM> (e.g., by shifting the section <NUM> of the image being monitored for intrusions to compensate for detected inclinations) other approaches for adjusting the protective field <NUM> are also within the scope of one or more embodiments. Inclination compensation can also be performed mechanically in some embodiments. For example, in some embodiments camera <NUM> can be installed in a gimbal stabilizer mounted to the front of AGV <NUM>, thereby maintaining a substantially consistent downward tilt of camera <NUM> regardless of the inclination of the AGV <NUM>.

The approaches for compensating for variations in the AGV's degree of incline can also ensure stable and consistent distance measurements dvo in scenarios in which the average slope of the ground or floor <NUM> is not horizontal. <FIG> is a diagram illustrating a scenario in which AGV <NUM> is traversing a downwardly sloping floor <NUM>. In this scenario, it is assumed that floor <NUM> has an uneven surface comprising various bumps or depressions, with an average slope conforming to the downward incline depicted in <FIG>. When AGV <NUM> begins traversing the downward incline, level component <NUM> will measure this change in inclination and adjust the protective field <NUM> accordingly, as discussed above. Thereafter, while the AVG <NUM> continues to traverse this downward incline, the level component <NUM> will hold this position of the protective field <NUM> relative to the average slope of the floor <NUM>, compensating for bumps and depressions encountered by the AGV <NUM> during its descent down the incline.

Although the examples described above in connection with <FIG> consider scenarios in which the AVG <NUM> tilts in one axis (namely, the front and rear of the AVG <NUM> moving up or down due to irregularities in the terrain), this approach for compensating for variations in the AGV's incline can also be extended to scenarios in which the AVG <NUM> tilts left or right as well.

Embodiments described herein can improve the safety reliability of collision avoidance systems by using a multi-pixel 3D TOF camera to monitor a larger volume in front of a mobile asset relative to a 2D scanner. By tilting the TOF camera such that the field of view monitors a volume that incudes the ground as well as any obstacles in front of the mobile asset, diagnostic features can be integrated into the collision avoidance system that leverage return light from the ground to confirm reliability and accuracy of the system's distance measurement components (including optical and imaging components), thereby further improving the safety rating of the system. The system's object detection reliability can be further improved by automatically adjusting the monitored protective field to compensate for uneven ground.

<FIG> illustrate methodologies in accordance with one or more embodiments of the subject application. While, for purposes of simplicity of explanation, the methodologies shown herein is shown and described as a series of acts, it is to be understood and appreciated that the subject innovation is not limited by the order of acts, as some acts may, in accordance therewith, occur in a different order and/or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a methodology could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all illustrated acts may be required to implement a methodology in accordance with the innovation. Furthermore, interaction diagram(s) may represent methodologies, or methods, in accordance with the subject disclosure when disparate entities enact disparate portions of the methodologies. Further yet, two or more of the disclosed example methods can be implemented in combination with each other, to accomplish one or more features or advantages described herein.

<FIG> illustrates an example methodology <NUM> for avoiding collisions between an autonomous mobile asset (e.g., an AGV, an AMR, or another type of guided or intelligently mobile vehicle or machine). Initially, at <NUM>, pulsed light is emitted into a space in front of an autonomous or guided mobile asset by a 3D TOF camera mounted on the front of the mobile asset at a downward incline. The TOF camera is inclined downward such that a portion of the floor is include in the protective field of the TOF camera and reflects a portion of light emitted by the camera back to the camera's photo-detector. At <NUM>, point cloud data is generated for the viewing space based on reflected pulses received at a photo-detector array of the TOF camera. The point cloud data comprises distance values dso representing distances from respective pixels of the photo-detector array to corresponding points on surfaces within the viewing space.

At <NUM>, a pixel of the photo-detector array is selected. At <NUM>, a distance dvo is calculated for the pixel. Distance dvo represents a horizontal distance from the mobile asset to a point on a surface corresponding to the pixel (that is, a point within the pixel's field of view), or a distance from the mobile asset to the point along a line substantially parallel with a slope of the ground or floor on which the mobile asset is moving. Distance dvo can be calculated based on the measured distance dso generated at step <NUM>, the inclination angle α of the TOF camera (that is, the angle of the camera's downward inclination relative to horizontal), and a field of view angle β for the pixel. In some embodiments, equation (<NUM>) above, or a variation thereof, can be used to calculate distance dvo based on these factors. Any suitable technique can be used to determine the inclination angle α of the TOF camera and the field of view angle β for the pixel. For example, the inclination angle α may be measured dynamically using a level component within the camera itself, which reports its inclination angle to the system. Alternatively, inclination angle α may be set manually based on an as-installed angle of tilt measured by an engineer and entered into the system. The field of view angle β for the pixel may be a fixed parameter of the pixel stored in memory and referenced by the system to calculate distance dvo.

At <NUM>, a determination is made as to whether remaining pixels require calculation of an associated distance dvo. In some embodiments, values of distance dvo may be calculated for all pixels of photo-detector's pixel array. In other embodiments, the system may only calculate values of distance dvo for a subset of pixels corresponding to the defined protective field. If additional pixels require calculation of distance dvo (YES at step <NUM>) the methodology returns to step <NUM>, where another pixel is selected, and distance dvo is calculated for the newly selected pixel at step <NUM>. Alternatively, if values of distance dvo have been calculated for all necessary pixels (NO at step <NUM>), the methodology proceeds to step <NUM>.

At <NUM>, a determination is made as to whether the distance values dvo calculated by iterations of steps <NUM>-<NUM> satisfy a criterion relative to a defined minimum safe distance of the protective field. In general, the criterion defines a condition indicative of intrusion of an obstacle within the protective field monitored by the camera, which requires initiation of a collision avoidance action. In some embodiments, the criterion may define a maximum number of pixels whose corresponding distance value dvo are permitted to be less than the defined minimum safe distance from the mobile asset. Other criteria indicative of an intrusion of an obstacle are also within the scope of one or more embodiments.

If the distance values dvo do not satisfy the criterion (NO at step <NUM>), the methodology returns to step <NUM> and steps <NUM>-<NUM> repeat. Alternatively, if the distance values dvo satisfy the criterion (YES at step <NUM>), the methodology proceeds to step <NUM>, where a control output is generated that alters operation of the mobile asset in a manner intended to mitigate a collision with the detected obstacle. In various embodiments, the control output may alter the current trajectory of the mobile asset, slow the mobile asset, or stop the mobile asset. The control output may also initiate an audible or visual warning of a possible collision.

<FIG> illustrates a first part of an example methodology 1700A for verifying that a 3D TOF camera that is part of a collision avoidance system is generating accurate distance measurements. The 3D TOF camera is mounted to the front of an autonomous or guided mobile asset at a downward inclination such that a portion of the light emitted by the camera is reflected back to the camera by the floor. Initially, at <NUM>, a determination is made as to whether a diagnostic sequence is initiated. In some embodiments, the collision avoidance system can be configured to initiate the diagnostic sequence at regular or semi-regular intervals (e.g., once an hour, once a day, etc.). In other embodiments, the system may initiate the diagnostic sequence in response to detected conditions indicative of a possible loss of distance measurement accuracy.

If a diagnostic sequence is initiated (YES at step <NUM>), the methodology proceeds to step <NUM>, where a determination is made as to whether an obstacle is detected in the TOF camera's field of view (e.g., based on the current measured distance values measured by the camera's pixel array). In general, the diagnostic sequence should be run only if no obstacles are currently within the camera's protective field. If an obstacle is detected (YES at step <NUM>), the diagnostic sequence is postponed (step <NUM>). When the diagnostic sequence is initiated and no obstacles are detected within the camera's field of view (NO at step <NUM>), the methodology proceeds to step <NUM>, where pulsed light is emitted into a space in front of the mobile asset. Since the camera is tilted downward, a portion of the emitted light is reflected back to the camera's photo-detector by the floor.

At <NUM>, point cloud data is generated for a portion of the space corresponding to the floor based on pulses reflected by the floor and received at the photo-detector array of the TOF camera. The point cloud data comprises distance values dsg representing distances from respective pixels of the photo-detector array to corresponding points on the floor (that is, points within the fields of view of the respective pixels).

At <NUM>, a variable N is set to zero. At <NUM>, a pixel of the camera's pixel array is selected from among the subset of pixels having the floor within their fields of view. At <NUM>, an expected distance dsg for the pixel selected at step <NUM> is calculated based on a height hc of the TOF camera from the ground, an inclination angle α of the camera, and a field of view angle β for the pixel. The inclination angle α and field of view angle β can be determined as described above in connection with step <NUM> of methodology <NUM>. In some embodiments, the height hc of the camera can be measured by height measurement components within the camera itself, which report the distance of the camera from the floor. In other embodiments, the height hc may be entered by an engineer as a fixed parameter based on as-installed measurements. In an example calculation approach, the expected value of distance dsg can be obtained using equation (<NUM>) or a variation thereof.

The methodology then proceeds to the second part 1700B illustrated in <FIG>. At <NUM>, a determination is made as to whether the measured value of dsg obtained at step <NUM> matches the expected value of dsg calculated at step <NUM> within a defined tolerance. If the measured value of dsg does not match the expected value of dsg within the defined tolerance (NO at step <NUM>), the methodology proceeds to step <NUM>, where the variable N is incremented. Variable N counts the number of pixels whose distance values dsg do not match their expected values. The methodology then proceeds to step <NUM>. If, at step <NUM>, it is determined that the measured value of dsg matches its expected value (YES at step <NUM>), the methodology proceeds to step <NUM> without incrementing N at step <NUM>.

At <NUM>, a determination is made as to whether more pixels are to be assessed. In some embodiments, the system may diagnose all pixels of the array having the floor within their field of view. In other embodiments, a smaller representative sample of pixels may be assessed. If more pixels are to be diagnosed (YES at step <NUM>), the methodology returns to step <NUM>, where another pixel is selected, and steps <NUM>-<NUM> are repeated for the newly selected pixel. When all pixels to be diagnosed have been assessed (NO at step <NUM>), the methodology proceeds to step <NUM>, where a determination is made as to whether the value of variable N is greater than a defined threshold indicative of an overall distance measurement fault. If the value of N is not greater than the threshold value (NO at step <NUM>), the camera has passed the diagnosis and the methodology returns to step <NUM> to await the next diagnostic sequence. Alternatively, if the value of N is greater than the threshold value (YES at step <NUM>), it is determined that the camera's distance measurement components are faulty and the methodology proceeds to step <NUM>, where the mobile asset is placed into a safe state. A notification indicating a distance measurement fault may also be generated.

<FIG> illustrates a first part of an example methodology 1800A for avoiding collisions using a collision avoidance system that adjusts its protective field to compensate for changes in inclination of an associated mobile asset. Initially, at <NUM>, a protective field is set for a 3D TOF camera mounted on the front of an autonomous or guided mobile asset at a downward inclination. The TOF camera is inclined downward such that a portion of the floor is included in the protective field of the TOF camera and reflects a portion of light emitted by the camera back to the camera's photo-detector for diagnostic purposes, as discussed above in connection with the methodology illustrated in <FIG>. The protective field defines a space within the camera's field of view to be monitored for obstructions, such that detection of an obstruction within the protective field causes the mobile asset to perform a collision avoidance action. In an example embodiment, the x-y boundaries of the protective field can be defined as a section of the camera's total pixel array to be monitored for intrusions, and the z-direction boundary can be defined as one or more minimum safe distances associated with the subset of pixels that make up the selected section of the pixel array.

At <NUM>, pulsed light is emitted into the space in front of the mobile asset by the 3D TOF camera. At <NUM>, point cloud data is generated for the space in front of the asset based on reflected pulses received at the camera's photo-detector array. The point cloud data comprises distance values dso representing distances from respective pixels of the photo-detector array to corresponding points on surfaces within the space.

At <NUM>, a direction and a degree of inclination of the mobile asset relative to horizontal is measured. This measurement can be performed, for example, by a level sensor mounted on or within the asset as part of the collision avoidance system. At <NUM>, a determination is made as to whether the degree of inclination deviates from horizontal in excess of a defined tolerance. If the degree of inclination deviates from horizontal in excess of the tolerance (YES at step <NUM>), the methodology proceeds to step <NUM>, where the protective field defined at step <NUM> is adjusted to compensate for the direction and degree of the inclination measured at step <NUM>. For example, if the asset is inclined upward, the protective field can be adjusted downward to ensure continuity of distance measurements on points within the space in front of the asset. The methodology then proceeds to the second part 1800B illustrated in <FIG>. If the degree of inclination does not deviate from horizontal in excess of the tolerance (NO at step <NUM>), the methodology proceeds direction to the second part 1800B without performing a protective field adjustment (skipping step <NUM>).

At <NUM>, a pixel is selected from a section of the photo-detector array corresponding to the protective field. At <NUM>, a distance dvo is calculated based on the pixel's measured distance value dso obtained at step <NUM> as well as an inclination angle α of the camera and a field of view angle β for the pixel. Angles α and β can be determined as described above in connection with step <NUM> of methodology <NUM>. Distance dvo represents the distance from the mobile asset to a point on a surface corresponding to the pixel (that is, a point within the pixel's field of view).

At <NUM>, a determination is made as to whether remaining pixels require calculated values of dvo. In one or more embodiments, steps <NUM> and <NUM> can be iterated for each pixel corresponding to the protective field. In other embodiments, steps <NUM> and <NUM> can be iterated for all pixels of the array, but only those corresponding to the protective field will be evaluated for intrusions in subsequent steps. If remaining pixels require distance calculations (YES at step <NUM>), the methodology returns to step <NUM>, where another pixel is selected, and a distance value dvo is calculated for the newly selected pixel.

When distance values dvo have been calculated for all necessary pixels (NO at step <NUM>), the methodology proceeds to step <NUM>, where a determination is made as to whether the distance values dvo satisfy a defined criterion relative to the minimum safe distance of the protective field (similar to step <NUM> of methodology <NUM>). The criterion defines a condition indicative of intrusion of an obstacle within the protective field, which requires initiation of a collision avoidance action. If the distance values dvo satisfy the criterion (YES at step <NUM>), the methodology proceeds to step <NUM>, where a control output is generated that alters operation of the mobile asset to mitigate a collision with a detected obstacle (similar to step <NUM> of methodology <NUM>). Alternatively, if the distance values dvo do not satisfy the criterion (NO at step <NUM>), the methodology returns to step <NUM> and the methodology repeats.

Embodiments, systems, and components described herein, as well as control systems and automation environments in which various aspects set forth in the subject specification can be carried out, can include computer or network components such as servers, clients, programmable logic controllers (PLCs), automation controllers, communications modules, mobile computers, on-board computers for mobile vehicles, wireless components, control components and so forth which are capable of interacting across a network. Computers and servers include one or more processors-electronic integrated circuits that perform logic operations employing electric signals-configured to execute instructions stored in media such as random access memory (RAM), read only memory (ROM), a hard drives, as well as removable memory devices, which can include memory sticks, memory cards, flash drives, external hard drives, and so on.

Similarly, the term PLC or automation controller as used herein can include functionality that can be shared across multiple components, systems, and/or networks. As an example, one or more PLCs or automation controllers can communicate and cooperate with various network devices across the network. This can include substantially any type of control, communications module, computer, Input/Output (I/O) device, sensor, actuator, and human machine interface (HMI) that communicate via the network, which includes control, automation, and/or public networks. The PLC or automation controller can also communicate to and control various other devices such as standard or safety-rated I/O modules including analog, digital, programmed/intelligent I/O modules, other programmable controllers, communications modules, sensors, actuators, output devices, and the like.

The network can include public networks such as the internet, intranets, and automation networks such as control and information protocol (CIP) networks including DeviceNet, ControlNet, safety networks, and Ethernet/IP. Other networks include Ethernet, DH/DH+, Remote I/O, Fieldbus, Modbus, Profibus, CAN, wireless networks, serial protocols, and so forth. In addition, the network devices can include various possibilities (hardware and/or software components). These include components such as switches with virtual local area network (VLAN) capability, LANs, WANs, proxies, gateways, routers, firewalls, virtual private network (VPN) devices, servers, clients, computers, configuration tools, monitoring tools, and/or other devices.

In order to provide a context for the various aspects of the disclosed subject matter, <FIG> and <FIG> as well as the following discussion are intended to provide a brief, general description of a suitable environment in which the various aspects of the disclosed subject matter may be implemented. While the embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the embodiments can be also implemented in combination with other program modules and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the inventive methods can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.

The illustrated embodiments of the embodiments herein can be also practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

Computing devices typically include a variety of media, which can include computer-readable storage media, machine-readable storage media, and/or communications media, which two terms are used herein differently from one another as follows. Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media or machine-readable storage media can be implemented in connection with any method or technology for storage of information such as computer-readable or machine-readable instructions, program modules, structured data or unstructured data.

Computer-readable storage media can include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disc (BD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, solid state drives or other solid state storage devices, or other tangible and/or non-transitory media which can be used to store desired information. In this regard, the terms "tangible" or "non-transitory" herein as applied to storage, memory or computer-readable media, are to be understood to exclude only propagating transitory signals per se as modifiers and do not relinquish rights to all standard storage, memory or computer-readable media that are not only propagating transitory signals per se.

Computer-readable storage media can be accessed by one or more local or remote computing devices, e.g., via access requests, queries or other data retrieval protocols, for a variety of operations with respect to the information stored by the medium.

Communications media typically embody computer-readable instructions, data structures, program modules or other structured or unstructured data in a data signal such as a modulated data signal, e.g., a carrier wave or other transport mechanism, and includes any information delivery or transport media. The term "modulated data signal" or signals refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in one or more signals. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

With reference again to <FIG>, the example environment <NUM> for implementing various embodiments of the aspects described herein includes a computer <NUM>, the computer <NUM> including a processing unit <NUM>, a system memory <NUM> and a system bus <NUM>. The system bus <NUM> couples system components including, but not limited to, the system memory <NUM> to the processing unit <NUM>. The processing unit <NUM> can be any of various commercially available processors. Dual microprocessors and other multi-processor architectures can also be employed as the processing unit <NUM>.

The system bus <NUM> can be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory <NUM> includes ROM <NUM> and RAM <NUM>. A basic input/output system (BIOS) can be stored in a non-volatile memory such as ROM, erasable programmable read only memory (EPROM), EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer <NUM>, such as during startup. The RAM <NUM> can also include a highspeed RAM such as static RAM for caching data.

The computer <NUM> further includes an internal hard disk drive (HDD) <NUM> (e.g., EIDE, SATA), one or more external storage devices <NUM> (e.g., a magnetic floppy disk drive (FDD) <NUM>, a memory stick or flash drive reader, a memory card reader, etc.) and an optical disk drive <NUM> (e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.). While the internal HDD <NUM> is illustrated as located within the computer <NUM>, the internal HDD <NUM> can also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment <NUM>, a solid state drive (SSD) could be used in addition to, or in place of, an HDD <NUM>. The HDD <NUM>, external storage device(s) <NUM> and optical disk drive <NUM> can be connected to the system bus <NUM> by an HDD interface <NUM>, an external storage interface <NUM> and an optical drive interface <NUM>, respectively. The interface <NUM> for external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) <NUM> interface technologies. Other external drive connection technologies are within contemplation of the embodiments described herein.

The drives and their associated computer-readable storage media provide nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For the computer <NUM>, the drives and storage media accommodate the storage of any data in a suitable digital format. Although the description of computer-readable storage media above refers to respective types of storage devices, it should be appreciated by those skilled in the art that other types of storage media which are readable by a computer, whether presently existing or developed in the future, could also be used in the example operating environment, and further, that any such storage media can contain computer-executable instructions for performing the methods described herein.

A number of program modules can be stored in the drives and RAM <NUM>, including an operating system <NUM>, one or more application programs <NUM>, other program modules <NUM> and program data <NUM>. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM <NUM>. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

Computer <NUM> can optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system <NUM>, and the emulated hardware can optionally be different from the hardware illustrated in <FIG>. In such an embodiment, operating system <NUM> can comprise one virtual machine (VM) of multiple VMs hosted at computer <NUM>. Furthermore, operating system <NUM> can provide runtime environments, such as the Java runtime environment or the. NET framework, for application programs <NUM>. Runtime environments are consistent execution environments that allow application programs <NUM> to run on any operating system that includes the runtime environment. Similarly, operating system <NUM> can support containers, and application programs <NUM> can be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

Further, computer <NUM> can be enable with a security module, such as a trusted processing module (TPM). For instance with a TPM, boot components hash next in time boot components, and wait for a match of results to secured values, before loading a next boot component. This process can take place at any layer in the code execution stack of computer <NUM>, e.g., applied at the application execution level or at the operating system (OS) kernel level, thereby enabling security at any level of code execution.

A user can enter commands and information into the computer <NUM> through one or more wired/wireless input devices, e.g., a keyboard <NUM>, a touch screen <NUM>, and a pointing device, such as a mouse <NUM>. Other input devices (not shown) can include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control, or other remote control, a joystick, a virtual reality controller and/or virtual reality headset, a game pad, a stylus pen, an image input device, e.g., camera(s), a gesture sensor input device, a vision movement sensor input device, an emotion or facial detection device, a biometric input device, e.g., fingerprint or iris scanner, or the like. These and other input devices are often connected to the processing unit <NUM> through an input device interface <NUM> that can be coupled to the system bus <NUM>, but can be connected by other interfaces, such as a parallel port, an IEEE <NUM> serial port, a game port, a USB port, an IR interface, a BLUETOOTH® interface, etc..

A monitor <NUM> or other type of display device can be also connected to the system bus <NUM> via an interface, such as a video adapter <NUM>. In addition to the monitor <NUM>, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc..

The computer <NUM> can operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s) <NUM>. The remote computer(s) <NUM> can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer <NUM>, although, for purposes of brevity, only a memory/storage device <NUM> is illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN) <NUM> and/or larger networks, e.g., a wide area network (WAN) <NUM>. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to a global communications network, e.g., the Internet.

When used in a LAN networking environment, the computer <NUM> can be connected to the local network <NUM> through a wired and/or wireless communication network interface or adapter <NUM>. The adapter <NUM> can facilitate wired or wireless communication to the LAN <NUM>, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter <NUM> in a wireless mode.

When used in a WAN networking environment, the computer <NUM> can include a modem <NUM> or can be connected to a communications server on the WAN <NUM> via other means for establishing communications over the WAN <NUM>, such as by way of the Internet. The modem <NUM>, which can be internal or external and a wired or wireless device, can be connected to the system bus <NUM> via the input device interface <NUM>. In a networked environment, program modules depicted relative to the computer <NUM> or portions thereof, can be stored in the remote memory/storage device <NUM>. It will be appreciated that the network connections shown are example and other means of establishing a communications link between the computers can be used.

When used in either a LAN or WAN networking environment, the computer <NUM> can access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devices <NUM> as described above. Generally, a connection between the computer <NUM> and a cloud storage system can be established over a LAN <NUM> or WAN <NUM> e.g., by the adapter <NUM> or modem <NUM>, respectively. Upon connecting the computer <NUM> to an associated cloud storage system, the external storage interface <NUM> can, with the aid of the adapter <NUM> and/or modem <NUM>, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interface <NUM> can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer <NUM>.

The computer <NUM> can be operable to communicate with any wireless devices or entities operatively disposed in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant, communications satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., a kiosk, news stand, store shelf, etc.), and telephone. This can include Wireless Fidelity (Wi-Fi) and BLUETOOTH® wireless technologies. Thus, the communication can be a predefined structure as with a conventional network or simply an ad hoc communication between at least two devices.

<FIG> is a schematic block diagram of a sample computing environment <NUM> with which the disclosed subject matter can interact. The sample computing environment <NUM> includes one or more client(s) <NUM>. The client(s) <NUM> can be hardware and/or software (e.g., threads, processes, computing devices). The sample computing environment <NUM> also includes one or more server(s) <NUM>. The server(s) <NUM> can also be hardware and/or software (e.g., threads, processes, computing devices). The servers <NUM> can house threads to perform transformations by employing one or more embodiments as described herein, for example. One possible communication between a client <NUM> and servers <NUM> can be in the form of a data packet adapted to be transmitted between two or more computer processes. The sample computing environment <NUM> includes a communication framework <NUM> that can be employed to facilitate communications between the client(s) <NUM> and the server(s) <NUM>. The client(s) <NUM> are operably connected to one or more client data store(s) <NUM> that can be employed to store information local to the client(s) <NUM>. Similarly, the server(s) <NUM> are operably connected to one or more server data store(s) <NUM> that can be employed to store information local to the servers <NUM>.

What has been described above includes examples of the subject innovation. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the disclosed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the subject innovation are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

In particular and in regard to the various functions performed by the above described components, devices, circuits, systems and the like, the terms (including a reference to a "means") used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., a functional equivalent), even though not structurally equivalent to the disclosed structure, which performs the function in the herein illustrated exemplary aspects of the disclosed subject matter. In this regard, it will also be recognized that the disclosed subject matter includes a system as well as a computer-readable medium having computer-executable instructions for performing the acts and/or events of the various methods of the disclosed subject matter.

In addition, while a particular feature of the disclosed subject matter may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "includes," and "including" and variants thereof are used in either the detailed description or the claims, these terms are intended to be inclusive in a manner similar to the term "comprising.

In this application, the word "exemplary" is used to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion.

Claim 1:
A collision avoidance system (<NUM>), comprising:
a three-dimensional camera (<NUM>) configured to mount on a front side of a mobile asset at a downward incline that causes a portion of a ground or floor in front of the mobile asset to be included in a field of view of the three-dimensional camera, the three-dimensional camera comprising:
an emitter component (<NUM>) configured to emit light pulses into a space in front of the mobile asset;
a photo-sensor component (<NUM>) comprising an array of pixels, wherein respective pixels of the array of pixels are configured to convert a subset of the light pulses received from surfaces within the space to electrical energy proportional to the subset of the light pulses received at the pixels; and
a distance determination component (<NUM>) configured to:
determine measured distance values dso associated with the respective pixels based on analysis of the electrical energy, and
determine calculated distance values dvo associated with the respective pixels based on the measured distance values dso associated with the respective pixels, field of view angles β of the respective pixels, and an inclination angle α of the three-dimensional camera; and
a control output component (<NUM>) configured to generate a control output in response to a determination that at least a subset of the calculated distance values dvo satisfy a defined criterion indicative of a presence of an obstacle within a protective field of the three-dimensional camera, wherein the control output is configured to initiate a safety action that alters operation of the mobile asset to mitigate a collision with the obstacle,
wherein the defined criterion specifies that the control output is to be generated if a number of pixels having associated calculated distance values dvo that are less than a minimum safe distance exceeds a defined threshold number of pixels,
characterised in that
the collision avoidance system (<NUM>) further comprises a level component (<NUM>) configured to:
measure a direction of inclination and a degree of inclination of the mobile asset, and
initiate an adjustment of the protective field based on the direction of inclination and the degree of inclination.