Patent Description:
Guidance systems for material handling vehicles can be used to guide a material handling vehicle within an operating environment (e.g., a warehouse) and, in particular, along an aisle formed between storage racking (e.g., pallet racking). Such conventional guidance systems generally include external infrastructure to operate.

For example, a guidance system for a material handling vehicle may utilize a wire installed along the middle of an aisle, which generates a frequency that can be detected by the material handling vehicle. The material handling vehicle can include an antenna that can detect the wire and a processor to analyze the signals from the antenna. Using the signals generated by the antenna, the processor can determine a distance between the material handling vehicle and the wire and can accordingly determine how to control steering so as to align (i.e., center) the material handling vehicle over the wire. However, such external infrastructure is expensive and cumbersome to implement, and the guidance systems will only work in areas of the operating environment where such external infrastructure has been implemented.

In addition, for certain types of vehicles there are training requirements imposed by various government agencies, laws, rules and regulations. For example, OSHA imposes a duty on employers to train and supervise operators of various types of material handling vehicles. Recertification every three years is also required. In certain instances, refresher training in relevant topics shall be provided to the operator when required. In all instances, the operator remains in control of the material handling vehicle during performance of any actions. Further, a warehouse manager remains in control of the fleet of material handling vehicles within the warehouse environment. The training of operators and supervision to be provided by warehouse managers requires among other things proper operational practices including among other things that an operator remain in control of the material handling vehicle, pay attention to the operating environment, and always look in the direction of travel.

Document <CIT> discloses a method and apparatus for using unique landmarks to locate industrial vehicles at start-up. The vehicles may be forklifts or AGV's. Document <CIT> discloses a navigation system for an Automatic Guided Vehicle (AGV). In particular, the invention is directed to a laser based navigation system which is capable of guiding the AGV when the AGV's conventional navigation laser is blocked by stacked product or the like.

The present disclosure relates generally to materially handling vehicles, and, more specifically, to material handling vehicle guidance within an operating environment.

According to some aspects of the present disclosure, a material handling vehicle is provided. The material handling vehicle can include a vehicle management system configured to control the movement of the material handling vehicle, a sensor configured to measure a plurality of distances, wherein each distance is a distance to one of a plurality of external objects, and output a distance information corresponding to the plurality of measured distances, and a processor unit in communication with the sensor and the vehicle management system, the processor unit being configured to receive the distance information from the sensor, transform the distance information from a sensor coordinate system to a material handling vehicle coordinate system, identify an aisle feature based on the transformed distance information, create an aisle model based on the identified aisle feature, determine a guidance instruction based on the aisle model, and provide the guidance instruction to the vehicle management system.

According to some aspects of the present disclosure, a guidance method for a material handling vehicle is provided. The method can include measuring, with a sensor, a plurality of distances, wherein each distance is a distance to one of a plurality of external objects, and, using a processor unit in communication with the sensor: transforming the plurality of distance measurements from a sensor coordinate system to a material handling vehicle coordinate system, performing pattern recognition on the plurality of transformed distance measurements to identify an aisle feature, creating an aisle model based on the identified aisle feature, determining a guidance instruction for the material handling vehicle based on the aisle model, and providing the guidance instruction to the material handling vehicle.

The foregoing and other aspects and advantages of the disclosure will appear from the following description. In the description, reference is made to the accompanying drawings, which form a part hereof, and in which there is shown by way of illustration a preferred configuration of the disclosure. Such configuration does not necessarily represent the full scope of the disclosure, however, and reference is made therefore to the claims and herein for interpreting the scope of the disclosure.

Before any aspects of the present disclosure are explained in detail, it is to be understood that the present disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. The invention is capable of other non-limiting examples and of being practiced or of being carried out in various ways. Likewise, "at least one of A, B, and C," and the like, is meant to indicate A, or B, or C, or any combination of A, B, and/or C. Unless specified or limited otherwise, the terms "mounted," "secured," "connected," "supported," and "coupled" and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings.

It is also to be understood that any reference to an element herein using a designation such as "first," "second," and so forth does not limit the quantity or order of those elements, unless such limitation is explicitly stated. Rather, these designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not mean that only two elements may be employed there or that the first element must precede the second element in some manner.

The following discussion is presented to enable a person skilled in the art to make and use embodiments of the invention. Various modifications to the illustrated embodiments will be readily apparent to those skilled in the art, and the generic principles herein can be applied to other embodiments and applications without departing from embodiments of the invention. Thus, embodiments of the invention are not intended to be limited to embodiments shown but are to be accorded the widest scope consistent with the principles and features disclosed herein. The following detailed description is to be read with reference to the figures, in which like elements in different figures have like reference numerals. The figures, which are not necessarily to scale, depict selected embodiments and are not intended to limit the scope of embodiments of the invention. Skilled artisans will recognize the examples provided herein have many useful alternatives and fall within the scope of embodiments of the invention.

It is also to be appreciated that material handling vehicles (MHVs) are designed in a variety of classes and configurations to perform a variety of tasks. It will be apparent to those of skill in the art that the present disclosure is not limited to any specific MHV, and can also be provided with various other types of MHV classes and configurations, including for example, lift trucks, forklift trucks, reach trucks, SWING REACH® vehicles, turret trucks, side loader trucks, counterbalanced lift trucks, pallet stacker trucks, order pickers, transtackers, tow tractors, and man-up trucks, and can be commonly found in warehouses, factories, shipping yards, and, generally, wherever pallets, large packages, or loads of goods can be required to be transported from place to place. The various systems and methods disclosed herein are suitable for any of operator controlled, pedestrian controlled, remotely controlled, and autonomously controlled material handling vehicles. Further, the various system and methods disclosed herein are suitable for other vehicles, such as automobiles, busses, trains, tractor-trailers, farm vehicles, factory vehicles, and the like.

Advantageously, a guidance system for a MHV according to the present disclosure may operate without external infrastructure (e.g., without wire guidance and the associated components thereof), and can automatically detect and adjust the guidance of the MHV based on the width and/or positions of the one or more sides of the aisle (in either case, whether fixed or variable along the length of the aisle) without requiring any predetermined map of the aisle or its features. Storage racking can be used to store a wide variety of objects, some of which may have a depth that is larger than a corresponding depth of a storage rack. Consequently, objects may sometimes protrude beyond the storage rack into the aisle.

A guidance system according to the present disclosure may be field installed onto existing MHV's to increase functionality and operational efficiency. Such guidance systems may be implemented (e.g., interface and/or run concurrently) with other types of automation systems. For example, a guidance system according to the present disclosure may also be used for/with two-way traffic control in order to guide multiple MHV's in wide aisle to stay on the correct side of the aisle, may interface with an obstacle avoidance system, and the like.

In some non-limiting examples, a guidance system according to the present disclosure can accommodate for variability in an effective width of an aisle and the relative location of an effective centerline of the aisle. The guidance system can include a sensor unit and a processor unit that can be configured to guide a MHV along an aisle formed between storage racking. The guidance system may be configured to detect and identify between storage racking and objects being stored on the storage racking, as well as controlling a steering angle of the MHV as the MHV moves along the aisle. In some embodiments, the guidance system may also be configured to control the speed and/or braking of the MHV.

<FIG> illustrates a non-limiting example of a MHV <NUM>, which can be an automated guided vehicle (AGV), such as a fully- or semi-autonomous AGV, or a manually-operated vehicle. The MHV <NUM> may comprise a guidance system <NUM>. The MHV <NUM> may be configured as a manually operable MHV that can include a vehicle frame <NUM>, a power section <NUM>, a traction wheel <NUM>, and an operator compartment <NUM>. The power section <NUM> can be disposed within the vehicle frame <NUM> and may include a power source, for example, a battery <NUM> configured to supply power to various components of the MHV <NUM>. The battery <NUM> can supply power to a motor (not shown) and/or transmission (not shown) disposed within the power section <NUM>, which can be configured to drive the traction wheel <NUM>. In other non-limiting examples, the MHV <NUM> can include other types of power sources, for example, an engine.

In the illustrated non-limiting example, the traction wheel <NUM> may be configured as a pair of traction wheels (only one shown) arranged under the power section <NUM>, which can be configured to work together to propel the MHV <NUM> and to control a direction of travel (e.g., a steering angle) of the MHV <NUM>. In other non-limiting examples, the traction wheel <NUM> can be arranged in another location under the vehicle frame <NUM>. In some non-limiting examples, the traction wheel <NUM> may include only one wheel or more than two wheels and the functions of propelling the MHV <NUM> and controlling a direction of travel may be carried out by separate wheels.

In some embodiments, the MHV <NUM> can include a mast <NUM> for raising and lowering a fork assembly <NUM> (or, in other non-limiting examples, a platform, an operator cabin, or other implement assemblies). The mast <NUM> may be in the form of a telescoping mast with the fork assembly <NUM> attached thereto such that the fork assembly <NUM> can be selectively raised and lowered by the mast <NUM>. The fork assembly <NUM> may include one or more forks <NUM> that can engage a load, for example, a pallet. In the illustrated non-limiting example, the fork assembly <NUM> can include a pair of forks <NUM>. In some non-limiting examples, the fork assembly <NUM> can be coupled to the mast <NUM> by a reach actuator.

The operator compartment <NUM> can include a control interface <NUM> that can be configured to control one or more functions of the MHV <NUM>. For example, the control interface <NUM> can be communicatively coupled with a vehicle management system (VMS) <NUM> via wired or wireless communication (e.g., via a Controller Area Network (CAN bus)) that can be configured to control the various functions of the MHV <NUM>. In some embodiments, the VMS <NUM> may be internal to the MHV <NUM>. In some embodiments, the VMS <NUM> may be external to the MHV <NUM>, for example in communication with the MHV <NUM> via wired or wireless communication. The guidance system <NUM> may be configured to communicatively couple with the VMS <NUM>, for example using wired or wireless communication.

In some embodiments, the control interface <NUM> can include a control handle <NUM> configured to allow the operator to control a speed and direction of travel of the MHV <NUM>. Alternatively or additionally, the control interface <NUM> can include a display <NUM> (e.g., a touch-controlled display) that can be configured to allow the operator to control the operation of the MHV <NUM> and/or to provide the operator with visual and/or audible feedback. In some embodiments, the control interface <NUM> may be configured to allow the operator to engage and/or disengage the guidance system <NUM>. In some non-limiting examples, the control interface <NUM> can include various types of input devices (e.g., joysticks, steering wheels, push-buttons, etc.) and feedback devices (e.g., lights, screens, haptic feedback systems, speakers, etc.) for controlling the MHV <NUM>.

<FIG> illustrates a non-limiting example of a guidance system <NUM> configured to guide a MHV <NUM> along an aisle. In some embodiments, the guidance system <NUM> may comprise a processor unit <NUM> and a sensor unit <NUM>. The sensor unit <NUM> may be communicatively coupled with the processor unit <NUM>, for example via a CAN Bus, Ethernet, wireless communication link, or the like. As will be described in greater detail below, the sensor unit <NUM> and the processor unit <NUM> can work together to detect and identify (e.g., classify) storage racking, objects being stored therein, and/or walls, and to provide input to the vehicle management system <NUM> to control, for example, a steering angle of the MHV <NUM> as it travels along an aisle.

Various components of the guidance system <NUM> may be implemented on one or more processor units <NUM>. The processor unit <NUM> may be configured to send and/or receive information (e.g., including instructions, data, values, signals, or the like) to/from the various components of the guidance system <NUM> and/or external components such as the VMS <NUM>. The processor unit <NUM> may comprise, for example, a processor, DSP, CPU, APU, GPU, microcontroller, application-specific integrated circuit, programmable gate array, and the like, any other digital and/or analog components, as well as combinations of the foregoing (whether distributed, networked, locally connected, or the like), and may further comprise inputs and outputs for receiving and providing control instructions, control signals, drive signals, power signals, sensor signals (e.g., current or voltage sensor output, image sensor output, volumetric scanner output, and the like), digital signals, analog signals, and the like. All such computing devices and environments are intended to fall within the meaning of the terms "controller," "control unit," "processor," "processor unit," "processing device," or "processing circuitry" as used herein unless a different meaning is explicitly provided or otherwise clear from the context. In some examples, the processor unit <NUM> may comprise one or more such processor devices.

The processor unit <NUM> may comprise processing circuitry configured to execute operating routine(s) stored in a memory. The memory may include any suitable volatile memory, non-volatile memory, storage, any other suitable type of storage medium, or any suitable combination thereof. For example, the memory may include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, the memory may have encoded thereon a computer program for controlling operation of the processor unit <NUM>, VMS <NUM>, sensor unit <NUM>, or the like. In some embodiments, the various components of the guidance system <NUM> may be implemented entirely as software, entirely as hardware, or any suitable combination thereof. In some embodiments, the operating routine(s) may comprise firmware.

In some embodiments, the guidance system <NUM> may be communicatively coupled (e.g., via the processor unit <NUM>) with the VMS <NUM>, either directly or indirectly. In some embodiments, the VMS <NUM> may comprise the processor unit <NUM>, for example sharing a processing device to implement the guidance system <NUM> as well as perform other vehicle management functions. In some embodiments, the guidance system <NUM> may be coupled to the VMS <NUM> via an arbitrator module <NUM>. As will be described in more detail below, the arbitrator module may facilitate communication between multiple systems and the VMS <NUM>.

The sensor unit <NUM> can include one or more sensors <NUM> that can be configured to detect objects in the surrounding environment, in particular, storage racking, objects being stored in the storage racking, walls, and the like. In some embodiments, the sensor <NUM> may comprise a volumetric sensor <NUM>, for example a three-dimensional LiDAR sensor. In some embodiments, the sensor <NUM> may comprise a three-dimensional depth camera. In some embodiments, the sensor unit <NUM> may comprise a single sensor <NUM> configured to monitor multiple sides of the MHV <NUM> simultaneously (e.g., the area to the left and right of the MHV <NUM> are within a field of view of the sensor <NUM>). In some embodiments, the sensor unit <NUM> may include two or more sensors <NUM>, each of which may be configured to monitor a section of the environment surrounding the MHV <NUM>. For example, in some embodiments, the sensor unit <NUM> may comprise two sensors <NUM>, each configured to monitor an area to a particular side of the MHV <NUM> (e.g., opposing left and right sides of an aisle). The field of view of multiple sensors <NUM> may be combined to form a field of view of the guidance system <NUM>. In other non-limiting examples, other types of sensors as known in the art can also be used, for example, RADAR, image sensors (e.g., RGB, RGB-D, etc.), stereo cameras, time of flight sensors, ultrasonic sensors, or the like. In yet other non-limiting examples, two-dimensional sensors can alternatively or additionally be used.

The sensor unit <NUM> and/or sensors <NUM> may be configured to be installed on the MHV <NUM>. For example, the sensor unit <NUM> may be installed onto an upper portion of the vehicle frame <NUM>. In other non-limiting examples, the sensor unit <NUM> can be installed in different configurations. For example, as shown in <FIG>, the sensors <NUM> can be installed at an elevated location on the MHV <NUM>, such as an upper portion of the frame <NUM>, so as to have a field of view <NUM> that is not impeded or minimally impeded by the mast <NUM> or other portions of the MHV <NUM>. In this way, the sensor unit <NUM> can have a minimally inhibited or completely uninhibited field of view <NUM> around the MHV <NUM>, so as to allow the guidance system <NUM> to align the MHV <NUM> within an aisle in one or both of a frame-first direction, where the MHV <NUM> enters the aisle in a reverse direction so that the vehicle frame <NUM> enters the aisle before the fork assembly <NUM>, and a forks-first direction, where the MHV <NUM> enters the aisle in a forward direction so that the fork assembly <NUM> enters the aisle before the vehicle frame <NUM>. The sensor unit <NUM> can also be installed on an exterior of the MHV <NUM> and in some embodiments may include a cover. In some embodiments, the specific mounting configuration may vary depending on operational factors, including but not limited to the type of MHV and the conditions of the operating environment.

The guidance system <NUM> may be configured to determine the position of external objects within a field of view of the sensor <NUM> and provide guidance instructions to the MHV <NUM> according to a determined distance from such objects. In some embodiments, the sensor <NUM> can detect the relative position of storage racking, objects stored in the storage racking, walls, rails, and other objects that can define an aisle along which the MHV <NUM> can travel. Determining the position of external objects such as storage racking, walls, etc., may include classifying detected objects as a specific type of object, for example, classifying a detected object as storage racking or a crossbeam. The sensor <NUM> can be used for both aligning the MHV <NUM> with an aisle (i.e., prior to entering the aisle) and for keeping the MHV <NUM> centered or otherwise aligned within the aisle as it moves along the aisle.

Referring to <FIG>, in some embodiments the processor unit <NUM> can be configured as an on-board processor unit that can operate without external infrastructure. That is, the processor unit <NUM> can be installed on the MHV <NUM> and can be configured to locally process and otherwise analyze the sensor output from the sensor unit <NUM>. In some embodiments, the processor unit <NUM> can include various modules implemented as processes, for example implemented as firmware, subroutines, or the like. In some embodiments, the modules may comprise a communication application programming interface (API) <NUM>, a pre-processing module <NUM>, a post-processing module <NUM>, and a guidance module <NUM>, that can work together to analyze the sensor output, determine distance, and output instructions for navigating (i.e., guiding) the MHV <NUM>.

In some embodiments, the pre-processing module <NUM> may be configured to perform pre-processing functions, for example data manipulations on the raw sensor output (i.e., raw data from sensor <NUM>) or a point cloud directly before a lane centering method or other post-processing methods are applied. For example, pre-processing functions can include filtering data returns (e.g., median filter), coordinate transformations of the sensor data cloud points from Polar coordinates to Cartesian coordinates, or another type of coordinate system, and/or stitching data returns from multiple sensors into a single point cloud and/or frame of view.

In some embodiments, the post-processing module <NUM> may be configured to perform post-processing functions, for example software modules unrelated to in-aisle guidance and centering software. For example, in some embodiments the sensor data can be used for object avoidance in addition to being used for in-aisle guidance and centering functions. In some embodiments, the guidance module <NUM> may be configured to calculate a travel path of the MHV <NUM> and any associated instructions to be carried out by the VMS <NUM> to allow the MHV <NUM> to travel along said path.

With the aisle identified, for example the walls and/or storage racking that defines the sides of aisle (as described in more detail below), the guidance system <NUM> (e.g., the processor unit <NUM>, or more specifically the guidance module <NUM>) can then control the direction of travel of the MHV <NUM> to keep the MHV <NUM> centered between the respective sides of the aisle. In some embodiments, the guidance system <NUM> may communicate with the vehicle management system <NUM> (e.g., via a CAN bus) to control the steering angle of the traction wheel <NUM>. For example, the processor unit <NUM> may perform a heading angle calculation, where the processor unit <NUM> can detect a current heading angle of the MHV <NUM> with respect to the aisle to determine whether a steering angle adjustment is desired for the MHV <NUM> to become or stay aligned within the aisle. The heading angle calculation may account for various factors such as a steering angle, speed, and type of the MHV <NUM>.

If the guidance system <NUM> determines that a steering angle adjustment is desired, the processor unit <NUM> may calculate the desired steering angle adjustment. The guidance system <NUM> can then send a corresponding command to the VMS <NUM> to adjust the steering angle of the traction wheel <NUM> accordingly. In some embodiments, the heading angle calculation may result in a function of time, for example various steering angles for the MHV <NUM> to execute over a period of time, and the guidance system <NUM> may send a respective series of commands to the VMS <NUM>. In some non-limiting examples, the guidance system <NUM> may determine whether a change in speed is desired, for example, when the MHV <NUM> reaches the end of an aisle and can also command the VMS <NUM> to accelerate or decelerate the MHV <NUM> (e.g., by controlling a motor and/or a braking system).

In some non-limiting examples, the guidance system <NUM> may be configured to communicate directly with the vehicle management system <NUM>. Alternatively, the guidance system <NUM> may be configured to communicate indirectly with the vehicle management system <NUM>. In some embodiments, the commands from the guidance system <NUM> (e.g., from processor unit <NUM>) may first be received by, for example, an arbitrator module <NUM> that may also receive commands from other modules, such as an autonomous operation module <NUM> that comprises one or more semi-autonomous or fully autonomous modules. The arbitrator module <NUM> may determine which commands are to be carried out by the vehicle management system <NUM> to control the MHV <NUM>.

For example, if the autonomous operation module <NUM> sends a command to adjust the steering angle to a value, and the guidance system <NUM> sends a command to adjust the steering angle to the same value as the other module has set, the arbitrator module <NUM> can allow both commands to pass through to the VMS <NUM>. However, if both the autonomous operation module <NUM> and the guidance system send conflicting commands to adjust the steering angle to differing values, the arbitrator module <NUM> may only allow the command from either the guidance system <NUM> or the autonomous operation module <NUM> to pass through to the VMS <NUM>.

The guidance system <NUM> may be configured to maintain a minimum buffer distance between one or more sides of the aisle and the MHV <NUM>. The minimum buffer distance can be set by an operator or a warehouse management system to account for variability in operating environments. For example, some warehouses may store oversized items that can protrude beyond a front face of the storage racking and into the aisle. Additionally, if the aisle is a double-wide aisle that is configured to allow two-way travel, the guidance system <NUM> can be configured to maintain the buffer on only a single side of the aisle, for example by using a centering offset, thereby allowing two-way travel to be maintained. In some embodiments, the guidance system <NUM> can be configured to maintain the MHV <NUM> at the minimum buffer distance from the side of the aisle, for example along whichever side of the aisle the MHV <NUM> enters along.

Accordingly, if the guidance system <NUM> determines that the minimum buffer distance has not been maintained, the guidance system <NUM> can instruct the MHV <NUM> to re-establish the minimum buffer distance. In some embodiments, if the minimum buffer distance cannot be maintained, the guidance system <NUM> can relinquish control back to the operator, in turn, providing a corresponding indication (e.g., an audible or visual indication via the control interface <NUM>) to the operator that the guidance system is no longer operating. If the guidance system <NUM> determines that the MHV <NUM> is not aligned in the aisle (e.g., centered or maintaining a minimum buffer distance), the operator may regain full control of the MHV <NUM>.

Referring to <FIG>, the sensor <NUM> may generate sensor output comprising information relating to the sensed environment. The sensor output may comprise two-dimensional (2D) or three-dimensional (3D) information, for example, such as a point cloud indicating the position of various detected points in 3D-space relative to the sensor <NUM>. For example, <FIG> representatively illustrates an environment comprising the side of an aisle having racking and various objects stored therein, from the perspective of a sensor <NUM> mounted to a MHV <NUM>, and <FIG> illustrates a 3D point cloud of the detected environment as generated by a LiDAR sensor, color coded based on distance from the sensor. In many applications, the sensor will be oriented at an incident angle with respect to some of the environment.

In some embodiments, the guidance system <NUM>, for example using the processor unit <NUM>, may transform the sensor output from the coordinate system of the sensor <NUM> to a coordinate system of or related to the MHV <NUM> (which may be referred to herein as a "spatial transform"). For example, the sensor position (x, y, z) and orientation (pitch, roll, yaw) with respect to a predetermined reference point of the MHV <NUM> may be used to perform a three-dimensional transform of the sensor output. In some embodiments, the sensor output may be transformed to the reference frame of the MHV <NUM> (e.g., oriented with respect to a predetermined reference point and coordinate system of the MHV <NUM>). In some embodiments, the sensor output may be transformed with respect to a two-dimensional (2D) reference plane aligned parallel with a respective side of the MHV <NUM>. For example, <FIG> illustrates a 3D LiDAR point cloud transformed to a coordinate system of the MHV <NUM>, with the points color coded based on distance perpendicular to the MHV <NUM> (or, in some embodiments, based on distance into the racking).

In some embodiments, the guidance system <NUM> may organize the transformed sensor output into a grid along the Y-Z plane, and each grid may be evaluated to provide a depth measurement along the X-axis. The depth of each grid may be determined by factors such as a heatmap or density of measurement data and may be influenced by the resolution of a given sensor. For example, with additional reference to <FIG>, a field of view (or partial field of view) <NUM> of the sensor <NUM> transformed to a plane parallel to the side of the MHV <NUM> (i.e., a left or right side that generally faces the side of the aisle) is illustrated. The field of view <NUM> of the sensor <NUM> is of a non-limiting example of a storage rack <NUM> configured as a pallet rack, with a plurality of objects <NUM> being stored in the storage rack <NUM> on pallets. The transformed sensor output indicates the distance of each point with respect to the side of the MHV <NUM>, with respect to a plane passing through the predetermined MHV <NUM> reference point and parallel with the side of the MHV <NUM>, with respect to a plane passing through the inferred or calculated path of travel of the MHV <NUM>, or the like.

Referring to <FIG>, the distances of the objects (relative or absolute) in the transformed field of view <NUM> to the respective reference frame are representatively illustrated by circles overlaid on a spatial grid, with larger circles indicating objects that are closer to the respective reference frame (e.g., closer to the respective coordinate system origin or reference plane) and smaller circles indicating objects that are located further from the respective reference frame. The spatial grid and circles shown in <FIG> are shown for illustrative purposes and the distance data measured by the sensor <NUM> can be configured differently. The processor unit <NUM> may be configured to store and execute one or more algorithms for processing the transformed sensor output from the sensor <NUM>. In some embodiments, the guidance system <NUM> may be configured to perform occupancy grid analysis using the transformed sensor output.

In general, the guidance system <NUM> may be configured (e.g., via one or more operating routines stored in the processor unit <NUM> memory) to carry out one or more methods (e.g., pattern recognition algorithms, machine learning models, or the like) on the transformed sensor output to differentiate between and/or identify crossbeams of the storage racking, uprights of the storage racking, the objects being stored in the storage racking, walls, and the like. In some embodiments, the guidance system <NUM> may be configured to detect and identify storage racking (e.g., storage rack <NUM>) using distance changes along a height of the field of view <NUM> (i.e., a Z-axis oriented in a vertical direction with respect to <FIG>) and/or changes along a width of the field of view <NUM> (i.e., a Y-axis oriented in a horizontal direction with respect to <FIG>), to identify which objects in the field of view <NUM> are likely to be storage racking or a wall. It will be recognized that any suitable pattern matching or other algorithms may be used to identify various aisle features according to a sensor type, for example object classification using RGB gradients, reflectivity values from LiDAR, machine learning models for various types of sensor data, and the like.

Referring to <FIG>, in some embodiments, the guidance system <NUM> may be configured to identify crossbeams in the transformed sensor output using pattern recognition. For example, the guidance system <NUM> may be configured to identify a pattern along the Y-axis that indicates a horizontal crossbeam. In some embodiments, the pattern recognition may comprise determining a depth gradient approaching zero (e.g., consistent depth in the X-axis direction) along the Y-axis at a certain Z-height (e.g., the Z-coordinate of the respective grids along the Y-axis). If pattern recognition identifies one or more crossbeams, then the guidance system <NUM> may then determine the respective Z-height(s) to be region(s) of interest <NUM>. In some embodiments, the guidance system <NUM> may be similarly configured to identify vertical crossbeams. It will be recognized that other pattern recognition may be used.

Referring to <FIG>, in some embodiments, the guidance system <NUM> may be configured to determine whether there are a predefined number and/or density of data points disposed along a certain length of the field of view <NUM> (e.g., along a Y-axis that is oriented in a horizontal direction with respect to <FIG>). As illustrated in <FIG>, a lower plurality of data points <NUM> has at least a predefined number and/or density of individual points extending along a predefined width of the field of view <NUM> so that the guidance system <NUM> can classify the lower plurality of data points as belonging to a crossbeam. Conversely, an upper plurality of data points <NUM>, for example representing an object <NUM> on the storage rack <NUM>, has fewer than the predefined number of individual points extending along the width of the field of view <NUM>. The guidance system <NUM> can classify the upper plurality of data points <NUM> as not belonging to a crossbeam. It is appreciated that the number of predefined data points in the above determinations can vary depending on the specific application. For example, a guidance system <NUM> with sensor unit <NUM> having a comparatively low resolution may utilize a smaller number of data points than a guidance system <NUM> with sensor unit <NUM> having a comparatively high resolution.

In some embodiments, the guidance system <NUM> may be configured to differentiate between storage racking and walls by identifying data points that are likely to be associated with crossbeams (e.g., cross-members) for storage racking by determining whether there are a predefined number and/or density of data points disposed within a certain height of one another, a predetermined depth gradient, or the like. For example, walls generally have heights that are larger than a corresponding height of a crossbeam. Accordingly, if the pattern recognition determines that a plurality of points is generally in-line with one another and extend along a predefined height of the field of view <NUM>, the points are likely to belong to a wall and not a cross-beam. Where a wall is found, the guidance system <NUM> may select a portion of the wall to be treated as a crossbeam of a storage rack. For example, in some embodiments, the guidance system <NUM> may determine that a wall is present to a side of the MHV <NUM> by determining that depth gradients approach zero at a large number of Z-heights, in which case the guidance system <NUM> may pick one or more such Z-heights as representing one or more proxy crossbeams. For example, the guidance system <NUM> may select one or more Z-heights where the wall would be expected to be continuous along its length, such as where windows, doors, or the like are not expected to exist. This may be parameterized and set on a case-by-case basis.

In some embodiments, it may be desirable to repeatedly determine the distance of the MHV <NUM> from one or more sides of the aisle as the MHV <NUM> moves along the aisle. For example, in some embodiments, the guidance system <NUM> may determine the distance of a reference frame (e.g., a plane passing through the predetermined MHV <NUM> reference point and parallel with a respective side of the MHV <NUM>) from the one or more determined objects (e.g., a crossbeam) as the MHV <NUM> moves along the aisle. In some embodiments, the guidance system <NUM> may determine the distance from the one or more determined objects multiple times a second, for example from about <NUM> times a second to about <NUM> times a second, <NUM> times a second, <NUM> times a second, and the like. The frequency of distance determination may depend on performance requirements of the system, such as speed of the MHV <NUM>, width of the aisle, sensor <NUM> output capabilities, likelihood of obstacles, and the like.

Each new distance determination may require a new sensor output transform, as described above. In some embodiments, the sensor output may comprise a 3D point cloud (e.g., from a LiDAR sensor) having hundreds of thousands of points for each LiDAR scan. Transforming and/or performing feature identification (e.g., pattern matching) using all of the sensor data for each scan, such as all the points in the point cloud, may be compute intensive and may result in a lower rate of coordinate transform and/or feature identification and thus a lower rate of distance determination and guidance updates.

Referring to <FIG>, a non-limiting of example of a method <NUM> for initiating a guidance system (also referred to as "aisle induction") is illustrated. The aisle induction method <NUM> may be implemented as one or more operating routines stored in the memory of the processor unit <NUM>. At step <NUM>, an operator can control the MHV <NUM> to approach an aisle, for example by aligning the MHV <NUM> generally along the direction of travel along the aisle. In some embodiments, to facilitate the guidance system <NUM> identifying an aisle and/or aisle feature, the operator may maneuver the MHV <NUM> to be within a certain proximity to a side of an aisle and/or to be at an angle with respect to the aisle that is less than a predetermined angle limit (e.g., to be substantially aligned with the direction of the aisle, within some allowed angle range).

At step <NUM>, the guidance system <NUM> can be activated (i.e., turned on) by an operator via the control interface <NUM>, for example, by touching a virtual button on the display <NUM>. In other non-limiting examples, guidance system <NUM> can be activated <NUM> in different ways, such as the guidance system <NUM> being configured to automatically detect an aisle nearby. Until the guidance system <NUM> is activated, the operator may have full control over both the speed and the direction of travel (i.e., steering) of the MHV <NUM>. In some embodiments, once the guidance system <NUM> has been activated at step <NUM>, the speed of the MHV <NUM> can be limited by the guidance system <NUM> at step <NUM> to provide additional time for the identification of the aisle (e.g., crossbeam positions) by the guidance system <NUM>. In some embodiments, the steering behavior may be limited at step <NUM>.

The guidance system <NUM> may acquire sensor output from the sensor unit <NUM> at step <NUM>, and transform the sensor output at step <NUM>, as described above. In some embodiments, the guidance system <NUM> may transform all sensor output, substantially all of the sensor output, a subset of the sensor output (e.g., excluding a predetermined peripheral area of the sensor), or the like. Sensor output from a plurality of sensors <NUM> may be transformed to a uniform reference coordinate system (e.g., an MHV reference frame), individual reference frames for each sensor <NUM> (e.g., reference frames oriented on the left side and right side of the MHV <NUM>), or any other suitable reference frame(s) from which the guidance system <NUM> can determine the absolute or relative location of one or more aisle features with respect to the MHV <NUM>. As used herein, an aisle feature may comprise a crossbeam, proxy crossbeam, wall, railing, or other regular feature that may define an aisle, and from which the relative and/or absolute position of the MHV <NUM> within the aisle may be determined.

At step <NUM>, the guidance system <NUM> may perform pattern recognition on the transformed sensor output, for example as described above, to identify one or more aisle features. In some embodiments, the guidance system <NUM> may identify and discard sensor output representing the floor. If an aisle feature is identified, then the guidance system <NUM> may, at step <NUM>, create a virtual (i.e., software-based) model of the aisle (herein referred to as an "aisle model"). In some embodiments, the aisle model may be stored in the memory of the processor unit <NUM>. The aisle model may comprise information corresponding to the aisle feature. The modeled aisle feature may be referred to herein as a "projected reference. " The aisle model may comprise information such as the type of aisle feature (horizontal crossbeam, vertical crossbeam, proxy crossbeam, etc.) being modeled, the number of projected references, the location along the Y-axis or Z-axis of each projected reference (e.g., height of a projected reference representing a horizontal crossbeam), start position of each projected reference, slope of each projected reference, figure of merit of each projected reference and/or the model, confidence interval of each projected reference and/or the model, or the like.

Because aisle features may be somewhat regular or continuous, the projected reference may be configured to project (predict) the position of the features further into the aisle beyond the field of view <NUM> of the sensor unit <NUM>. In some embodiments, the projected reference information may be stored with respect to the transformed reference frame (e.g., the reference frame of the MHV <NUM>), may be stored with respect to the reference frame of the sensor, or both.

In some embodiments, the guidance system <NUM> may determine whether the aisle (e.g., using the model or actual measurements from the sensor output) is too wide or too narrow. For example, in some non-limiting examples, use of the guidance system <NUM> can be limited to specific operating scenarios, such as in aisles having a width that is less than a maximum permitted width or greater than a minimum permitted width. The maximum and/or minimum width of an aisle in which the guidance system <NUM> is permitted to operate can be set by an operator or warehouse management system and may correspond with, for example, a width of a single-wide aisle, a double-wide aisle, width of the MHV <NUM>, or the like.

In some embodiments, if an aisle feature is identified and an aisle model is successfully created, the guidance system <NUM> may, at step <NUM>, remove the speed limit of step <NUM> or otherwise relinquish control of the speed of the MHV <NUM> to the operator, AGV (e.g., via an autonomous operation module <NUM>), warehouse management system, or the like. As the operator continues moving the MHV <NUM> toward or into the aisle, the guidance system <NUM> can then navigate the MHV <NUM> within the aisle, for example centering the MHV <NUM> approximately equidistant from each of the respective sides of the aisle, aligning the MHV <NUM> offset from the center of the aisle by a predefined distance, aligning the MHV <NUM> at a minimum, or an offset from the minimum, distance (e.g., a buffer distance) from a side of the aisle, or the like. In some embodiments, the operator may control the speed of the MHV <NUM> while the guidance system <NUM> aligns the MHV <NUM> within the aisle.

For example, referring to <FIG>, a non-limiting of example of a method <NUM> for aisle navigation is illustrated. The aisle navigation method <NUM> may be implemented as one or more operating routines stored in the memory of the processor unit <NUM>. At step <NUM>, the guidance system <NUM> may determine the distance of the MHV <NUM> from one or more aisle features using the aisle model. The guidance system <NUM> may determine the distance (relative or absolute) from the aisle feature to MHV <NUM> based on the information stored in the aisle model, for example by determining the horizontal distance (e.g., perpendicular to the MHV <NUM>) of the projected reference to the relevant reference frame of the MHV <NUM>. In some embodiments, the guidance system <NUM> may determine the distance of the left and/or right side of the MHV <NUM> from aisle features on the left and/or right side of the MHV <NUM>, using respective projected reference information from the aisle model. In some embodiments, the guidance system <NUM> may determine the distance of the center of the MHV <NUM> from aisle features on the left and/or right side of the MHV <NUM>, using respective projected reference information from the aisle model. In some embodiments, the guidance system <NUM> may determine the heading angle of the MHV <NUM> with respect to the one or more projected references or may receive heading information from the VMS <NUM>.

At step <NUM>, the guidance system <NUM> may perform path planning to determine a guidance instruction. The guidance instruction may comprise one or more instructions to guide the MHV <NUM> onto the desired travel path (e.g., desired position with respect to the aisle features) through the aisle. A guidance instruction may comprise any suitable command that is usable by the MHV <NUM> to control its location in the environment. In some embodiments, the guidance system <NUM> may perform path planning based on the determined distance(s) and/or angle(s) from step <NUM>, in combination with the desired position of the MHV <NUM> with respect to the projected references (centered, offset, minimum distance from edge, etc.). In some embodiments, a guidance instruction may comprise a steering angle, steering direction, speed, target coordinates, amount to shift left or right, or the like. In some embodiments, the guidance instruction may comprise a steering angle limit to limit how much steering may be applied, for example if the MHV <NUM> is on or almost on the desired travel path. In some embodiments, the guidance system <NUM> may send the guidance instruction to the MHV <NUM>, for example to the VMS <NUM>.

For example, with additional reference to <FIG>, the MHV <NUM> is shown moving along an aisle <NUM> having a first side <NUM> (e.g., right side) opposite a second side <NUM> (e.g., left side). As illustrated, each of the first side <NUM> and the second side <NUM> of the aisle are formed from storage racking. The first side <NUM> is formed from a first actual crossbeam <NUM> and the second side <NUM> is formed from a second actual crossbeam <NUM>. Accordingly, from the view of the aisle model, the first side <NUM> is defined by a first projected reference <NUM> (i.e., projected crossbeam) and the second side <NUM> is defined by a second projected reference <NUM> (i.e., a projected crossbeam). Not illustrated is that the projected references <NUM>, <NUM> extend over the indicated actual crossbeams <NUM>, <NUM>. In some embodiments, the guidance system <NUM> may use the positions of the projected references <NUM>, <NUM> of the aisle <NUM> to determine (step <NUM>) a centerline <NUM> of the aisle <NUM> (e.g. a desired travel path) that is disposed between the projected references <NUM>, <NUM>. Likewise, the guidance system <NUM> can determine a current path of travel <NUM> or static location of the MHV <NUM> (step <NUM>), for example based on the current angle of the MHV <NUM> with respect to the projected references <NUM>, <NUM> and the position of the MHV <NUM> along the X-axis between the projected references <NUM>, <NUM>.

In the illustrated example of <FIG>, the path of travel <NUM> of the MHV <NUM> is not aligned (i.e., coincident) with the centerline <NUM> of the aisle <NUM>, the MHV <NUM> being four inches from the right projected reference <NUM> and seven inches from the left projected reference <NUM>. The guidance system <NUM> can calculate a guidance instruction to cause the path of travel <NUM> of the MHV <NUM> to become aligned with the centerline <NUM> of the aisle <NUM> as the MHV <NUM> continues to move along the aisle <NUM>. In another non-limiting example, when the aisle <NUM> is wide enough to allow for two-way traffic, the guidance system may also be configured to perform a centering offset calculation. The centering offset calculation allows the desired path travel path (e.g. centerline <NUM>) to be offset closer to one side of the aisle <NUM> as compared to the other side of the aisle <NUM>. In this way, the guidance system <NUM> can allow for two-way traffic. While the centerline <NUM> of the aisle <NUM> is shown being a straight line, this may not always be the case. For example, the aisle <NUM> may be curved, and as described in more detail below, the guidance system may continually update the aisle model, including projected reference information, such that the MHV <NUM> may be appropriately guided through the aisle.

In some embodiments, the guidance system may continually or otherwise repeatedly acquire new sensor output. Referring again to <FIG>, the guidance system <NUM> may acquire new sensor output at step <NUM>, similar to or the same as in step <NUM> of the aisle induction method <NUM>. It will be understood that the sensor output may be acquired <NUM> in any suitable order or timing with respect to the other steps of the navigation method <NUM>.

Referring back to <FIG>, in a first set of embodiments, at step <NUM>, the guidance system <NUM> may transform a limited subset of the sensor output acquired in step <NUM> to the appropriate reference frame (e.g., a MHV reference frame), wherein the subset is based on the projected reference information in the aisle model. Following that, at step <NUM>, the guidance system <NUM> may perform pattern recognition on the transformed subset of sensor output, for example pattern recognition similar to step <NUM> of the aisle induction method <NUM>, to identify one or more aisle features. In some embodiments, the subset of sensor output transformed <NUM> corresponds to the expected location(s) of one or more aisle features as identified by the projected references.

In a second set of embodiments, at step <NUM>, the guidance system <NUM> may transform the sensor output acquired in step <NUM> to the appropriate reference frame (e.g., a MHV reference frame), similar to or the same as step <NUM> of the aisle induction method <NUM>. The sensor output transformed may be irrespective of the expected location(s) of one or more aisle features. Following that, at step <NUM>, the guidance system <NUM> may perform pattern recognition on a subset of the transformed sensor output to identify one or more aisle features, wherein the subset is based on the projected reference information in the aisle model. In some embodiments, the subset of transformed sensor output corresponds to the expected location(s) of one or more aisle features as identified by the projected reference.

In some cases, the subset of sensor output transformed in the first set of embodiments, or the subset of transformed sensor output from which aisle features are identified in the second set of embodiments, may also include sensor output (or transformed sensor output, respectively) within a tolerance of the expected location(s) of the aisle features as identified by the projected references. For example, the subset operated upon may also include a predetermined height above and below the expected location of a horizontal crossbeam. The tolerance may provide for detecting unexpected changes, such as a change in the position of the respective aisle feature and may contribute to the confidence of the model by confirming that no aisle features exist outside of the expected regions. Advantageously, processing requirements are reduced compared to transforming and/or performing feature identification on the entire sensor output, allowing for faster updates and more responsive guidance.

In some embodiments, based on the results of pattern recognition <NUM>, the guidance system <NUM> may update the aisle model at step <NUM>. In some embodiments, the guidance system <NUM> may update the projected reference information based on the results of pattern recognition <NUM>. The update of the projected reference information may be done through one or more filtering methods, such as moving average or median, weighted average or median, Kalman filter, or the like. It will be understood that acquiring sensor output <NUM>, transforming sensor output <NUM>, identifying an aisle feature <NUM>, and updating the model <NUM> may be done at a different frequency from determining distance from an aisle feature <NUM> and determining a guidance instruction <NUM>.

In some embodiments, the guidance system <NUM> may, at step <NUM>, determine whether to continue providing guidance to the MHV <NUM>. If the guidance system <NUM> determines to continue guidance, it may proceed to step <NUM> to determine a new distance from an aisle feature and continue to provide guidance accordingly. If the guidance system <NUM> determines to discontinue providing guidance, the guidance system <NUM> may stop the aisle navigation method <NUM> (i.e., deactivate guidance at step <NUM>). In some embodiments, the guidance system <NUM> may, at step <NUM>, be configured to first perform a portion of the aisle induction method <NUM> (e.g., steps <NUM> to <NUM>) to attempt to reidentify the aisle feature(s). If the aisle induction method <NUM> is successful then the guidance system <NUM> may continue with the aisle navigation method <NUM>, otherwise it may continue to deactivate guidance <NUM>.

For example, at step <NUM> the guidance system <NUM> may be configured to determine whether the MHV <NUM> is nearing or is at the end of the aisle using the acquired sensor output from step <NUM>, the transformed sensor output from step <NUM>, or the result of the pattern recognition at step <NUM>. If the guidance system <NUM> determines that the MHV <NUM> is not near an end of the aisle, the guidance system <NUM> can proceed back to step <NUM> as described above. If the guidance system <NUM> determines that the MHV <NUM> is nearing or is at the end of the aisle, the guidance system <NUM> may deactivate guidance <NUM>. In some embodiments, the deactivating guidance <NUM> may comprise initiating an end of aisle procedure, for example slowing the MHV <NUM> and/or giving full control of the MHV <NUM> back to the operator. Additionally, the guidance system <NUM> may turn off or go into a standby mode until reactivated (e.g., at step <NUM> of the aisle induction method <NUM>).

In another exemplary embodiment, the guidance system <NUM> may be configured to determine, at step <NUM> and based on the result of pattern recognition <NUM> or model update <NUM>, that the model no longer accurately represents the aisle features and may be configured to accordingly deactivate guidance at step <NUM>. For example, the guidance system <NUM> may determine that a confidence interval, figure of merit, or other measure of model reliability has fallen outside of an acceptable range, which may for example happen if there is an abrupt change in a number of aisle features.

Thus, according to the systems and methods described above, the guidance system <NUM> may first initialize (<NUM>) based on measured values of aisle features, but as the MHV <NUM> travels along aisle the guidance system <NUM> guides (<NUM>) the MHV <NUM> based on expected (predicted) locations of aisle references (i.e., the projected references in the aisle model). Advantageously, the guidance system <NUM> operating on the aisle model can prevent errant measurements, protrusions, or the like from throwing the MHV <NUM> off course or otherwise upsetting navigation. In some cases, protrusions into the aisle may exist (e.g., a pallet sticking out from racking). In some embodiments, the protrusion may be handled by the autonomous operation module <NUM> which may turn off guidance or otherwise take control of MHV <NUM> in appropriate circumstances.

Further, even though aisle features such as crossbeams may not run an entire length of an aisle, the guidance system <NUM> can continue to operate on the expected (predicted) locations of aisle features by way of the projected references in the aisle model and may continue to successfully navigate the aisle as the aisle model is updated with new or updated projected references. Accordingly, operating on projected references "smooths out" responses to unexpected or unideal situations such as an object overlaying (hiding) racking, a protruding object, gaps in the aisle feature, and the like.

In some embodiments, the guidance system <NUM> may temporarily store the aisle model as it navigates an aisle but may not permanently store the aisle model. In other words, the guidance system <NUM> may create a new aisle model (<NUM>) each time it begins traversing a given aisle, even if it has traversed the aisle previously. The guidance system <NUM> may accordingly be flexible to handle changes in an aisle without prior training or mapping of the aisle.

While various spatial and directional terms, such as top, bottom, lower, mid, lateral, horizontal, vertical, front, and the like may be used to describe examples of the present disclosure, it is understood that such terms are merely used with respect to the orientations shown in the drawings. The orientations may be inverted, rotated, or otherwise changed, such that an upper portion is a lower portion, and vice versa, horizontal becomes vertical, and the like.

Claim 1:
A material handling vehicle (<NUM>), comprising:
a vehicle management system (<NUM>) configured to control a movement of the material handling vehicle (<NUM>);
a sensor (<NUM>) configured to:
measure a plurality of distances, wherein each distance is a distance to one of a plurality of external objects; and
output a distance information corresponding to the plurality of measured distances; and
a processor unit (<NUM>) in communication with the sensor (<NUM>) and the vehicle management system (<NUM>), the processor unit (<NUM>) being configured to:
receive the distance information from the sensor (<NUM>);
transform the distance information from a sensor (<NUM>) coordinate system to a material handling vehicle coordinate system;
identify an aisle feature based on the transformed distance information;
create an aisle model based on the identified aisle feature;
determine a guidance instruction based on the aisle model; and
provide the guidance instruction to the vehicle management system (<NUM>),
the material handling vehicle (<NUM>) being characterised in that said processor unit (<NUM>) is further configured to:
receive a second distance information from the sensor (<NUM>) corresponding to a second plurality of measured distances to the plurality of external objects;
transform a subset of the second distance information from the sensor coordinate system to the material handling vehicle coordinate system, wherein the subset of the second distance information corresponds to an expected location of the identified aisle feature;
reidentify the aisle feature based on the transformed second distance information;
update the aisle model based on the reidentified aisle feature;
determine a second guidance instruction based on the updated aisle model; and
provide the second guidance instruction to the vehicle management system (<NUM>)..