Patent Description:
Trailers are usually unpowered vehicles, tools, or equipment that are pulled by a powered tow vehicle. When referring to farming equipment the powered tow vehicle is typically referred to as a tractor (rear or front loading) or any other vehicle configured to attach to the trailer and pull the trailer. The trailer, may be any type of farm implement, such as a bailer, a trailer bed, a plow, a seeder, a harvester, a baler, etc. In the instance of front loading tractors, the trailer (i.e. the vehicle is unpowered) may actually be located at the front of the tow vehicle (i.e. the vehicle providing power).

The trailer may be attached to a tractor using a trailer hitch. A receiver hitch mounts on the tractor and connects to the trailer hitch to form a connection. When dealing with farm equipment the connection point, i.e. hitch, between the tractor and the trailer may vary depending on the type of implement of the trailer.

The trailer hitch may be a ball and socket, a fifth wheel and gooseneck, trailer jack, eye and ring, pintle hitch, drawbar, or three-point hitch. Other attachment mechanisms may also be used. In addition to the mechanical connection between the trailer and the tractor, in some examples, the trailer is electrically connected to the tractor. As such, the electrical connection allows the trailer to take the feed from the powered tractor's rear light circuit, allowing the trailer to have lights that are in sync with the powered tractor's lights.

Some of the challenges that face tractor operators are connecting the tractor to the trailer, because more than one person is needed. For example, one person drives the tractor, e.g., the operator, and another one or more people are needed to view the tractor and the trailer and provide the operator with direction regarding the path the tractor has to take to align with the hitch. If the people providing directions to the operator are not accustomed to hitching a tractor to a trailer, then they may have difficulty providing efficient instructions for directing the path of the tractor.

Recent advancements in sensor technology have led to improved safety systems for vehicles. Arrangements and methods for detecting and avoiding collisions are becoming available. Such operator assistance systems use sensors located on the vehicle to detect an ongoing collision. In some examples, the system may warn the operator of one or more driving situations to prevent or minimize collisions. Additionally, sensors and cameras may also be used to alert a operator of possible obstacles when the vehicle is traveling in a forward direction. Therefore, it is desirable to provide a system that includes sensors to overcome the challenges faced by operators of tractors.

<CIT> describes a method of maneuvering a tow vehicle in reverse for attachment to a trailer. The method includes receiving one or more images from one or more cameras positioned on a back portion of the tow vehicle and identifying one or more trailers within the one or more images. An image with the one or more trailer representations is displayed on a user interface, such as a touch screen display, included in the vehicle, wherein the driver can select one of the trailer representations by pointing the driver's finger on the desired trailer representation or via a rotary knob or a mouse indicating that the driver wants the tow vehicle to autonomously drive and connect to the trailer associated with the selected trailer representation. Then, a path planning system of the tow vehicle determines a path between the tow vehicle and the trailer based on the location of the selected trailer relative to a position of the tow vehicle. The vehicle path includes maneuvers configured to direct the vehicle in a rearward direction along the tow vehicle path from the initial position to the final position, wherein the path is determined such that the tow vehicle, in an intermediate position within a predetermined distance from the trailer, is in an orientation aligned generally parallel with the trailer and facing away from the trailer, where a hitch of the tow vehicle is substantially aligned with a hitch of the trailer. The method also includes executing one or more behaviors causing the vehicle to take an action to autonomously follow the determined vehicle path and execute the maneuvers. Furthermore, as the tow vehicle is moving about the path, the path can be adjusted by the path planning system based on one or more objects that may be identified along the path by a sensor system of the tow vehicle in order to avoid a collision.

<CIT> describes a trailer hitching assist system for a vehicle which includes a camera disposed at a rear portion of a vehicle and having a field of view rearward of the vehicle. A control includes an image processor operable to process image data captured by the camera. The image processor, via image processing of image data captured by the camera, detects a trailer and trailer hitch rearward of the vehicle and determines, based on estimating the physical position of the trailer relative to the vehicle, a first path of travel for the vehicle to follow so as to maneuver the vehicle so as to have its tow ball aligned with the trailer hitch. The control maneuvers the vehicle along the determined first path of travel. Responsive to detection of an object entering the first path of travel, the control determines a second path of travel and maneuvers the vehicle along the second path of travel to avoid the detected object entering the determined path of travel.

<CIT> describes a hitch assist system for a vehicle which comprises a camera having a field of view rearward of the vehicle, an input device connected for the hitch assist system, and a controller. The controller includes instructions for detecting a trailer proximate to the vehicle with the camera, determining a vehicle hitch ball location, and determining a trailer hitch location. If there are multiple trailers detected by the controller, the input device provides an option to select which trailer is intended for hitching by outlining the trailer on a screen for the input device and allowing the vehicle driver to select the appropriate trailer by touching the intended trailer or moving a cursor over the intended trailer and selecting. The controller further includes instructions for calculating a vehicle path from an initial position to a final position, wherein the vehicle hitch ball is laterally aligned with the trailer hitch in the final position, and wherein the path is calculated to avoid a collision with objects detected by surrounding sensors of the vehicle. The calculated path is shown as a plotted path virtually on the input device allowing the vehicle driver to confirm prior to the vehicle moving. The controller also includes instructions for calculating the steering and braking maneuvers necessary to move the vehicle along the path to the final position and sending instructions to a vehicle steering system and a vehicle brake system to perform the calculated maneuvers.

<CIT> describes a method of maneuvering a vehicle in reverse for attachment to a trailer. The method includes receiving a trailer hitch receiver image location shown within one or more images from at least one vehicle camera. The method also includes determining, a pixel angular difference in the image between a tow vehicle fore-aft axis and a trailer fore-aft axis, determining a pixel distance between a tow vehicle hitch ball and a hitch receiver, and determining a vehicle path from an initial position to a final position adjacent the trailer. The vehicle path includes maneuvers configured to move the vehicle along the vehicle path from the initial position to the final position. Then autonomously following, at a drive system in communication with the computing device, the vehicle path from the initial position to the final position. Furthermore, if an object is detected along the path, the path is recalculated based on data relating to the position of the object to avoid a collision.

The invention relates to a method according to claim <NUM> and to a system according to claim <NUM>.

One general aspect includes a method of maneuvering a tractor in reverse for attachment to a trailer using a hitch assist system, the method including: entering a hitch assist mode. The method also includes displaying a camera image on a user interface, where the displayed image shows at least one camera. The method also includes receiving, at a user interface in communication an indication of a selected trailer. The method also includes determining a planned path with a computing device a tractor path from an initial position to a final position adjacent the trailer, the tractor path including maneuvers configured to move the tractor along the tractor path from the initial position to the final position. The method also includes autonomously following, at a drive system in communication with the computing device, the tractor path from the initial position to the final position.

Implementations may include one or more of the following features. The method further including: continuously detecting, at the neural network, one or more objects within the tractor path as the tractor is moving along the tractor path. The method may also include when detecting an object, altering the tractor path at the computing device. The method where detecting one or more trailers includes: capturing, at one or more imaging devices in communication with the neural network, one or more images, at least one of the one or more imaging devices positioned on a back side of the trailer facing a rearward direction. The method may also include determining, at the neural network, the one or more trailers within the one or more images. The method where displaying on the user interface further includes receiving, at a controller, one or more images from one or more cameras positioned on a back portion of the tractor and in communication with the controller, and overlaying, at the controller, a path representation indicative of an expected path the tractor drives along, the expected path starting at a tractor hitch. The method where selecting the trailer further includes receiving, at the controller, a first command by way of a user interface, the first command indicative of a change in the path representation such that the path representation ends at a point of interest, and adjusting, at the controller, the path representation based on the first command and where the path planning is completed by a controller for the hitch assist system. The method further including: stopping or halting, at the drive system, the tractor at an intermediate position before reaching the final position, the intermediate position being closer to the final position than the initial position. The method may also include modifying, at the drive system, one or more tractor suspensions associated with the tractor to align a tractor hitch with a trailer hitch. The method may also include autonomously following, at the drive system, the tractor path from the intermediate position to the final position. The method may also include connecting, at the drive system, the tractor hitch with the trailer hitch. The method where connecting a tractor hitch with a trailer hitch includes modifying one or more tractor suspensions associated with the tractor to align a tractor hitch with a trailer hitch.

One aspect of the disclosure provides a method of autonomously driving a vehicle in a rearward direction towards a point of interest. The method includes receiving, at data processing hardware, one or more images from a camera positioned on a back portion of the vehicle and in communication with the data processing hardware. The method also includes receiving, at the data processing hardware, a operator planned path from a user interface in communication with the data processing hardware. The operator planned path includes a plurality of waypoints. The method includes transmitting, from the data processing hardware to a drive system in communication with the data processing hardware, one or more commands causing the vehicle to autonomously maneuver along the operator planned path. The method includes determining, at the data processing hardware, a current vehicle position. In addition, the method includes determining, at the data processing hardware, an estimated subsequent vehicle position based on the operator planned path. The estimated subsequent vehicle position being at a subsequent waypoint along the operator planned path from the current vehicle position. The method also includes determining, at the data processing hardware, a path adjustment from the current vehicle position to the estimated subsequent vehicle position. Additionally, the method includes transmitting, from the data processing hardware to the drive system, instructions causing the vehicle to autonomously maneuver towards the estimated subsequent vehicle position based on the path adjustment.

Another aspect of the disclosure provides a system for autonomously maneuvering a vehicle in a rearward direction towards a point of interest. The system includes: data processing hardware; and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations that include the methods described above.

Implementations of the aspects of the disclosure may include one or more of the following optional features. In some implementations, the method includes overlaying a path on the one or more images and receiving a command by way of the user interface in communication with the data processing hardware. The command including instructions to adjust the path as the operator planned path. The command may include instructions to adjust a distance of the path. In some examples, the command includes instructions to adjust an angle of the path. The command may include instructions to adjust an angle of an end portion of the path.

In some examples, determining the current vehicle position includes receiving wheel encoder sensor data associated with one or more wheels and receiving steering angle sensor data. The current vehicle position is based on the wheel encoder sensor data and the steering angle sensor data.

A tractor, such as, but not limited to a rear load tractor or a front load tract, hereinafter referred to as a tractor may be configured to tow a trailer. The tractor may be other types of vehicles and farm equipment that are powered and configured to pull other farm implements. The trailer, may be any type of farm implement, such as a bailer, a trailer bed, a plow, a seeder, a harvester, a baler, etc. In the instance of front loading tractors, the trailer (i.e. the vehicle is unpowered) may actually be located at the front of the tractor (i.e. the vehicle providing power). Hereinafter all such equipment will be referred to as the trailer.

The tractor connects to the trailer by way of a trailer hitch. It is desirable to have a tractor that is capable to autonomously maneuvering towards a trailer and attaching to the trailer, thus eliminating the need for a operator to drive the tractor (e.g. in a rearward direction) while another one or more people provide the operator with directions regarding the path that the has to take to align with the trailer and ultimately a hitch of the trailer. As such, a tractor with an autonomous driving and hitching feature provides a operator with a safer and faster experience when hitching the tractor to the trailer.

Referring to <FIG>-2B, in some implementations, a operator of a tractor <NUM> wants to tow a trailer <NUM>, 200a-c, for example, a specific trailer 200a-c from a group of trailers <NUM>, 200a-c. The tractor <NUM> may be configured to receive an indication of a operator selection <NUM> associated with a selected trailer <NUM>, 200a-c and autonomously drive towards the selected trailer <NUM>, 200a-c. The tractor <NUM> may include a drive system <NUM> that maneuvers the tractor <NUM> across a surface based on drive commands having x, y, and z components, for example. As shown, the drive system <NUM> includes a front right wheel <NUM>, 112a, a front left wheel <NUM>, 112b, a rear right wheel <NUM>, 112c, and a rear left wheel <NUM>, 112d. The drive system <NUM> may include other wheel configurations as well. The drive system <NUM> may also include a brake system <NUM> that includes brakes associated with each wheel <NUM>, 112a-d, and an acceleration system <NUM> that is configured to adjust a speed and direction of the tractor <NUM>. In addition, the drive system <NUM> may include a suspension system <NUM> that includes tires associates with each wheel <NUM>, 112a-d, tire air, springs, shock absorbers, and linkages that connect the tractor <NUM> to its wheels <NUM>, 112a-d and allows relative motion between the tractor <NUM> and the wheels <NUM>, 112a-d. The suspension system <NUM> improves the handling of the tractor <NUM> and provides a better ride quality by isolating noise, bumps, and vibrations. In addition, the suspension system <NUM> is configured to adjust a height of the tractor <NUM> allowing the tractor hitch <NUM> to align with the trailer hitch <NUM>, which allows for autonomous connection between the tractor <NUM> and the trailer <NUM>.

The tractor <NUM> may move across the surface by various combinations of movements relative to three mutually perpendicular axes defined by the tractor <NUM>: a transverse axis X, a fore-aft axis Y, and a central vertical axis Z. The transverse axis x, extends between a right side R and a left side of the tractor <NUM>. A forward drive direction along the fore-aft axis Y is designated as F, also referred to as a forward motion. In addition, an aft or rearward drive direction along the fore-aft direction Y is designated as R, also referred to as rearward motion. When the suspension system <NUM> adjusts the suspension of the tractor <NUM>, the tractor <NUM> may tilt about the X axis and or Y axis, or move along the central vertical axis Z. The example shown herein is a trailer <NUM> that is located in a rear position of the tractor <NUM>. However, this is merely shown as exemplary in nature and one skilled in the art would be able to apply this embodiment to a trailer <NUM> that is located on a fore or generally aft position of the tractor <NUM> as well. Alternatively, one skilled in the art would be able to apply this embodiment to a tractor <NUM> that would have a front-loading tractor hitch <NUM>. Further, the tractor <NUM> and trailer <NUM> which are shown are using a hitch with a ball style hitch and receiver for the sake of drawing simplicity. Other styles of hitches may be used with the exemplary embodiments.

The tractor <NUM> may include a user interface <NUM>, such as, a display. The user interface <NUM> receives one or more user commands from the operator via one or more input mechanisms or a touch screen display <NUM> and/or displays one or more notifications to the operator. The user interface <NUM> is in communication with a controller <NUM>, which is in turn in communication with a sensor system <NUM>. In some examples, the user interface <NUM> displays an image of an environment of the tractor <NUM> leading to one or more commands being received by the user interface <NUM> (from the operator) that initiate execution of one or more behaviors. The controller <NUM> includes a computing device (or processor) <NUM> (e.g., central processing unit having one or more computing processors) in communication with non-transitory memory <NUM> (e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s)).

The controller <NUM> executes a operator assistance system <NUM>, which in turn includes a path following sub-system <NUM>. The path following sub-system <NUM> receives a planned path <NUM> (<FIG> and <FIG>) from a path planning system <NUM> and executes behaviors <NUM>-<NUM> that send commands <NUM> to the drive system <NUM>, leading to the tractor <NUM> autonomously driving about the planned path <NUM> in a rearward direction R.

The path following sub-system <NUM> includes, a braking behavior <NUM>, a speed behavior <NUM>, a steering behavior <NUM>, a hitch connect behavior <NUM>, and a suspension adjustment behavior <NUM>. Each behavior <NUM>, 330a-c cause the tractor <NUM> to take an action, such as driving backward, turning at a specific angle, breaking, speeding, slowing down, among others. The controller <NUM> may maneuver the tractor <NUM> in any direction across the surface by controlling the drive system <NUM>, more specifically by issuing commands <NUM> to the drive system <NUM>. For example, the controller <NUM> may maneuver the tractor <NUM> from an initial position (as shown in <FIG>) to a final position (as shown in <FIG>). In the final position, a hitch coupler <NUM> of the tractor <NUM> aligns with a hitch coupler <NUM> of the trailer <NUM> connecting the tractor <NUM> and the selected trailer <NUM>, 200a-c.

The tractor <NUM> may include a sensor system <NUM> to provide reliable and robust autonomous driving. The sensor system <NUM> may include different types of sensors that may be used separately or with one another to create a perception of the tractor's environment that is used for the tractor <NUM> to autonomously drive and make intelligent decisions based on objects and obstacles detected by the sensor system <NUM>. The sensors may include, but not limited to, one or more imaging devices (such as cameras) <NUM>, and sensors <NUM> such as, but not limited to, radar, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), etc. In addition, the camera(s) <NUM> and the sensor(s) <NUM> may be used to alert the operator of possible obstacles when the tractor <NUM> is traveling in the forward direction F or in the rearward direction R, by way of audible alerts and/or visual alerts via the user interface <NUM>. Therefore, the sensor system <NUM> is especially useful for increasing safety in tractors <NUM> which operate under semi-autonomous or autonomous conditions.

In some implementations, the tractor <NUM> includes a rear camera <NUM>, 410a that is mounted to provide a view of a rear driving path for the tractor <NUM>. Additionally, in some examples, the tractor <NUM> includes a front camera <NUM>, 410b to provide a view of a front driving path for the tractor <NUM>, a right camera <NUM>, 410c positioned on the right side of the tractor <NUM>, and a left camera <NUM>, 410d positioned on the left side of the tractor <NUM>. The left and right cameras <NUM>, 410c, 410d provide additional side views of the tractor <NUM>. In this case, the tractor <NUM> may detect object and obstacles positioned on either side of the tractor <NUM>, in addition to the objects and obstacle detected along the front and rear driving paths. The camera(s) <NUM>, 410a-d may be a monocular camera, binocular camera, or another type of sensing device capable of providing a view of the rear travelling path of the tractor <NUM>.

In some implementations, the tractor <NUM> includes one or more Neural Networks (NN) <NUM>, for example, Deep Neural Networks (DNN) to improve the autonomous driving of the tractor <NUM>. DNNs <NUM> are computational approaches used in computer science, among other disciplines, and are based on a large collection of neural unites, loosely imitating the way a biological brain solves problems with large clusters of biological neurons connected by axons. DNNs <NUM> are self-learning and trained, rather than programed, and excel in areas where the solution feature detection is difficult to express in a traditional computer program. In other words, DNNs <NUM> are a set of algorithms that are designed to recognize patterns. DNNs <NUM> interpret sensor system data <NUM> (e.g., from the sensor system <NUM>) through a machine perception, labeling or clustering raw input. The recognized patters are numerical, vectors, into which all-real-world data, such as images, text, sound, or time series is translates. The DNN <NUM> includes multiple layers of nonlinear processing units <NUM> in communication with DNN non-transitory memory <NUM>. The DNN non-transitory memory <NUM> stores instructions that when executed on the nonlinear processing units <NUM> cause the DNN <NUM> to provide an output <NUM>, <NUM>. Each nonlinear processing unit <NUM> is configured to transform an input or signal (e.g., sensor system data <NUM>) using parameters that are learned through training. A series of transformations from input (e.g., sensor system data <NUM>) to outputs <NUM>, <NUM> occurs at the multiple layers of the nonlinear processing units <NUM>. Therefore, the DNN <NUM> is capable of determining the location based on images <NUM> or sensor data <NUM> eliminating the need to have a DGPS or a GPS.

The DNN <NUM> receives sensor system data <NUM> (including images <NUM> and/or sensor data <NUM>) and based on the received data <NUM> provides an image output <NUM> to the user interface <NUM> and/or a data output <NUM> to the controller <NUM>. In some examples, the DNN <NUM> receives image(s) <NUM> of a rear view of the tractor <NUM> from the camera <NUM> in communication with the DNN <NUM>. The DNN <NUM> analyzes the image <NUM> and identifies one or more trailers <NUM> in the received image <NUM>. The DNN <NUM> may also receive sensor data <NUM> from the sensors <NUM> in communication with the DNN <NUM>, and analyze the received sensor data <NUM>. Based on the analyzed images <NUM> (or the analyzed images <NUM> and the sensor data <NUM>), the DNN <NUM> identifies the location of each identified trailers <NUM> relative to the tractor <NUM>, for example by way of a coordinate system. As such, the DNN <NUM> displays on the user interface <NUM> the received images <NUM> displaying trailer representations <NUM>, 146a-c of the identified trailers <NUM>, 200a-c located at a distance behind the tractor <NUM>. As shown in <FIG>, first, second, and third trailers 200a, 200b, 200c are positioned behind the tractor <NUM>. As such, the user interface <NUM> displays first, second, and third trailer representations 146a, 146b, 146c associated with the first, second, and third trailers 200a, 200b, 200c respectively.

The operator may select one of the trailer representations <NUM>, 146a-c indicating that the operator wants the tractor <NUM> to autonomously drive and connect to the trailer <NUM>, 200a-c associated with the selected trailer representation <NUM>, i.e., the operator selection <NUM>. In some examples, the user interface is a touch screen display <NUM>; as such, the operator may point his finger and select the trailer representation <NUM>. In other examples, the user interface <NUM> is not a touchscreen and the operator may use an input device, such as, but not limited to, a rotary knob or a mouse to select one of the trailer representations <NUM>, 146a-c.

When the operator selects which trailer <NUM>, 200a-c he/she wants the tractor <NUM> to connect to, a path planning system <NUM> plans a path <NUM> (<FIG>) between the tractor <NUM> and the trailer <NUM> based on the location of the selected trailer <NUM>, 200a-c (determined by the DNN <NUM> from the received sensor system data <NUM>) relative to a position of the tractor <NUM> (e.g., orientation and distance). As the tractor <NUM> is autonomously backing up towards the selected trailer <NUM> (the first trailer 200a as shown in <FIG>). The planned path <NUM> allows the tractor <NUM> to autonomously drive and connect to the trailer <NUM>. The path planning system <NUM> plans the path <NUM> for the tractor <NUM> to autonomously maneuver such that the tractor <NUM>, in an intermediate position within a predetermined distance D from the trailer <NUM>, is in an orientation aligned generally parallel with the trailer <NUM> and facing away from the trailer <NUM>, where the hitch <NUM> of the tractor <NUM> is substantially aligned with the hitch <NUM> of the trailer <NUM>.

In some examples, the path planning system <NUM> is part of the controller <NUM> as shown in FIG. 2A; while in other examples, the path planning system <NUM> is part of the DNN <NUM> as shown in FIG. Referring to FIG. 2A, when the operator selects the trailer representation 146a associated with the trailer 200a that the operator wants the tractor <NUM> to autonomously drive towards and connect with, the DNN 500a sends the controller <NUM> data output <NUM> including the selected trailer 200a and the location of the selected trailer 200a with respect to the tractor <NUM>. In this case, the path planning system 550a plans the path <NUM> between the tractor <NUM> and the selected trailer 200a. The path planning system 550a may use several methods to determine the path <NUM>. <FIG> and <FIG> provide a method for path planning. In some examples, the path planning system 550a extends its fore-aft axis Y in the rearward direction R while the trailer 200a extends a fore-aft axis about the length of the trailer <NUM> in a forward direction. The path planning system 550a draws a first circle <NUM> tangent at a first tangent point <NUM> to the tractor fore-aft axis Y facing the trailer fore-aft axis, and a second circle <NUM> tangent at a second tangent point <NUM> to the trailer fore-aft axis facing the tractor fore-aft axis Y. The first and second circles <NUM>, <NUM> intersect at an intersection point <NUM>. The size of the first and second circles <NUM>, <NUM> may be adjusted and manipulated based on the distance between the tractor <NUM> and the trailer <NUM>, obstacles and object positioned between the tractor <NUM> and the trailer <NUM>, and any other considerations. The path planning system 550a determines the path <NUM> by following the tractor fore-aft axis Y until the first tangent point <NUM>, then moving along an arc of the first circle <NUM> until the intersection point <NUM>, then moving along an arc of the second circle <NUM> until the second tangent point <NUM>, then following the trailer fore-aft axis. As such, the planned path <NUM> positions the tractor <NUM> in an orientation aligned generally parallel with the trailer <NUM> and facing away from the trailer 200a, where the hitch <NUM> of the tractor <NUM> is substantially aligned with the hitch receiver <NUM> of the trailer hitch <NUM> for the trailer <NUM>. In other words, where the fore-aft axis Y of the tractor <NUM> is substantially aligned with the fore-aft axis T of the trailer 200a. <FIG> shows an example of the path <NUM> where the fore-aft axis Y of the tractor <NUM> is substantially parallel to the fore-aft axis T of the trailer 200a. While <FIG> shows an example of the path <NUM> where the fore-aft axis Y of the tractor <NUM> is not substantially parallel to the fore-aft axis T of the trailer 200a.

With continued reference to FIGS. 2A, <FIG> and <FIG>, in some examples, when the tractor <NUM> is autonomously driving along the planned path <NUM>, the DNN <NUM> continuously sends the controller <NUM> the location of the selected trailer 200a with respect to the tractor <NUM> as the tractor <NUM> moves along the planned path <NUM> based on the received sensor system data <NUM>, i.e., images <NUM>. Since the images <NUM> are updated as the tractor <NUM> approaches the selected trailer 200a, then the location of the selected trailer 200a with respect to the tractor <NUM> also changes as the tractor <NUM> moves closer to the selected trailer 200a. In some examples, the DNN <NUM> identifies one or more objects along the planned path <NUM> and sends the path planning system 550a data relating to the position of the one or more objects. In this case, the path planning system 550a may recalculate the planned path <NUM> to avoid the one or more objects. In some examples, the path planning system 550a determined a probability of collision and if the probability of collision exceeds a predetermined threshold, the path planning system 550a adjusts the path and sends it to the path following sub-system <NUM>.

Referring back to FIG. 2B, in some implementations the DNN <NUM> includes the path planning system 550b. As such, the path planning system 550b may determine a path <NUM> based on learned behaviors. For example, the DNN <NUM> determines the position of the selected trailer 200a, and based on the position determines a path <NUM> for the tractor <NUM>. In some examples, the determined path <NUM> may be similar to the path described with respect to <FIG> and <FIG>. However, other path determination methods may also be possible. Similarly, the DNN <NUM>, i.e., the path planning system 550b of the DNN <NUM> adjusts the planned path <NUM> based on one or more objects (moving or stationary) that may be identified within the planned path <NUM> as the tractor <NUM> is moving about the path <NUM>. In some examples, the path planning system 550b determined a probability of collision and if the probability of collision exceeds a predetermined threshold, the path planning system 550b adjust the path and sends it to the path following sub-system <NUM>.

Referring back to FIGS. 2A and 2B, once the path planning system <NUM> plans a path <NUM>, the path following sub-system <NUM> is configured to execute behaviors the cause the drive system <NUM> to autonomously follow the planned path <NUM>. Therefore, the path following sub-system <NUM> includes one or more behaviors <NUM>-<NUM> that once executed allow for the autonomous driving of the tractor <NUM> along the planned path <NUM>. The behaviors <NUM>-<NUM> may include, but are not limited to a braking behavior <NUM>, a speed behavior <NUM>, a steering behavior <NUM>, a hitch connect behavior <NUM>, and a suspension adjustment behavior <NUM>.

The braking behavior <NUM> may be executed to either stop the tractor <NUM> or to slow down the tractor based on the planned path <NUM>. The braking behavior <NUM> sends a signal or command <NUM> to the drive system <NUM>, e.g., the brake system <NUM>, to either stop the tractor <NUM> or reduce the speed of the tractor <NUM>.

The speed behavior <NUM> may be executed to change the speed of the tractor <NUM> by either accelerating or decelerating based on the planned path <NUM>. The speed behavior <NUM> sends a signal or command <NUM> to the brake system <NUM> for decelerating or the acceleration system <NUM> for accelerating.

The steering behavior <NUM> may be executed to change the direction of the tractor <NUM> based on the planned path. As such, the steering behavior <NUM> sends the acceleration system <NUM> a signal or command <NUM> indicative of an angle of steering causing the drive system <NUM> to change direction.

<FIG> show the tractor <NUM> at an initial position Pi (<FIG>), an intermediate position PM (<FIG>), and a final position PF (<FIG>) or a connected position, with respect to the selected trailer 200a. Referring to <FIG>, the tractor <NUM> is at the initial position Pr relative to the trailer <NUM> prior to initiating autonomous maneuvering towards the selected trailer 200a. In some examples, an initial distance DI between the tractor <NUM> and the selected trailer 200a is about <NUM> meters. The tractor <NUM> autonomously maneuvers along the planned path <NUM> until the tractor <NUM> reaches an intermediate position PM being an intermediate distance DM from the selected trailer 200a, as shown in <FIG>. In the intermediate position PM, the tractor hitch <NUM> is in an orientation aligned generally parallel with the selected trailer 200a and the tractor hitch <NUM> is substantially aligned with the trailer hitch receiver/coupler <NUM> of trailer hitch <NUM>. In other words, the tractor fore-aft Y defines a plane that extends along the tractor vertical axis Z and along the trailer fore-aft T along a trailer vertical axis. In some examples, the intermediate distance DM is about <NUM> meter.

Referring to <FIG>, in some examples, when the tractor <NUM> is in the intermediate position PM the hitch connect behavior <NUM> executes to connect the tractor hitch <NUM> with the trailer hitch <NUM>. The example shown is using a ball style hitch and receiver. However, the exemplary embodiment of <FIG> may be implemented with any style hitch that utilizes a vertical arrangement between the tractor hitch and the hitch receiver. For hitch styles which involve a horizontal arrangement between the tractor hitch and the hitch receiver, vertical maneuvers may only be necessary to obtain the desired horizontal alignment between the components, or not necessary at all.

The controller <NUM> determines a relative height HR between a top portion of the tractor hitch coupler <NUM> and a bottom portion of the trailer hitch coupler <NUM>. To connect the tractor <NUM> and the selected trailer 200a, the trailer hitch coupler <NUM> releasably receives the tractor hitch coupler <NUM>. Therefore, to connect the tractor hitch coupler <NUM> to the trailer hitch coupler <NUM>, the relative height HR has to equal zero allowing the tractor hitch coupler <NUM> to move under and be inserted in the trailer hitch coupler <NUM>. Therefore, when the hitch connect behavior <NUM> receives the relative height HR that is greater than zero between the tractor hitch coupler <NUM> and the trailer hitch coupler <NUM> from the controller <NUM>, the hitch connect behavior <NUM> sends a command to the suspension adjustment behavior <NUM> to execute and issue a command <NUM> to the suspension system <NUM> to adjust the height of the tractor <NUM> reducing the relative height HR based on the measurements from the controller <NUM>. When the hitch connect behavior <NUM> receives the relative height HR that is equal to zero, then the hitch connect behavior <NUM> issues a command <NUM> to the drive system <NUM> to maneuver along the remainder of the path <NUM>, i.e., from the intermediate position PM to the final position PF (<FIG>), connecting the tractor <NUM> to the selected trailer 200a.

Referring to <FIG> and <FIG> a second exemplary tractor <NUM> is illustrated. Again, the tractor <NUM> may include a camera <NUM> configured to capture images <NUM> of the environment of the tractor <NUM>. The images <NUM> are displayed on a display <NUM> that shows a trailer representation of the trailer <NUM> positioned behind the tractor <NUM>. The operator identifies a trajectory or path <NUM> within the received images <NUM> that will position the tractor hitch coupler <NUM> under or adjacent the trailer hitch coupler <NUM>. Based on the operator identified path <NUM> within the images <NUM>, the tractor <NUM> autonomously drives along the identified path <NUM>.

The tractor <NUM> may include a user interface <NUM>. The user interface <NUM> may include the display <NUM>, a knob <NUM>, and a button <NUM>, which are used as input mechanisms. In some examples, the display <NUM> may show the knob <NUM> and the button <NUM>. While in other examples, the knob <NUM> and the button <NUM> are a knob button combination. In some examples, the user interface <NUM> receives one or more operator commands from the operator via one or more input mechanisms or a touch screen display <NUM> and/or displays one or more notifications to the operator. The user interface <NUM> is in communication with a tractor controller <NUM>, which is in turn in communication with a sensor system <NUM>. In some examples, the display <NUM> displays an image of an environment of the tractor <NUM> leading to one or more commands being received by the user interface <NUM> (from the operator) that initiate execution of one or more behaviors. In some examples, the user display <NUM> displays a representation image of the rearward environment of the tractor <NUM>. In this case, the operator can select a position within the representation of the image that is indicative of the environment that the operator wants the vehicle to autonomously maneuver towards. In some examples, the user display <NUM> displays one or more representations of trailers <NUM> positioned behind the tractor <NUM>. In this case, the operator selects which representation of a trailer <NUM> the tractor <NUM> to autonomously maneuver towards.

The display <NUM> displays an expected path <NUM> of the tractor <NUM> that is superimposed on a camera image <NUM> of the rearward environment of the tractor <NUM>. The operator may change the expected path <NUM> using the user interface <NUM>. For example, the operator may turn the knob <NUM>, which simulates a virtual steering wheel. As the operator is turning the knob <NUM>, the expected path shown on the display <NUM> updates. The operator adjusts the displayed path <NUM> until an expected path displayed on the display <NUM> intersects the trailer representation or other object that the user wants the tractor <NUM> to drive towards. Once the user is satisfied with the expected path <NUM> displayed, then the operator executes an action indicative of finalizing the path <NUM> which allows the tractor <NUM> to autonomously follow the planned path <NUM>. The planned path <NUM> is the path from the tractor hitch coupler <NUM> to the trailer hitch receiver/coupler <NUM>. The wheel path <NUM> is the estimated path the wheels will move along the planned path <NUM>.

In some implementations, the user may adjust three path modes <NUM>, including an angle mode <NUM> for adjusting a curvature angle of the path as shown in <FIG>, a distance mode <NUM> for adjusting a length of the expected path <NUM> as shown in <FIG>, and optionally a bi-arc mode <NUM> for adjusting and "approach angle" for how the vehicle will be oriented with respect to the trailer (or other object) at the end of the maneuver as shown in <FIG>. Since the bi-arc mode is optional, if the user is satisfied with the currently constructed path (a circular arc), then the user can finalize the maneuver choice by pressing the button another time. Otherwise, the user adjusts the knob for the third time to change the shape of a bi-arc or other suitable path. This allows for adjusting the final approach angle to the trailer or other object. Once the user is satisfied with the choice of approach angle, he/she presses the button to finalize the path choice.

The tractor controller <NUM> includes a computing device (or processor) <NUM> (e.g., central processing unit having one or more computing processors) in communication with non-transitory memory <NUM> (e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s) <NUM>.

The tractor controller <NUM> executes a operator assistance system <NUM> that receives a planned path <NUM> from a path system <NUM> and executes behaviors <NUM>, 330a-330c that send commands <NUM> to the drive system <NUM>, leading to the tractor <NUM> autonomously driving about the planned path <NUM> in a rearward direction R (in the example shown).

The path following behaviors <NUM> include a braking behavior 330a, a speed behavior 330b, and a steering behavior 330c. In some examples, the path following behaviors <NUM> also include a hitch connect behavior, and a suspension adjustment behavior. Each behavior <NUM>, 330a-330c causes the tractor <NUM> to take an action, such as driving backward, turning at a specific angle, breaking, speeding, slowing down, among others. The tractor controller <NUM> may maneuver the tractor <NUM> in any direction across the surface by controlling the drive system <NUM>, more specifically by issuing commands <NUM> to the drive system <NUM>.

The tractor <NUM> may include a sensor system <NUM> to provide reliable and robust driving. The sensor system <NUM> may include different types of sensors that may be used separately or with one another to create a perception of the environment of the tractor <NUM> that is used for the tractor <NUM> to drive and aid the operator in make intelligent decisions based on objects and obstacles detected by the sensor system <NUM>. The sensor system <NUM> may include the one or more cameras <NUM>. In some implementations, the tractor <NUM> includes a rear camera <NUM>, 410a that is mounted to provide a view of a rear-driving path for the tractor <NUM>. The rear camera <NUM> may include a fisheye lens that includes an ultra wide-angle lens that produces strong visual distortion intended to create a wide panoramic or hemispherical image. Fisheye cameras capture images having an extremely wide angle of view. Moreover, images captured by the fisheye camera have a characteristic convex non-rectilinear appearance. Other types of cameras may also be used to capture images of the rear of the tractor <NUM>.

The sensor system <NUM> may also include the IMU (inertial measurement unit) <NUM> configured to measure the tractor's linear acceleration (using one or more accelerometers) and rotational rate (using one or more gyroscopes). In some examples, the IMU <NUM> also determines a heading reference of the tractor <NUM>. Therefore, the IMU <NUM> determines the pitch, roll, and yaw of the tractor <NUM>.

The sensor system <NUM> may include other sensors such as, but not limited to, radar, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), etc..

The tractor controller <NUM> executes a hitch assist system <NUM> that receives images <NUM> from the camera <NUM> and superimposes a tractor path <NUM> on the received image <NUM>.

Referring back to <FIG>, the tractor controller <NUM> executes a hitch assist system <NUM> that aids the operator in selecting a path <NUM> for autonomously driving the tractor <NUM> towards the trailer <NUM>. The operator may initiate execution of the hitch assist system <NUM> by way of the user interface <NUM>, for example, making a selection on the display <NUM>. Once initiated, the hitch assist system <NUM> instructs the display <NUM> to display a path <NUM> of the tractor <NUM> that is superimposed on the camera image <NUM> of the rearward environment of the tractor <NUM>. The operator may change the planned path <NUM> using the user interface <NUM>. For example, the operator may turn the knob <NUM>, which simulates a virtual steering wheel. As the operator is turning the knob <NUM>, the planned path <NUM> shown on the display <NUM> is updated. The operator adjusts the displayed path <NUM> until an updated planned path <NUM> displayed on the display <NUM> intersects the trailer representation <NUM> or other object that the operator wants the tractor <NUM> to drive towards. Once the operator is satisfied with the planned path <NUM> displayed, then the operator executes an action indicative of finalizing the path <NUM> which allows the tractor <NUM> to autonomously follow the planned path.

In some implementations, the hitch assist system <NUM> includes a trajectory generator <NUM>, a motion estimator <NUM>, and a path tracker <NUM>. The trajectory generator <NUM> determines an estimated position of the tractor <NUM> based on the operator selected path <NUM>. The motion estimator <NUM> determines an actual position of the tractor <NUM>, and the path tracker <NUM> determines an error <NUM> based on the estimated position Pe and the actual position Pa and adjusts the planned path <NUM> of the tractor <NUM> to eliminate the error <NUM> between the actual position Pa and the estimated position Pe.

In some implementations, the trajectory generator <NUM> receives images <NUM> from the camera <NUM> and superimposes the vehicle path <NUM> on the received image <NUM>. The operator may adjust the path <NUM> selection based on one or more path modes <NUM>. In some examples, the path modes <NUM> include an arc mode <NUM> having an angle sub-mode <NUM> and a distance sub-mode <NUM>. In some examples, the path modes <NUM> may include a bi-arc mode <NUM>. Therefore, the operator may select between the angle sub-mode <NUM>, the distance sub-mode, and/or the bi-arc mode <NUM> for determining and adjusting the path <NUM> to a trailer <NUM> or an object.

Referring to <FIG>, in some examples, the angle sub-mode <NUM> and the distance sub-mode <NUM> are part of the arc-mode <NUM> (<FIG>), therefore, the operator first selects a mode <NUM>, <NUM>, and then selects the sub-mode within the selected mode <NUM>, <NUM>. As such, for examples, the display <NUM> may display an arc mode button <NUM> and a bi-arc mode button <NUM>, that the operator can select from. <FIG> shows an example, where each sub-mode/mode <NUM>, <NUM>, <NUM> is independent. Therefore, a press or push of the button <NUM> rotates between the three modes <NUM>, <NUM>, <NUM>.

The angle sub-mode <NUM> is configured to adjust a curvature angle of the path <NUM> as shown in <FIG>. Therefore, the operator can turn the knob <NUM> to the right causing the displayed path <NUM> to have a curvature to the right as shown in <FIG>. In addition, the operator my turn the knob <NUM> to the left causing the displayed path <NUM> to have a curvature to the left as shown in <FIG>. The distance sub-mode <NUM> is configured to adjust a length of the expected path <NUM> as shown in <FIG>. For examples, referring to <FIG>, the operator can rotate the knob <NUM> to position a destination of the path <NUM> adjacent the trailer representation <NUM> in the image <NUM>. Referring to <FIG>, the image <NUM> shows the path <NUM> having a shorter length than the path shown in <FIG>. Therefore, in this case, the operator may want that the tractor <NUM> to autonomously move few meters in the rearward direction R. The bi-arc mode <NUM> is configured to adjust an approach angle indicative of how the tractor <NUM> will be oriented with respect to the trailer <NUM> (or other object) at the end of the path <NUM> as shown in <FIG>. For example, the bi-arc mode <NUM> aids the operator in aligning the tractor <NUM> with the trailer <NUM> such that the fore-aft axis Y of the tractor <NUM> is aligned with a fore-aft axis Y of the trailer <NUM>, which helps the operator during the hitching process between the tractor <NUM> and the trailer <NUM>. Referring to <FIG>, the arc mode <NUM> and the bi-arc mode <NUM> both have the same end point; however, the bi-arc mode <NUM> allows for an adjustment of the approach angle towards the trailer <NUM>. In some examples, when the operator switches to bi-arc mode <NUM>, the trajectory generator <NUM> keeps the start and end locations the same. In the bi-arc mode <NUM>, the operator may adjust only the approach angle to the trailer <NUM>. The operator does not adjust the distance in the bi-arc mode <NUM>. In some examples, the radius and length of the operator selected path <NUM> determine the final position of the tractor <NUM>. The vehicle controller <NUM> uses Dubins path to determine the path <NUM> which is an optimal calculation.

In some implementations, where the bi-arc mode <NUM> is optional, if the operator is satisfied with the path <NUM> based on the arc mode <NUM> selection, then the operator may finalize the path <NUM> by pressing the button <NUM>. Otherwise, the operator adjusts the knob <NUM> for the third time to change the shape of a bi-arc or other suitable path <NUM>. This allows for adjusting the final approach angle to the trailer <NUM> or other object. Once the operator is satisfied with the choice of approach angle, he/she presses the button <NUM> to finalize the path choice.

In some implementations, the operator parks the tractor <NUM> in a location where the trailer <NUM>, or other object or point of interest, is within a field of view of the rear camera <NUM> of the tractor <NUM>. The engine of the tractor <NUM> may be idling, and the transmission in Park position. The operator may initiate the trajectory generator <NUM> by pressing the button <NUM> and/or making a selection on the display <NUM>. In some examples, the display <NUM> shows a selectable option or button <NUM> allowing the operator to initiate the Arc mode <NUM>. The trajectory generator <NUM> begins by executing the angle sub-mode <NUM> of the arc mode <NUM>, as shown in <FIG>. The operator switches to distance sub-mode <NUM> to adjust the distance of the path <NUM>, by for example, pressing the button <NUM>. In some examples, the operator may adjust the path <NUM> by switching between angle sub-mode <NUM> and distance sub-mode <NUM> and adjusting the path <NUM> until the desired path <NUM> is shown on the display. The operator adjusts the path <NUM> such that the outer boundaries <NUM> of the path <NUM> interest the trailer <NUM> (i.e., the trailer representation <NUM> within the image <NUM>) or other point of interest.

In some implementations, the final approach angle to the trailer <NUM> or the point of interest is important, for example, for aligning the vehicle fore-aft axis Y with the trailer fore-aft axis Y. In this case, the operator may select or press the "Arc/Bi-Arc Mode" button <NUM> (displayed on the display <NUM>) and switch to the bi-arc mode <NUM>. In the bi-arc mode <NUM> the previously set endpoint of the path <NUM> stays constant, and the operator adjusts the final approach angle with the knob <NUM>. When the operator is satisfied with the final approach angle and with the complete trajectory or path <NUM>, the operator may confirm the selected path <NUM> by executing an action. In some examples, the operator switches the transmission to reverse which is indicative that the operator is satisfied with the displayed path <NUM>. In some examples, the operator switches the transmission into reverse with the brake on, then releases the brake, and the tractor <NUM> follows the selected path <NUM>. In some examples, while the vehicle is autonomously maneuvering in the rearward direction R along the path <NUM>, the operator may stop the tractor <NUM> by, for example, pressing the brake. This causes the vehicle controller <NUM> to exit the hitch assist system <NUM>.

In some implementations, the trajectory generator <NUM> sets the path distance at a default, which allows the operator to only adjust the steering angle until it intersects the trailer <NUM> or other point of interest.

In some implementation, the final approach angle is not adjusted. Instead, the final approach angle is always the same as the initial vehicle departure angle. So, the final vehicle fore-aft axis Y is parallel to the initial vehicle fore-aft axis Y. In this case, the operator adjusts the final location of the path <NUM> to interest with the trailer.

In some examples, while the tractor <NUM> is maneuvering in the rearward direction R along the path <NUM>, the display <NUM> may show a progress of the tractor <NUM> along the path <NUM>. For example, the display <NUM> may show an original trajectory projected on the ground, but updated by the vehicle's changing position. The display <NUM> may also show an indication of how well the vehicle is following this trajectory.

In some implementations, the trajectory generator <NUM> receives data from other vehicle systems to generate the path <NUM>. In some examples, the trajectory generator <NUM> receive vehicle pose data defined by (x, y, θ) where x is the position of a center of the tractor <NUM> along the transverse axis X in and X-Y plane, y is the position of a center of the vehicle along the fore-aft axis Y in the X-Y plane, and θ is the heading of the tractor <NUM>. In addition, the trajectory generator <NUM> may receive a position of the knob <NUM>, e.g., a knob angle, from the knob <NUM>. The trajectory generator <NUM> may also receive a mode button state (i.e., arc mode <NUM> or bi-arc mode <NUM>), and the sub-mode button state (i.e., angle sub-mode <NUM> or distance sub-mode <NUM>). Based on the received data, the trajectory generator <NUM> adjusts the path <NUM> and instructs the display <NUM> to display the path <NUM>. In some examples, the path <NUM> includes outer boundaries <NUM> and a tow ball path <NUM> being the estimated path of the tow ball <NUM>. The trajectory generator <NUM> may also instruct the display <NUM> to show the current mode or sub-mode status indicative of the mode/sub-mode the drive has selected to adjust the path <NUM>.

Referring back to <FIG> and <FIG>, once the operator indicates via the user interface <NUM> that his path selection is complete, the vehicle controller <NUM> executes the operator assistance system <NUM> to follow the planned path <NUM>. The operator assistance system <NUM> includes path following behaviors <NUM>. The path following behaviors <NUM> receive the selected path <NUM> and executes one or more behaviors 330a-b that send commands <NUM> to the drive system <NUM>, causing the tractor <NUM> to autonomously drive along the planned path <NUM>. As the tractor <NUM> is autonomously maneuvering along the planned path <NUM>, the hitch assist system <NUM> continuously updates the path <NUM> based on the motion estimator <NUM> and the path tracker as discussed below.

Referring back to <FIG> and <FIG>, the motion estimator <NUM> determines a current position Pa of the tractor <NUM> as it is autonomously maneuvering in the rearward direction R along the path <NUM>. As previously discussed, the trajectory generator <NUM> determines where the tractor <NUM> should be based on the planned path <NUM>, i.e., estimated position Pe; therefore the motion estimator determines an actual position Pa of the tractor <NUM>. In some examples, the motion estimator includes a motion estimation algorithm that outputs relative vehicle position and speed. For example, the motion estimation algorithm may include an Extended Kalman Filter (EKF). The EKF uses measurements, such as, but not limited to, encoders from <NUM> wheels (e.g., <NUM> ticks per revolution), and steering angle. The motion estimator <NUM> fuses the measurements to determine the actual position of the tractor <NUM>. The motion estimator <NUM> may use bicycle model due to the slow speed of the tractor <NUM> as it is autonomously moving in the rearward direction. The bicycle model uses a single steered wheel in front to represent both front wheels, and it uses a single non-steered wheel in the rear to represent both rear wheels. The wheels are joined by a single rigid link. Motion is restricted to a two-dimensional horizontal ground plane. The inputs of the bicycle model are speed and steer angle, while its state is position and heading. The motion estimator <NUM> estimates linear and rotational speeds, and position (e.g., orientation of the vehicle). In some examples, the motion estimator <NUM> considers sensor data from the sensor system <NUM>, for example, camera, radar, GPS measurement to improve any drift.

In some implementations, the motion estimator <NUM> uses an Extended Kalman Filter (EKF). The EKF equations are also provided below as equations (<NUM>) - (<NUM>). The state vector has nine elements shown in equation <NUM>: <MAT> The first three are the "pose" of the vehicle, (x, y, θ). The next two are the linear and angular speeds, (v, ω). The final four are the distances traveled by the four tires, (dlr, drr, dlf, drf), left-rear, right-rear, left-front, right-front, respectively.

The complete measurement vector has five elements shown in equation <NUM>: <MAT> The first four are, again, the distances traveled by the four tires, (dlr, drr, dlf, drf). The final element ϕ is the average front wheel angle (not the steering wheel angle but the average angle of the front tires with respect to the longitudinal axis). The motion estimator <NUM> provides a vehicle speed estimate. It is common to compute an approximate speed by dividing distance change by time change (Δd/Δt), however, this would be very noisy for the situation here, where there are relatively few wheel encoder counts, and the vehicle is moving relatively slowly. Thus, to avoid a direct calculation involving dividing distance change by time change (Δd/Δt), the motion estimator <NUM> estimates the vehicle linear speed v by using the EKF, based on the measured wheel accumulated distances, and there is no explicit rate calculation involving division.

The Extended Kalman Filter may be written as two Prediction equations and three Measurement equations. The Prediction equations are: <MAT> <MAT> Equation (<NUM>) provides an update to the state µ. Equation (<NUM>) provides an update to the covariance Σ. The covariance provides an estimate of the current uncertainty of the states. The matrix R is the noise covariance for the state µ.

The measurement update equations are: <MAT> <MAT> Σt = (I - KtHt)Σt. (<NUM>)Equation (<NUM>) sets the value of the optimal Kalman gain K. Equation (<NUM>) provides an update to the state µ. Equation (<NUM>) provides an update to the covariance Σ. The matrix Q is the noise covariance for the measurement z.

For prediction the nonlinear vector function g needs to be defined. The matrix G is the derivative of this vector function. For convenience, it will be provided as well. The vector function g is given by: <MAT> Here, w is the "track width" of the vehicle. More specifically, it is the lateral distance from the center of a left tire to the center of a right tire. It is assumed that the front and rear distances between tires are the same. The wheelbase is denoted by l. Note that in the last two elements, there is a minus sign where expressions are subtracted from dlf and drf. This minus sign assumes backward motion. Thus, this prediction equation could not be used for forward motion. However, if there were some measurement of the direction of the vehicle (forward or backward), then it would be a simple matter to change the sign of the last two elements (positive for forward, negative for backward) to make the equations valid for both forward and backward directions.

The matrix G has nine rows and nine columns. Let <MAT> I is the <NUM>×<NUM> identity matrix. Then, <MAT> All other elements are zero.

The complete update assumes measurements of the wheel ticks and the wheel angle are all available at the same time. If only the wheel ticks are available, then they may be incorporated separately, and if only the wheel angle is available, it may be incorporated separately. For the complete update, the vector h is defined. The matrix H is the derivative of this vector function. For convenience, it will be provided as well. The vector function h is given by: <MAT> The H matrix is the derivative of h and is <NUM>×<NUM>. The non-zero elements are <MAT> Note that certain quantities, h<NUM>, H<NUM>, H<NUM>, involve divisors which could easily be zero. Thus, implementation requires testing that these divisors are not zero before performing the divisions.

A measurement consisting of wheel ticks alone can be accommodated, as can a measurement consisting of steering angle alone. However, these variations are not included because they are straightforward, given the information that has been provided.

During the vehicle's autonomous maneuvering in the rearward direction R along the planned path <NUM>, the trajectory generator <NUM> determines where the tractor <NUM> should be based on the planned path <NUM>, i.e., estimated position Pe; while the motion estimator <NUM> determines an actual position Pa of the tractor <NUM>; therefore the path tracker <NUM> determines an error <NUM> based on the estimated position Pe and the actual position Pa. The path tracker <NUM> adjusts the vehicle's current position Pa based on the error <NUM> such that the tractor <NUM> continues to follow the planned path <NUM>.

Referring to <FIG>, in some implementations, the path tracker <NUM> executes a pure pursuit approach to keep the tractor <NUM> on the planned path <NUM>. The operator selected path <NUM> is sampled at predefined intervals of time, for examples, every minute, to produce a plurality of waypoints <NUM> positioned along the tow ball path <NUM> of the planned path <NUM>. The path tracker <NUM> (e.g., algorithm) compares a current tow-ball position and heading Pa received from the motion estimator <NUM> with the next waypoint position Pb. The vehicle hitch assist system <NUM> constantly adjusted the vehicle steering toward the current waypoint Pb, i.e., the waypoint that the vehicle is driving towards. The path tracker <NUM> allows the vehicle tow-ball <NUM> to track each waypoint <NUM>. In other words, the path tracker <NUM> allows for the tow-ball <NUM> to go to each waypoint along the tow ball path <NUM>. In some examples, the waypoint <NUM> is transformed into vehicle coordinates from world coordinates. For example, the path tracker <NUM> calculates a center Cc of a turning circle based on the tow-ball position Pa and the waypoint position Pb. Then the path tracker <NUM> calculates a vehicle turning radius Rr based on the center of the turning circle. Finally, the path tracker <NUM> calculates a steering angle based on the center of turning circle using Ackermann angle. In other words, the path tracker <NUM> compares an estimated location Pe with the current location Pa to make sure the vehicle is following the path and determines a next waypoint Pb and determines or adjusts the path from the current vehicle position and heading to the next or subsequent waypoint Pb. Thus the path tracker <NUM> maintains that the tractor <NUM> autonomously maneuvers along the planned path and adjusts the vehicle's behavior or driving when the tractor <NUM> veers from the planned path.

In some examples, the controller includes an object detection system (not shown) that identifies one or more objects along the planned path <NUM>. In this case, the hitch assist system <NUM> adjusts the path <NUM> to avoid the detected one or more objects. In some examples, the hitch assist system <NUM> determines a probability of collision and if the probability of collision exceeds a predetermined threshold, the hitch assist system <NUM> adjusts the path <NUM> and sends it to the operator assistance system <NUM>.

Once the operator indicated that the selected path <NUM> is entered, then the vehicle controller <NUM> executes a operator assistance system <NUM>, which in turn includes path following behaviors <NUM>. The path following behaviors <NUM> receive the selected path <NUM> and executes one or more behaviors 330a-b that send commands <NUM> to the drive system <NUM>, causing the tractor <NUM> to autonomously drive along the planned path in the rearward direction R.

The path following behaviors 330a-b may include one or more behaviors, such as, but not limited to, a braking behavior 330a, a speed behavior 330b, and a steering behavior 330c. Each behavior 330a-b causes the tractor <NUM> to take an action, such as driving backward, turning at a specific angle, breaking, speeding, slowing down, among others. The vehicle controller <NUM> may maneuver the tractor <NUM> in any direction across the road surface by controlling the drive system <NUM>, more specifically by issuing commands <NUM> to the drive system <NUM>.

The braking behavior 330a may be executed to either stop the tractor <NUM> or to slow down the tractor <NUM> based on the planned path. The braking behavior 330a sends a signal or command <NUM> to the drive system <NUM>, e.g., the brake system (not shown), to either stop the tractor <NUM> or reduce the speed of the tractor <NUM>.

The speed behavior 330b may be executed to change the speed of the tractor <NUM> by either accelerating or decelerating based on the planned path <NUM>. The speed behavior 330b sends a signal or command <NUM> to the brake system <NUM> for decelerating or the acceleration system <NUM> for accelerating.

The steering behavior 330c may be executed to change the direction of the tractor <NUM> based on the planned path <NUM>. As such, the steering behavior 330c sends the acceleration system <NUM> a signal or command <NUM> indicative of an angle of steering causing the drive system <NUM> to change direction.

<FIG> illustrates an example arrangement of operations for a method <NUM> of autonomously maneuvering a tractor <NUM> (as shown in <FIG>) towards a selected trailer <NUM>, 200a. At block <NUM>, the method <NUM> includes initiating a hitch assist mode by receiving an indication that a operator wants to autonomously hitch the tractor <NUM> to a trailer <NUM>, 200a. The indication may be by way of a selection on the user interface <NUM> of the tractor <NUM>, putting the tractor in reverse (without reversing), or any other indication. At block <NUM>, the DNN <NUM> detects and locates one or more trailers <NUM>, 200a-c behind the tractor <NUM> and displays by way of the user interface <NUM> trailer representations <NUM>, 146a-c associated with the one or more identified trailers <NUM>, 200a-c respectively. Alternatively, the camera image is shown on the display142. At decision block <NUM>, the method <NUM> waits for a operator selection <NUM>. The operator may select from the displayed image by choosing a from the one or more trailer representations <NUM>, 146a-c on the display or by manipulating a user input knob <NUM> until a path representation, i.e. path tracker <NUM>, is aligned with a point of interest, i.e. the trailer representation <NUM> including the hitch image position, on the displayed image. When the operator selects a trailer representation <NUM>, 146a-c associated with a selected trailer 200a, at block <NUM>, the method <NUM> includes planning a tractor path <NUM>, from an initial position PI of the tractor <NUM> to a final position PF with respect to the selected trailer <NUM>. In some examples, the path planning system <NUM>, 550a, 550b plans the path <NUM>. The path planning system <NUM>, 550a, 550b may be part of the controller <NUM> or part of the DNN <NUM>. At block <NUM>, the method <NUM> includes executing the path following sub-system <NUM>. At decision block <NUM>, the method <NUM> determines if the tractor <NUM> is within a predetermined distance from the selected trailer 200a, i.e., the method <NUM> determines if the tractor <NUM> has reached the intermediate position PM. When the tractor <NUM> reaches the intermediate position PM, the method <NUM> at decision block <NUM> determines a relative height HR between a top portion of the tractor hitch coupler <NUM> of the tractor <NUM> and a bottom portion of the hitch coupler <NUM> of the selected trailer 200a and determines if the hitch coupler <NUM> can releasably receive the tractor hitch couler <NUM> based on the relative height HR. In other words, the method <NUM> determines if the relative height HR equals to zero. If the relative height HR is not equal to zero, then at block <NUM>, the method <NUM> adjusts the suspension of the tractor <NUM> and then determines the relative height HR and checks if the relative height HR equals zero at block <NUM>. Once relative height HR is equal to zero, then the method <NUM> at block <NUM> continues maneuvering about the path <NUM> from the intermediate position PM to a final position PF connecting the tractor hitch coupler162 of the tractor <NUM> with the hitch coupler <NUM> of the selected trailer 200a.

<FIG> provides an example arrangement of operations of a method <NUM> for autonomously maneuvering a tractor <NUM> (e.g., a tow vehicle) in a rearward direction R towards a point of interest, such as a trailer <NUM>, using the system described in <FIG>. At block <NUM>, the method <NUM> includes, receiving, at data processing hardware <NUM>, one or more images <NUM> from a camera <NUM> positioned on a back portion of the tractor <NUM> and in communication with the data processing hardware <NUM>. At block <NUM>, the method <NUM> includes receiving, at the data processing hardware <NUM>, a operator planned path <NUM> from a user interface <NUM> in communication with the data processing hardware <NUM>. The operator planned path <NUM> includes a plurality of waypoints <NUM>. At block <NUM>, the method <NUM> includes transmitting, from the data processing hardware <NUM> to a drive system <NUM> in communication with the data processing hardware <NUM>, one or more commands <NUM>, <NUM> causing the tractor <NUM> to autonomously maneuver along the operator planned path <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a current vehicle position Pa. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, an estimated subsequent vehicle position based on the operator planned path, the estimated subsequent vehicle position being at a subsequent waypoint Pb along the operator planned path <NUM> from the current vehicle position Pa. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a path adjustment from the current vehicle position Pa to the estimated subsequent vehicle position Pb. At block <NUM>, the method <NUM> includes transmitting, from the data processing hardware <NUM> to the drive system <NUM>, instructions causing the tractor <NUM> to autonomously maneuver towards the estimated subsequent vehicle position Pb based on the path adjustment.

<FIG> provides an example arrangement of operations of another method <NUM> for autonomously maneuvering a tractor <NUM> (e.g., a tow vehicle) in a rearward direction R towards a point of interest, such as a trailer <NUM>, using the system described in <FIG>. At block <NUM>, the method <NUM> includes receiving, at data processing hardware <NUM>, one or more images <NUM> from one or more cameras <NUM> positioned on a back portion of the tractor <NUM> and in communication with the data processing hardware <NUM>. At block <NUM>, the method <NUM> includes receiving, at the data processing hardware <NUM>, a operator planned path <NUM> from a user interface <NUM> in communication with the data processing hardware <NUM>. At block <NUM>, the method <NUM> includes transmitting, from the data processing hardware <NUM> to a drive system <NUM> in communication with the data processing hardware <NUM>, one or more commands causing the tractor <NUM> to autonomously maneuver along the operator planned path <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, an estimated vehicle position Pe based on the operator planned path <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a current vehicle position Pa. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, an error <NUM> based on the estimated vehicle position Pe and the current vehicle position Pa. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, one or more path adjustment commands causing the tractor <NUM> to autonomously maneuver from the current vehicle position Pa to the estimated vehicle position Pe eliminating the error <NUM>. At block <NUM>, the method <NUM> includes transmitting, from the data processing hardware <NUM> to the drive system <NUM>, the one or more path adjustment commands.

In some examples, the method <NUM>, <NUM> includes ing a path on the one or more images <NUM> and receiving a command by way of the user interface <NUM> in communication with the data processing hardware <NUM>. The command includes instructions to adjust the path as the operator planned path <NUM>. In some examples, the command includes instructions to adjust a distance of the path. The command may include instructions to adjust an angle of the path and or instructions to adjust an angle of an end portion of the path.

In some implementations, determining the current vehicle position includes receiving wheel encoder sensor data <NUM> associated with one or more wheels <NUM> and receiving steering angle sensor data <NUM>. The current vehicle position Pa is based on the wheel encoder sensor data <NUM> and the steering angle sensor data <NUM>.

As previously discussed, the proposed algorithm is designed to work in real time in a standard CPU with or without the use of GPU, graphic accelerators, training, or FPGAs. In addition, the proposed approach provides an automated method that only needs initial input from the operator. In addition, the described system provides a compromise between providing guidelines for the operator and automating all rearward functions.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Moreover, subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The terms "data processing apparatus", "computing device" and "computing processor" encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.

In certain circumstances, multi-tasking and parallel processing may be advantageous.

Claim 1:
A method of maneuvering a tractor (<NUM>) in reverse for attachment to a trailer (<NUM>, 200a-c) using a hitch assist system (<NUM>), the method comprising:
entering a hitch assist mode;
displaying a camera image on a user interface (<NUM>), wherein the displayed image shows at least one trailer (<NUM>, 200a-c);
selecting a trailer (<NUM>, 200a-c);
receiving, at the user interface (<NUM>) in communication an indication of a selected trailer (<NUM>, 200a-c);
determining with a computing device (<NUM>) a tractor path (<NUM>) from an initial position (PI) to a final position (PF) adjacent the trailer (<NUM>, 200a-c), the tractor path (<NUM>) comprising maneuvers configured to move the tractor (<NUM>) along the tractor path (<NUM>) from the initial (Pi) position to the final position (PF); and
autonomously following, at a drive system (<NUM>) in communication with the computing device (<NUM>), the tractor path (<NUM>) from the initial position (Pi) to the final position (PF),
wherein displaying on the user interface (<NUM>) further comprises receiving, at a controller (<NUM>), one or more images from one or more cameras (<NUM>) positioned on a back portion of the tractor (<NUM>) and in communication with the controller (<NUM>), and overlaying on the displayed camera image, at the controller (<NUM>), a path representation indicative of an expected path the tractor (<NUM>) drives along, the expected path starting at a tractor hitch (<NUM>),
and wherein selecting the trailer (<NUM>, 200a-c) further comprises receiving, at the controller (<NUM>), a first command by way of the user interface (<NUM>), the first command indicative of a change in the path representation such that the path representation ends at a point of interest, and adjusting, at the controller (<NUM>), the path representation based on the first command and wherein the path planning is completed by the controller (<NUM>) for the hitch assist system (<NUM>).