Patent Description:
Trailer hitch assist systems require information about an orientation and a position of the tow hitch mounted on the vehicle. Trailer hitch assist systems may also need information about a position of the tow hitch in relation to a position of a camera on the vehicle.

The tow hitch positions may be measured manually, but this approach can lead to inaccuracies, is costly, and time consuming. This is especially true when the tow coupling is not mounted during the manufacturing of the vehicle or in cases where the tow coupling can be moved manually or electronically on the vehicle. There is a desire to improve the accuracy and simplicity of determining a position of the tow hitch in relation to the vehicle.

According to one aspect of the invention, a method for determining a position of a tow ball in an image is provided. The method includes obtaining at least one real-time image from a vehicle. The at least one real-time image is processed with a controller on the vehicle to obtain a feature patch (f) describing at least one real-time image. A convolution is performed of the feature patch (f) and each filter (h) from a set of filters (H) with the filter (h) being based on data representative of known tow hitches fixed to the vehicle. A location of a tow ball on the tow hitch is identified in the at least one real-time image is based on the convolution between the feature patch (f) and each filter (h) from the set of filters (H).

In a further embodiment of any of the above, the set of filters (H) is stored in memory of the controller on the vehicle.

In a further embodiment of any of the above, the set of filters (H) includes an optimization of at least one image of multiple known tow hitches fixed to the vehicle.

In a further embodiment of any of the above, the at least one real time image includes consecutive real-time images. Reducing a search area of the consecutive real-time images includes comparing the consecutive real-time images to identify areas with objects that did not move between the consecutive images and searching the areas with objects that did not move.

In a further embodiment of any of the above, the convolution is only performed on the area that did not move.

In a further embodiment of any of the above, the at least one real-time image is captured by a camera on a rear of the vehicle.

In a further embodiment of any of the above, a location of the tow ball identified on a display by a user is received.

In a further embodiment of any of the above, the set of filters (H) is developed based on an algorithm evaluating extracted features from a database of rear-view images of the vehicle.

In a further embodiment of any of the above, the extracted features include at least one of a histogram of oriented gradients ("HOG"), Color Names, RGB, or grayscale.

In a further embodiment of any of the above, the set of filters (H) is developed based on an algorithm that identifies learned features from a database of rear-view images of the vehicle.

In a further embodiment of any of the above, identifying the location of the tow ball includes utilizing an algorithm on the controller that identifies a location of highest probability of the location of the tow ball through a voting process.

In a further embodiment of any of the above, the voting process identifies at least one pixel in the real-time image from the vehicle with the greatest correlation to the location of the tow ball from the set of filters (H).

According to another aspect of the invention, a tow ball identification assembly is provided which includes a camera for capturing images of a tow ball. A controller is configured to obtain at least one real-time image from a vehicle and processing the at least one real-time image with a controller on the vehicle to obtain a feature patch (f) describing the at least one real-time image. A convolution is performed of the feature patch (f) and a set of filters (H) with the filter (h) being based on data representative of known tow hitches fixed relative to the vehicle. A location of a tow ball on the tow hitch is identified in the at least one real-time image is based on the convolution between the feature patch (f) and each filter (h) from the set of filters (H).

In a further embodiment of any of the above, the set of filters (H) is performed by a computing device having a greater computing power than a computing power of the controller on the vehicle.

In a further embodiment of any of the above, the set of filters (H) is stored in memory of the controller on the vehicle. The set of filters (H) includes an optimization of multiple images of known tow hitches fixed to the vehicle and performing the convolution between the feature patch (f) with each filter (h) from the set of filters (H), which produces a probability of the tow ball position in the image.

In a further embodiment of any of the above, the set of filters (H) is developed based on an algorithm evaluating extracted features from a database of rear-view images of the vehicle.

In a further embodiment of any of the above, wherein identifying the location of the tow ball includes utilizing an algorithm on the controller that identifies a location of highest probability of the location of the tow ball through a voting process.

In a further embodiment of any of the above, wherein the voting process identifies at least one pixel in the real-time image from the vehicle with the greatest correlation to the location of the tow ball from the set of filters (H).

It is common today for vehicles <NUM> to include a tow hitch <NUM> that is fixed relative to the vehicle <NUM> by a receiver <NUM>. In the illustrated example shown in <FIG>, the tow hitch <NUM> includes a ball mount <NUM> for supporting a ball <NUM> adjacent a first end and attaching to the receiver <NUM> adjacent a second end. The ball mount <NUM> is removeable attached to the receiver <NUM> and not be retained by a pin. The ball <NUM> on the tow hitch <NUM> is used to form a connection with a trailer (not shown) to allow the trailer to rotate around the ball <NUM> while maneuvering the vehicle <NUM>, particularly when turning or reversing the vehicle <NUM> when attached to the trailer.

The vehicle <NUM> also includes a camera <NUM> for obtaining real-time images <NUM> (See <FIG>) of the rear of the vehicle <NUM> including the tow hitch <NUM>. The images can be projected on a display to users for assisting in viewing a rear of the vehicle <NUM> and a surrounding area. A controller <NUM> is located on the vehicle <NUM> and is in electrical communication with the camera <NUM> to capture, store, and/or process images from the point of view of the camera <NUM>. The controller <NUM> includes a processor in electrical communication with memory for performing these steps and the ones outlined further below. The controller <NUM> can also include further inputs and outputs to for communicating with other parts of the vehicle <NUM>.

In the illustrated example, the vehicle <NUM> is located on a flat ground segment <NUM>, such that a top of the ball <NUM> is located a height H1 above the ground segment <NUM>. The tow hitch <NUM> also includes a drop in the ball mount <NUM>. The drop is defined by a height H2 between un upper surface of the ball mount <NUM> adjacent the receiver and an upper surface of the ball mount <NUM> adjacent the ball <NUM>. The drop of the tow hitch <NUM> contributes to the change in height H1 of the ball <NUM> above the ground segment <NUM>. While the illustrated example shows the drop H1 positioning the ball <NUM> closer to the ground segment <NUM> relative to the receiver <NUM>, there may be applications where the drop H1 positions the ball <NUM> further away from the ground segment <NUM> to accommodate different trailer attachment heights.

<FIG> illustrates an example method <NUM> for determining a position of the tow ball <NUM> in the image <NUM> in real time. A portion <NUM> of the method <NUM> is performed offline or remote from the vehicle <NUM> and stored in memory on the controller <NUM> for use by the controller <NUM> to determine the location of the tow ball <NUM> in real time. In the example method <NUM>, the portion <NUM> is performed on a higher-powered computing device <NUM>, while the controller <NUM> performs the remaining steps with a microprocessor onboard the controller <NUM>. The controller <NUM> differs from the computing device <NUM> in that the controller <NUM> does not possess as high of a level of computing power as the computing device <NUM>. For example, the onboard controller may be a single Central Processing Unit (CPU) and does not need to any hardware accelerators such as a Vision Processing Unit (VPU), or a Graphics Processing Unit (GPU). As shown in <FIG>, the computing device <NUM> includes memory <NUM> for storing data, a microprocessor <NUM>, and an input/output <NUM>.

The portion <NUM> requires the higher-powered computing device <NUM> to perform the steps <NUM> and <NUM> because the controller <NUM> is not capable of performing the steps <NUM> and <NUM> on a real-time basis that would be helpful to a user of the vehicle <NUM>. However, as will be described in greater detail below, the controller <NUM> is nevertheless able to identify the location of the tow ball <NUM> in the real-time images <NUM> because the controller <NUM> can access the data calculated by the higher-powered computing device <NUM> stored in the memory of the controller <NUM>.

To perform the portion <NUM> of the method <NUM>, the high-powered computing device <NUM> obtains access to a database <NUM> (<FIG>) of rear-view images <NUM>. The database <NUM> includes images from the rear-view camera <NUM> of the vehicle <NUM> with known tow hitches <NUM>-D (step <NUM>). The information included in the database <NUM> about the known tow hitches <NUM>-D can include the position (in pixels) on the image <NUM>, drop, ball diameter, and distance of the tow ball <NUM> from the receiver <NUM>. Additionally, for each known tow hitch <NUM>-D, the database <NUM> may include a single rear-view image <NUM> or a series of sequential rear-view images <NUM> that assist in identifying elements that are fixed locations between images <NUM>.

For each image in the rear-view images <NUM> from the database <NUM>, the computing device <NUM> performs an optimization to obtain the best filter (h) to identify the tow ball <NUM> in the rear-view image <NUM> of the vehicle <NUM> (step <NUM>). The domain of the filter (h) is given in pixels, whose size, width and height, is given by the size of the training region. When different filters (h) are grouped together, it is referred as a set of filters (H). The optimization is performed through an algorithm. In one example, the algorithm extracts a feature patch (f), such as a histogram of oriented gradients ("HOG"), Color Names, RGB, and/or grayscale, for each image <NUM> in the database <NUM> to characterize the known tow hitches <NUM>-D. The algorithm performs the optimization by looking at the extracted feature patch (f) in a training region of the image <NUM> and the given tow ball location information (ground truth) stored in the database <NUM>. The training region of the image <NUM> includes a given height and width in pixels. The algorithm outputs the learned filter (h) for each rear-view images <NUM> from the database <NUM>.

In another example of (step <NUM>), the computing device <NUM> can use another algorithm to identify its own learned features (f) to optimize and obtain the best filter (h) from the rear-view images <NUM>. The algorithm under this example will identify the learned features (f) by examining different scenarios displaying the tow ball <NUM> in the images <NUM> instead of looking at the specific extracted features identified above in the other example algorithm. Given an initial condition of the extracted features (f), the algorithm gets the filter (h) (as shown in the previous example). Next, the algorithm fixes the filter (h) found in the previous step to optimize and obtain a new feature patch (f). This process is repeated until the change on the feature patch (f) and/or the filter (h) is under a predetermined threshold or a maximum number of iterations is achieved. In this example, note that step <NUM> must use its own learned features (f).

At step <NUM>, the controller <NUM> receives the image <NUM> in real time from the camera <NUM> of a rear-view of the vehicle <NUM>. Once the controller <NUM> has received at least one real-time image <NUM> from the rear-view camera <NUM>, the controller <NUM> can pre-process the image <NUM> to extract features (f) describing the image <NUM> (step <NUM>). Some examples of a feature patch (f) may be a histogram of oriented gradients ("HOG"), Color Names, RGB, and/or grayscale The image <NUM> can be divided into patches or small segments for the controller <NUM> to identify the most relevant patches or segments to search in the image <NUM> (step <NUM>). The controller <NUM> may identify the most relevant patches to search by identifying objects that did not move between consecutive rear-view images <NUM> from the camera <NUM>. Identifying areas where objects did not move between consecutive areas helps to reduce the search area to identify the location of the tow ball <NUM> because the tow ball <NUM> does not move between consecutive images relative to the camera <NUM> while the surrounding environment will if the vehicle <NUM> is moving.

The controller <NUM> then performs a 2D convolution between the extracted feature patch (f) (from step <NUM>) and for each filter(h) (from step <NUM>). The last process is done for each image in the database <NUM> and each patch the controller <NUM> generated from the rear-view image <NUM> (step <NUM>). A 1D convolution is a mathematical operation on two functions (f and h) that produces a third function (f * h) that expresses how the shape of one is modified by the other. The term convolution refers to both the result function and to the process of computing it. It is defined as the integral of the product of the two functions after one is reversed and shifted. And the integral is evaluated for all values of shift, producing the convolution function. A 2D convolution is just an extension of 1D convolution by convolving both horizontal and vertical directions in <NUM>-dimensional spatial domain.

The controller <NUM> utilizes an algorithm with a voting process that identifies the position of high probability corresponding to the location of the tow ball <NUM> in the rear-view image <NUM>. The voting process identifies a region of pixels or a single pixel in the image <NUM> with the greatest correlation to the location of the tow ball using each filter (h) learned in step <NUM>. One advantage of performing the convolution to each patch, is that the algorithm can be run in real time on the controller <NUM> without the need for a high-end processor, like with the computing device <NUM>.

The method <NUM> may also incorporate a calibration process to aid in selecting the correct position of the tow ball <NUM>. In the calibration process, a user may select a position of the tow ball on a display <NUM> (<FIG>) projecting the rear-view image <NUM>. The display <NUM> may be a touch screen display to allow the user to select the location of the tow ball <NUM> through touching the portion of the display <NUM> showing the tow ball <NUM>. The controller <NUM> can then perform the steps above to verify the location of the tow ball <NUM>.

Once the controller <NUM> has identified the location of the tow ball <NUM> by performing the steps of the method <NUM> above, the location of the tow ball <NUM> can be presented on the display <NUM> or used by the controller <NUM>.

Although the different non-limiting examples are illustrated as having specific components, the examples of this disclosure are not limited to those particular combinations. It is possible to use some of the components or features from any of the non-limiting examples in combination with features or components from any of the other non-limiting examples.

Claim 1:
A computer-implemented method (<NUM>) for determining a position of a tow ball (<NUM>) in an image (<NUM>), the method (<NUM>) comprising:
obtaining at least one real-time image (<NUM>) from a vehicle (<NUM>);
processing the at least one real-time image (<NUM>) with a controller (<NUM>) on the vehicle (<NUM>) to obtain a feature patch (f) describing at least one real-time image (<NUM>);
performing a convolution of the feature patch (f) and each filter (h) from a set of filters (H) with the filter (h) being based on data representative of known tow hitches (<NUM>-D) fixed to the vehicle (<NUM>); and
identifying a location of a tow ball (<NUM>) on the tow hitch (<NUM>) in the at least one real-time image (<NUM>) is based on the convolution between the feature patch (f) and each filter (h) from the set of filters (H).