Tow Ball Position Detection System

A method and system are disclosed for performing a trailer operation for a vehicle having a tow ball. The method includes receiving a first image from a rear camera disposed along a rear portion of the vehicle in a first position; and detecting a pixel position of a representation of the tow ball in the first image. Following the rear camera being moved from the first position to a second position along the rear portion of the vehicle, the method includes receiving a second image from the rear camera. The method also includes detecting a second pixel position of a representation of the tow ball in the second image; and estimating a position of the tow ball relative to the vehicle based on the first and second positions of the rear camera and the detected pixel positions of the representations of the tow ball in the first and second images.

TECHNICAL FIELD

This disclosure relates to a tow ball position detection system of a tow ball supported by a tow vehicle.

BACKGROUND

Trailers are usually unpowered vehicles that are pulled by a powered tow vehicle. A trailer may be a utility trailer, a popup camper, a travel trailer, livestock trailer, flatbed trailer, enclosed car hauler, and boat trailer, among others. The tow vehicle may be a car, a crossover, a truck, a van, a sports-utility-vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to attach to the trailer and pull the trailer. The trailer may be attached to a powered vehicle using a trailer hitch. A receiver hitch mounts on the tow vehicle and connects to the trailer hitch to form a connection. The trailer hitch may be a ball and socket, a fifth wheel and gooseneck, or a trailer jack.

The position of a tow ball (also referred to as hitch ball) is important for several automated trailer assist functions, and accurate knowledge of the tow ball position is the enabler for such functions.

For instance, a trailer hitch assist system uses the information of the tow ball position along with the coupler position to control the vehicle to a suitable hitching position. However, there is no standard size for the tow ball, or the mount used to attach the tow ball to the hitch on the vehicle. Moreover, the tow ball height is also often adjustable. As a result, there is a need for an accurate tow ball position estimation feature for a tow vehicle.

SUMMARY

Example embodiments are directed to a method and system for performing a trailer operation for a vehicle having a tow ball mounted thereto. The method includes receiving, by data processing hardware, a first image from a rear camera disposed along a rear portion of the vehicle in a first position. A pixel position of a representation of the tow ball in the first image is detected by the data processing hardware. Following the rear camera being moved from the first position to a second position along the rear portion of the vehicle, the method includes receiving, by the data processing hardware, a second image from the rear camera, and detecting, by the data processing hardware, a second pixel position of a representation of the tow ball in the second image. The data processing hardware estimates a position of the tow ball relative to the vehicle based on the first and second positions of the rear camera and the detected pixel positions of the representations of the tow ball in the first and second images.

The method may also determine, by the data processing hardware, a change in position of the rear camera in world coordinates between the first position and the second position, wherein the position of the tow ball is estimated based upon the change in position of the rear camera. The change in position of the rear camera may be based in part on a change in angular position, relative to the vehicle, of a tailgate, a liftgate or a trunk of the vehicle to which the rear camera is mounted.

The method may further include, after receiving the first image, sending, by the data processing hardware, an instruction to move the tailgate, liftgate or trunk of the tow vehicle to which the rear camera is mounted from a first component position, corresponding to the first position of the rear camera, to a second component position corresponding to the second position of the rear camera. The instruction to move the tailgate, liftgate or trunk of the tow vehicle may be sent to one of a display to prompt a tow vehicle user to manually move the tailgate, liftgate or trunk, or a component positioning system to move the tailgate, liftgate or trunk without user intervention.

The method may further include determining, by the data processing hardware, a change in pixel coordinates between the pixel position of the tow ball representation in the first image and the pixel position of the tow ball representation in the second image, wherein the position of the tow ball is estimated based upon the change in pixel coordinates.

The method may also include, before receiving the first image, sending an instruction to move the tailgate, liftgate or trunk to which the rear camera is mounted to a first component position, corresponding to the first position of the rear camera.

In one aspect, the method includes receiving a third image from the rear camera while in the first position or the second position, identifying a representation of the tow ball in the third image, and determining a pixel position of the representation of the tow ball in the third image. The method may compare or correlate the determined pixel position of the representation of the tow ball in the third image with the estimated position of the tow ball. Based upon the comparing or correlating, the method determines whether to use the estimated tow ball position in trailer assist functions.

The method may include receiving a third image from the rear camera while in the first position or the second position. Based upon the third image and the rear camera being in the first or second position, the method may determine that a tow ball representation is not in the third image. The method send an instruction to one or more vehicle systems of the vehicle to temporarily disable trailer assist operations.

In some aspects, the second position is a predetermined position having known position coordinates or is a position from which position coordinates are determined.

In an example embodiment, a system for performing a trailer operation for a vehicle having a tow ball mounted thereto includes data processing hardware; and non-transitory memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform a method as described hereinabove.

DETAILED DESCRIPTION

A tow vehicle, such as, but not limited to a car, a crossover, a truck, a van, a sports-utility-vehicle (SUV), and a recreational vehicle (RV) may be configured to tow a trailer. The tow vehicle connects to the trailer by way of a trailer hitch. It may be difficult to driver the vehicle-trailer system in a rearward direction.

Referring toFIGS.1and2, in some implementations, a vehicle-trailer system100includes a tow vehicle102capable of being attached to a trailer106. The tow vehicle102includes a vehicle tow ball104(also referred to as hitch ball) supported by a vehicle hitch bar105. The vehicle tow ball104may be coupled to the trailer106by way of a trailer hitch coupler108supported by a trailer hitch bar109of the trailer106when the vehicle and trailer are hitched. The tow vehicle102may include a drive system110that maneuvers the tow vehicle102across a road surface based on drive commands having x, y, and z components, for example. As shown, the drive system110includes a front right wheel112,112a,a front left wheel112,112b,a rear right wheel112,112c,and a rear left wheel112,112d.The drive system110may include other wheel configurations as well. The drive system110may also include a brake system (not shown) that includes brakes associated with each wheel112,112a-d,and an acceleration system (not shown) that is configured to adjust a speed and direction of the tow vehicle102. In addition, the drive system110may include a suspension system (not shown) that includes tires associates with each wheel112,112a-d,tire air, springs, shock absorbers, and linkages that connect the tow vehicle102to its wheels112,112a-dand allows relative motion between the tow vehicle102and the wheels112,112a-d.

The tow vehicle102may move across the road surface by various combinations of movements relative to three mutually perpendicular axes defined by the tow vehicle102: a transverse axis XV, a fore-aft axis YV, and a central vertical axis ZV. The transverse axis XVextends between a right-side and a left-side of the tow vehicle102. A forward drive direction along the fore-aft axis YVis designated as FV, also referred to as a forward motion. In addition, an aft or rearward drive direction along the fore-aft direction YVis designated as RV, also referred to as rearward motion. In some examples, the tow vehicle102includes a suspension system (not shown), which when adjusted causes the tow vehicle102to tilt about the XVaxis and/or the YVaxis or move along the central vertical axis ZV.

The tow vehicle102may include a user interface120. The user interface120may include a display122, a knob, and a button, which are used as input mechanisms. In some examples, the display122may show the knob and the button, while in other examples, the knob and the button are a knob button combination. In some examples, the user interface120receives one or more driver commands from the driver via one or more input mechanisms or a touch screen display122and/or displays one or more notifications to the driver. The user interface120is in communication with a controller140. In some examples, the display122displays an image133of an environment of the tow vehicle102. The display122may be part of the tow vehicle102or separate therefrom, such as the display of a handheld electronic device.

The tow vehicle102may include a sensor system130to provide reliable and robust driving. The sensor system130may include different types of sensors that may be used separately or with one another to create a perception of the environment of the tow vehicle102and the trailer106that is used for the tow vehicle102to drive and aid the driver in make intelligent decisions based on objects and obstacles detected by the sensor system130. The sensor system130may include the one or more cameras132,132a-dsupported by the vehicle-trailer system100. In some implementations, the tow vehicle102includes a rear vehicle camera132a(i.e., a first camera) that is mounted to provide a view of a rear-driving path for the tow vehicle102, or in other words, the rear vehicle camera132acaptures images133of a rear environment of the tow vehicle102. The rear vehicle camera132amay be positioned on the tailgate of the tow vehicle102.

In some implementation, the rear vehicle camera132amay include a fisheye lens having an ultra-wide-angle lens that produces strong visual distortion intended to create a wide panoramic or hemispherical image. Fisheye cameras capture images having an extremely wide angle of view. Moreover, images captured by the fisheye camera have a characteristic convex non-rectilinear appearance. Other types of cameras may also be used to capture the images133.

The sensor system130may also include other sensors134that detect the vehicle motion, i.e., speed, angular speed, position, etc. The other sensors134may include an inertial measurement unit (IMU) configured to measure the vehicle's linear acceleration (using one or more accelerometers) and rotational rate (using one or more gyroscopes). In some examples, the IMU also determines a heading reference of the tow vehicle102. Therefore, the IMU determines the pitch, roll, and yaw of the tow vehicle102. The other sensors134may also include, but are not limited to, radar, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), ultrasonic, HFL (High Resolution 3D Flash LIDAR), etc. In some implementations, the sensor system130may provide external sensor data received from other systems or vehicles, such as by way of V2X communication or any other communication. Still other sensors130may include at least one position sensor which senses the position of a movable component of the tow vehicle102, as described in greater detail below.

In example embodiments, the rear camera132ais mounted to a movable component of the tow vehicle102. Such movable component of the tow vehicle102is movable relative to the rest of the tow vehicle. In one example, the tow vehicle102is a pickup truck, the movable component is the truck's tailgate160and the rear camera132ais mounted along the tailgate. In another example, the tow vehicle102is a sedan, the movable component is the trunk and the rear camera132ais mounted along the trunk. In yet another embodiment, the tow vehicle is a sport utility vehicle (SUV), the movable component is the liftgate of the SUV and the rear camera132ais mounted along the liftgate. In every example, the movable component may be moved between at least two positions in which each position the rear camera132ais capable of capturing an image having depicted therein the tow ball104.

The controller140includes a computing device (or processor)142(e.g., central processing unit having one or more computing processors) in communication with non-transitory memory144(e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s)142. The controller140may be supported by the tow vehicle102, the trailer106, or both the tow vehicle102and the trailer106. In some examples, the controller140executes an tow ball position detection system150that provides the driver of the tow vehicle102with a panoramic view152of the environment behind the trailer106.

In an example embodiment, the tow vehicle102includes a component positioning system170(FIG.2) which is electronically coupled to the controller140. Component positioning system170selectively positions the above-described movable vehicle component relative to the tow vehicle102. In an embodiment in which the tow vehicle102is a pickup truck, the component positioning system170is operably associated with the tailgate160of the truck for opening and closing the tailgate. In an embodiment in which the tow vehicle102is a sedan, the component positioning system170is operably associated with the trunk of the sedan for opening and closing the trunk. In an embodiment in which the tow vehicle102is a SUV, the component positioning system170is operably associated with the liftgate of the SUV for opening and closing same. The component positioning system170may itself include a motor and controller (not shown) communicatively coupled thereto, for controlling or setting the open position of the movable vehicle component. The component positioning system170is communicatively coupled to the controller140so that the controller140may send instructions to the component positioning system170to selectively position the movable vehicle component. At least one sensor134may also form a position sensor which senses the position of the movable vehicle component (tailgate160, trunk, liftgate, etc.) and provides sensed position data135to the controller140and/or the component positioning system170.

According to example embodiments, the tow vehicle102includes a tow ball position detection system150which estimates and/or determines the position of the tow ball104. The system150detects, estimates or otherwise determines the position of the tow ball104as part of, for example, a calibration sequence. The calibration sequence may be performed, for example, during a power up sequence of operations of the tow vehicle102or at the onset of a trailer assist function or operation. Once detected, the location of the tow ball104may be used as part of a trailer assist function of the tow vehicle102. The tow ball position detection system150may be used with the rear facing camera132awhich is mounted to a movable vehicle component that is movable between at least two positions, with each position capturing an image which includes a representation of the tow ball104. The tow ball position detection system150will be described below for a tow vehicle102having a tailgate160for illustrative purposes, but it is understood that the tow ball position detection system150may be used for any other such movable vehicle portion to which the rear camera132ais mounted. In an example embodiment, the tailgate160may be moved between a first, closed position as shown inFIG.3(i.e., a first component position) and a second, fully open position as shown inFIG.5(i.e., a second component position). In the example embodiment, the rear camera132ais fixedly mounted along the top, center portion of the closed tailgate160.

According to an example embodiment, the tow ball position detection system150estimates the position of the tow ball104using at least two images of the tow ball captured by the rear camera132a,with each image captured by the rear camera132awhile in a different position relative to the tow vehicle102. The tow ball estimation is based upon the change in position of the rear camera, in world coordinates, between a first image and a second image as well as a change in pixel position of a representation of the tow ball104in the first and second images. The position of the tow ball104as estimated in a calibration operation may then be used in a subsequent trailer assist operations.

FIG.8illustrates a method800for determining and/or estimating tow ball position according to at least one example embodiment. It is understood that the sequence of operations may occur in a different order that the order presented herein. Initially, a request is received at802to start a calibration operation. In some aspects, the request is received from the user of the tow vehicle102in response to a prompt provided by the processor140via the display122. It is understood that the request may be received from other systems of the tow vehicle102, such as a startup sequence at the time the tow vehicle is started or turned on. In response to the received request, the tow ball position detection system150begins the calibration sequence at804.

In some example embodiments, the tow ball position detection system150begins the calibration by sending instructions at806to the tow vehicle user via the display122or to the component positioning system170to move the movable tow vehicle component to the first component position. Due to the rear camera132abeing mounted to the movable vehicle component, the movement of the movable tow vehicle component to the first component position corresponds to the rear camera132amoving to a first position. For purposes of this description, the first component position is considered to be the closed position for the tailgate160, trunk or liftgate, but it is understood that the first position may alternatively be an open position of the tailgate160, trunk or liftgate.

In an alternative implementation, the tow ball position detection system150may detect the current position of the movable tow vehicle component, via a position sensor or using object recognition of a captured image, and only send the instructions in the event the movable tow vehicle component is in the open position (in this embodiment in which the closed position is considered the first component position). In still another alternative implementation, the tow ball position detection system150may detect the current position of the movable tow vehicle component and start the calibration with the movable tow vehicle component in the detected current position. In other words, the controller140assigns the detected current position as the first component position of the calibration operation.

The controller140receives at808a first image of the rear environment of the tow vehicle102captured by the rear camera132awhile in the first position. The first image is depicted inFIG.4, including a representation104′ of the tow ball104. The pixel position of the representation104′ of the tow ball104in the first image is determined or otherwise detected at810. The pixel position of the tow ball representation may be performed by one of any of a number of approaches. For example, the controller140may send instructions to the display122to display the first image and prompt the user of the tow vehicle102to select the tow ball representation by touching the portion of the touch screen display122. With the tow ball representation selected by the tow vehicle user, controller140may detect the particular pixel position within the portion of the touch screen display122touched by the user. Alternatively, the tow ball position detection system150may automatically detect the tow ball representation in the first image using object recognition by computer vision, artificial intelligence and/or a neural network(s). One computer vision based approach for detecting the tow ball representation is a Hough transform for detecting circles in the image.

At812, the controller140sends an instruction(s) so that the rear camera132ais moved to a second position. In one embodiment, the instruction is sent to the display122to prompt the tow vehicle user to manually move the movable vehicle component to a second component position corresponding to the rear camera132abeing in the second position. In another embodiment, the instruction is sent to the component positioning system170to do the same. With the first component position being the tailgate160in the closed position, the second component position may be the tailgate160being in the fully open position. In an alternative implementation in which the calibration sequence is performed during operation of the tow vehicle102, the tow ball position detection system150does not provide instructions for moving the movable vehicle component to a component position and instead waits for the movable vehicle component to be moved as initiated by the driver or other tow vehicle user. In this case, the calibration sequence does not continue until the movable vehicle component is moved. In this implementation, the calibration sequence is performed automatically without involving the driver or other tow vehicle user.

A second image is received by the controller140at814that is captured by the rear camera132awhile in the second position. The second image is shown inFIG.6. The pixel position of the representation104′ of the tow ball140in the second image is detected at816using the approaches described above pertaining to block810.

At818, the tow ball position detection system150determines in world coordinates the change in position of the rear camera132afrom the first position to the second position. This determination may be based upon a known, fixed amount of change, such as a tailgate160having moved from the first component (closed) position to the second component (fully open) position. In addition or in the alternative, this determination may be based upon sensor data from a position sensor associated with the movable vehicle component to which the rear camera132ais mounted. This change in the rear camera position, relative to the tow vehicle102, due to the motion of the movable component results in a different viewing angle of the tow ball104, thereby allowing tow ball position estimation using machine vision.

At820, the tow ball position detection system150determines a change in pixel position of the representation104′ of the tow ball104in the first image and the representation104′ in the second image. Based upon the change in position of the rear camera132ain world coordinates and upon the change in position of the representation104′ of the tow ball104′ in pixel coordinates, the tow ball position detection system150determines at822the position of the tow ball104in world coordinates. This determination may utilize triangulation or linear algebra to estimate the position of the tow ball104.FIG.7illustrates triangulation in which the tow ball104is captured in two images and/or image planes by the rear camera132a,each resulting in the tow ball representation104′ being in a different location in the image/image plane.

With the position of the tow ball104determined, the estimated tow ball position may be used by vehicle assist systems of the tow vehicle102. In some embodiments, the calibration sequence for determining the tow ball position in world coordinates concludes at this point.

In some examples, the tow ball position detection system150records the tow ball position in the image at more than two different angles of the movable vehicle component (tailgate16, trunk or liftgate) and uses this additional information to further estimate tow ball position with an increased accuracy.

Further, the tow ball position detection system150verifies that the position of the tow ball104as estimated and/or detected in a previous calibration operation remains valid for use in trailer assist operations or functions. Specifically, the system detects the tow ball representation104′ in an image recently captured by the rear camera132a,compares the previously estimated position for the tow ball104to the representation104′ of the tow ball104in the image, and verifies that the tow ball representation104′ in the recently captured image is in an expected position in the image. This verification is helpful to confirm that the tow ball104and/or the hitch bar105that were subject to a prior calibration were not replaced since the last calibration was performed.

FIG.9illustrates a flowchart900describing the method for verifying a previous calibration sequence remains valid for use in trailer assist operations, according to an example embodiment. At902, the controller140receives a third image recently captured by the rear camera132a,with the rear camera being at a known position, such as with the tailgate160in the closed position or the open position. The pixel position of the representation104′ of the tow ball104is detected in904. The position, in pixel coordinates, of the tow ball representation104′ may utilize any of the approaches described above with respect to block810. The pixel location of the tow ball representation104′ in the third image is processed and correlated at906to the estimate of the position of the tow ball104from the prior calibration sequence. If the correlation is not achieved or otherwise indicates at908that the tow ball representation104′ in the recent image does not correspond to the previously estimated position of the tow ball104, thereby suggesting that a different tow ball is now being used, the tow ball position detection system150requests at910that a new calibration sequence be performed or sends an instruction to the display122to prompt the tow vehicle user to request recalibration.

In blocks any of blocks810and816ofFIG.8or block904ofFIG.9, if the tow ball position detection system150cannot identify the tow ball representation104′ in the corresponding camera image, the tow ball position detection system150assumes that no tow ball is attached to the tow vehicle102. This information may be used to inhibit the activation of certain trailer assist functions such as the trailer hitch assist function.

The current invention automates the calibration process and eliminates or otherwise reduces the chance for human error.