Patent ID: 12258065

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings and the illustrative embodiments depicted therein, a vehicle10includes a trailer maneuvering or guidance system12that is operable to guide during backing up or reversing of the vehicle with a hitched trailer16. The trailer16is hitched at the vehicle via, for example, a hitch14(such as via a trailer coupler at the end of a trailer tongue16ahitching to the hitch ball of the hitch of the vehicle to establish a pivoting joint connection of the trailer16to the vehicle10). The trailer guidance system12may guide and maneuver the vehicle10and trailer16toward a desired or selected location.

In the illustrated embodiment, the trailer guidance system12includes at least one exterior viewing vehicle-based imaging sensor or camera, such as a rearward viewing imaging sensor or camera or rear backup camera18(and the system may optionally include multiple exterior viewing imaging sensors or cameras, such as a sideward/rearward viewing camera at respective sides of the vehicle and/or trailer), which captures image data representative of the scene exterior and rearward of the vehicle10, which includes the hitch14and/or trailer16, with the rear backup camera18having a lens for focusing images at or onto an imaging array or imaging plane or imager of the camera (FIG.1). The hitch14comprise a trailer ball hitch at which the trailer tongue16aof the trailer16hitches to form a pivoting joint attaching the trailer16to the vehicle10. The trailer maneuver guidance system12includes a control or electronic control unit (ECU)19comprises electronic circuitry, including memory chips, at least one data processor, and associated software. The electronic circuitry includes a processor that is operable to process image data captured by the camera or cameras and may detect objects or the like and/or provide displayed images at a display device for viewing by the driver of the vehicle (the control and/or display device may be part of or incorporated in or at an interior rearview mirror assembly of the vehicle, or the control and/or the display device may be disposed elsewhere at or in the vehicle). The data transfer or signal communication from the camera to the ECU may comprise any suitable data or communication link, such as a vehicle network bus or the like of the equipped vehicle.

The trailer guidance system12receives an input from a user (e.g., the driver of the vehicle) that indicates a desired direction or trailer angle for the trailer16hitched to the vehicle10. Based on the received driver input, the system controls steering of the towing vehicle (e.g., moves the steering wheel) to direct the vehicle and the trailer hitched to the vehicle toward an orientation where the current trailer angle relative to the vehicle (as determined via processing of image data captured by the rearward viewing camera or rear backup camera or determined via other trailer angle detection systems) is at the desired or selected trailer angle. For example, the trailer guidance system calculates a current angle between the centerline longitudinal axis20of the vehicle and centerline longitudinal axis22of the trailer16(at least of the trailer tongue16aof the trailer16) using the image data from the rearward viewing camera18and/or a truck bed camera and/or one or more side cameras. The trailer guidance system calculates a direction of the steering wheel and amount of steering wheel rotations required to convert the current trailer angle to the desired trailer angle. The system supports all types of conventional trailers including vehicle haulers, box trailers, utility trailers, loaded and unloaded boat trailers, snow mobiles, and any other custom trailers, fifth wheels, or goose neck types of trailers. The system performs independent of ambient conditions (e.g., day, night, sun, cloud, rain, snow) and environmental surfaces (e.g., concrete, asphalt, gravel, grass, dirt, etc.). In some examples, the system displays information pertaining to the current trailer angle of the trailer relative to the vehicle and the desired trailer angle of the trailer relative to the vehicle, and the jackknife and collision angles of the trailer16hitched to the towing vehicle10.

Referring now toFIG.2, the user or driver input200(as part of a human machine interface or HMI of the vehicle), which is actuated or used by a user or driver in the vehicle, may transmit a user input indication201to a user input encoder210. The encoder210converts the user input indication201into a desired trailer angle (DTA)212that represents the trailer angle desired by the user. The image data202captured by one or more cameras18is transmitted to a trailer angle detection module220. The trailer angle detection module220processes the provided image data202to determine a current trailer angle (CTA)222, which represents the angle between the centerline axis of the vehicle20and the centerline axis of the trailer22.

A gap finder module230receives the DTA212from the encoder210and the CTA222from the trailer angle detection module. The gap finder module may also receive a current steering angle (CSA)242from an electronic power steering (EPS) module240of the vehicle. Using the DTA212, the CTA222, and the CSA242, the gap finder module230determines a target steering angle (TSA)232that represents an angle the steering wheel may be set to that will direct the vehicle and trailer in a manner to achieve the DTA212. The gap finder module230may transmit the TSA to the EPS240to automatically control steering of the steering wheel to the TSA232.

Referring now toFIG.3, in some examples, the gap finder module230determines a target steering angle232using Equation (1), where @ represents the front wheel angle, a represent a distance between the rear axle of the vehicle and the hitch, b represents a beam length of the trailer (which may be entered via a user input or which may be estimated or determined via a beam length estimation process or system, such as by utilizing aspects of the systems described in U.S. provisional applications, Ser. No. 63/199,155, filed Dec. 10, 2020, and/or Ser. No. 62/705,968, filed Jul. 24, 2020, which are hereby incorporated herein by reference in their entireties), γ represents the CTA222, γdesrepresents the DTA212, L represents the vehicle wheelbase, and u represents a velocity of the vehicle.

φ=a⁢tan⁢(Lu+u⁢ab⁢cos⁢(γ-yd⁢e⁢s)[ub⁢sin⁢(γ-γd⁢e⁢s)-K⁡(γ-γd⁢e⁢s)])(1)

The gap finder module230may convert the front wheel angle (φ) determined from Equation (1) into the TSA232for the EPS240.

Optionally, and such as shown inFIGS.4A-4C, the user input200may comprise a slider input or slider200a. The slider200aincludes an indicator302that indicates a current value of the slider200a. The current value of the slider may represent the trailer angle desired by the user. The slider200aalso includes a plurality of bins or demarcations304that indicate different values the indicator302may be set to. That is, the indicator302may be positioned at any of the demarcations304to generate the user input indication201with the value of the demarcation304that the indicator302is set to.

The slider200amay include a minimum value and a maximum value. For example, the slider200amay have a minimum value at the left-most demarcation304and a maximum value at a right-most demarcation304. The minimum value and the maximum value may correspond with a minimum trailer angle and a maximum trailer angle relative to the vehicle that may be selected. In some examples, the minimum trailer angle may be −90 degrees while the maximum trailer angle may be +90 degrees. The slider200amay have a center demarcation304that indicates or corresponds to a trailer angle of 0 degrees relative to the vehicle.FIG.4Aillustrates the indicator302at the center demarcation304for a desired trailer angle of 0 degrees, whileFIG.4Billustrates the indicator302at the maximum trailer angle (i.e., the right-most demarcation304) of +90 degrees andFIG.4Cillustrates the indicator302at the minimum trailer angle of −90 degrees (i.e., the left-most demarcation304).

Thus, in some examples, the user may adjust the position of the indicator302of the slider200ato one of the demarcations304that corresponds with the trailer angle desired by the user. When the user desires a trailer angle of 0 (i.e., the vehicle's centerline axis20is aligned with the trailer's centerline axis22), the user may position the indicator302at the center demarcation304. When the user desires a maximum trailer angle (e.g., +90 degrees), which corresponds with the trailer positioned towards the passenger side of the vehicle (which in a left-hand drive vehicle, such as used in the United States, is at the right-hand side of the vehicle), the user may position the indicator302at the right-most demarcation304. Similarly, when the user desires a minimum trailer angle (e.g., −90 degrees), which corresponds with the trailer positioned towards the driver side of the vehicle, the user may position the indicator302at the left-most demarcation304. That is, the direction of the indicator with relation to the slider200amay be aligned with the direction of the trailer in the vehicle's coordinate system (i.e., when the indicator302is positioned to the right of the center point, the user desires the trailer to move to the right of the vehicle, and when the indicator302is positioned to the left of the center point, the user desires the trailer to move to the left of the vehicle). The user may position the indicator302at any demarcation304in-between the left-most demarcation304and the right-most demarcation304to set a desired trailer angle between the minimum trailer angle and the maximum trailer angle. In some examples, indicator positions to the right of the center demarcation represent positive trailer angles while indicator positions to the left of the center demarcation represent negative trailer angles.

The slider200amay include any number of demarcations304. In some examples, a number of demarcations (N) is based on a resolution (R) of the slider200a, such that R=1/N. Each demarcation304may represent a regular interval of a desired trailer angle. For example, each demarcation304may represent an increase or decrease of 10 degrees from the previous demarcation304. That is, if the left-most demarcation304represents a desired trailer angle of −90 degrees, when the interval is 10 degrees, the demarcation to the right of the left-most demarcation may represent a desired trailer angle of −80 degrees. The indicator need not (and desirably does not) display the trailer angles represented by the demarcations, as the driver, when selecting a demarcation, may only initially approximate a desired angle between zero degrees and 90 degrees or between zero and negative 90 degrees, without knowing the actual angle being selected. Although for illustrative purposes a scale in terms of angles is discussed above, a driver utilizing the present system need not know (and often does not care to know) particular angles of the trailer relative to the vehicle at any given moment of time. Thus, whatever scale provided to the driver is merely indicative of degree of turn to the right or degree of turn to the left and could, for instance, be a numerical scale of 0+/−10 or could be a colored scale with the intensity of color increasing or decreasing as the driver respectively slides from the center to, for example, the right, with, for example, the rightmost sided position being the darkest color and the positions closer to the central point be lighter color or semi-transparent.

Referring now toFIGS.5-7, the trailer angle detection module220measures or determines (e.g., using the rearward viewing camera(s)18) the current trailer angle222of the trailer (i.e., the angle of the trailer's centerline axis22relative the vehicle's centerline axis20). The trailer angle detection module220may determine the trailer angle is zero degrees when the centerline axis of the trailer22is aligned with the centerline axis of the vehicle20(FIG.5). The trailer angle detection module220may determine that the current trailer angle222is positive when the trailer is at the passenger side or the right side of the vehicle (respective to the centerline axis20of the vehicle) (FIG.6) and the trailer angle detection module220may determine that the current trailer angle222is negative when the trailer is at the driver side or the left side of the vehicle (respective to the centerline axis20of the vehicle) (FIG.7).

Optionally, and such as shown inFIG.8, the system provides visual feedback of the current trailer angle222, the desired trailer angle212(determined from the user input200) and other threshold angles, such as jackknife angles and collision angles. For example, a display24(FIG.1) disposed within the vehicle may display images to the driver of the vehicle. In some examples, images captured by one or more cameras18(e.g., a fisheye camera) that include at least a portion of the trailer are transformed into a cylinder or cylindrical view700. A cylindrical view represents images that have been projected onto a surface of a cylinder and then the cylinder has been unrolled to form a flat plane. In this example, rotation of the trailer around the hitch (e.g., increasing and decreasing the current trailer angle222) results in a lateral movement of the trailer in the cylindrical view. For example, as the current trailer angle222increases, the trailer may appear to move laterally to the right in the displayed images, while conversely when the current trailer angle222decreases, the trailer may appear to move laterally to the left in the displayed images. In some examples, the cylindrical view corresponds to 180 degree field of view behind the vehicle which corresponds to a maximum trailer angle of 90 degrees and a minimum trailer angle of −90 degrees.

In the cylindrical view700(and such as can be seen with reference toFIGS.9A and9B), each column of pixels in the display may correspond to a vertical plan representative of a selected trailer angle. For example, when the display has a resolution of 1280×800 pixels, the display has 1280 columns of pixels. In this example, column 0 (i.e., the left-most column of pixels in the display) may correspond to a trailer angle of −90 degrees. That is, when the center of the trailer displayed in the cylindrical view is aligned with column 0, the current trailer angle is −90 degrees. Furthering this example, a vertical line at column319(i.e., the 320th column from the left) of pixels (FIG.9B) may correspond to a trailer angle of −45 degrees (FIG.9A), column639may correspond to a trailer angle of 0 degrees, column959may correspond to a trailer angle 45 degrees, and column1279(i.e., the right-most column) may correspond to a trailer angle of 90 degrees. Thus, each column of pixels in the displayed cylindrical view700may be directly converted to a trailer angle.

Referring back toFIG.8, the cylindrical view700may include a current trailer angle indicator710. The current trailer angle indicator710may be a vertical bar that aligns with the centerline axis22of the trailer. The current trailer angle indicator710visually indicates to the driver the current trailer angle222relative to the vehicle. The cylindrical view may also include the selected or desired trailer angle indicator720that updates to reflect the current desired trailer angle212derived from the user input200. For example, when the user slides the indicator302of the slider200ato the left, the desired trailer angle indicator720may correspondingly move to the left on the displayed cylindrical view700. In some examples, the display may provide numerical indicators of the current trailer angle and the desired trailer angle. For example, as illustrated inFIG.8, the display may indicate that the current trailer angle is 0 degrees and the desired trailer angle is −25 degrees (but such angle values may not be displayed as they would not be readily understood by the driver and would not help the driver is selecting the desired angle for the particular reversing maneuver).

Preferably, the human machine interface (HMI) that enables the driver to laterally adjust the indicator720has a form and function economical for the automaker to install and provide in the vehicle and convenient and easy to be used by the driver of the vehicle when trailering. The HMI may comprise a slider or laterally slidable element (where the driver laterally slides a sliding element or slides or moves his or her finger across a touch screen), or may comprise buttons or inputs (seeFIG.12), where the user may press a right-hand button to increase the desired trailer angle to the right or may press a left-had button to increase the desired trailer angle to the left (e.g., to set the desired trailer angle to a larger negative value). Such a button (or touch input) HMI may include three buttons, with a middle button provided to allow the driver to reset the desired trailer angle to zero degrees at any time. The driver thus presses or touches the desired input (left, middle, right) to cause the displayed indicator720to move at the display screen. Optionally, the rate of speed at which indicator moves (responsive to pressing or touching the left button or input or the right button or input) is pressure dependent (i.e., the harder the button/input is pressed, the faster the displayed indicator moves in that direction). Optionally, such a three-button HMI, for use during trailering, can be used when not trailering to control the exterior mirror element respectively present in a driver-side/left-side exterior mirror assembly and a passenger-side/right-side exterior mirror assembly present on the equipped vehicle.

Optionally, although discussed above in terms of a laterally movable (left to right or right to left) slider being provided as the HMI, other HMIs may be used, including rotary HMIs. Such a rotary HMI (seeFIG.13) provides a rotatable user input that the driver rotates clockwise or counter-clockwise to adjust the desired trailer angle, with the degree of rotation corresponding to the gradations on the dial or the like. With such a rotary HMI, the input may have a desired trailer angle of zero degrees when the rotatable input points upward, and may have a desired trailer angle of +90 degrees when the rotatable input is rotated 90 degrees clockwise (or 180 degrees clockwise as shown inFIG.13) and a desired trailer angle of −90 degrees when the rotatable input is rotated 90 degrees counter-clockwise (or 180 degrees counter-clockwise).

Note that actuation of the user input (such as lateral movement of the slider or pressing at a touch input or button or rotating a rotatable user input) towards the right or towards the left can continuously adjust the desired trailer angle. Alternatively, the user input, when actuated, can adjust the indicator (and selected or desired trailer angle) in predefined or preselected steps, with such steps corresponding to the demarcation indicators. The HMI may be executed or operated as a physical thing separate from the display screen or touch screen (such as a separate electro-mechanical slider or buttons or rotatable input), or may be incorporated as part of the display screen/touch screen (such as a slider or buttons that are actuated via touching at appropriate areas of the touch screen, with the slider or buttons displayed at the touch screen at those areas). For example, the HMI may be located at the instrument panel of the vehicle, at a center console of the vehicle, at the steering wheel of the vehicle or the like. Optionally, the button or toggle user input may be combined with an exterior mirror control. Optionally, a voice activated HMI may be provided (whereby the system may set the desired trailer angle responsive to detection and recognition of a voice input received at one or more microphones in the vehicle), or a gesture detection system may be provided to provide gesture control (whereby the system may set the desired trailer angle responsive to detection and recognition of a gesture responsive to processing of image data captured by one or more cameras in the vehicle).

Use of a lateral user input element that is laterally movable left and right is advantageous in that, for example, lateral movement by the driver of the user input towards the right side of the vehicle will cause the trailer to turn towards that side. In a conventional vehicle, a driver must turn his or her steering wheel in a direction opposite to the direction or location toward which the trailer is to be maneuvered. Thus, the laterally adjustable user input element provides a more readily understood means for steering the vehicle and trailer so that the current trailer angle approaches the desired or selected or set trailer angle.

When the driver actuates a button or other user actuatable device to indicate to the system that the driver is entering a trailering mode to initiate a trailering maneuver, the power steering system of the vehicle then controls steering of the vehicle. Optionally, when the trailering maneuver is initiated, the steering wheel (that the driver can turn clockwise or counter-clockwise) can then function as the user input or HMI for setting the desired trailer angle. For example, when the trailering maneuver is initiated, the steering wheel functions in a steering wheel-trailering mode, whereby the driver turning the steering wheel clockwise selects or sets a desired trailer angle to the right and causes the power steering system of the vehicle to turn the vehicle (by turning the front wheels to the left) such that the trailer hitched to the vehicle is guided to the right as the vehicle and trailer move rearward. Similarly, when the driver turns the steering wheel counter-clockwise, the trailer is caused to travel to the left of the vehicle.

When the driver of the vehicle initiates a trailering maneuver (such as by pressing a button or actuating a trailering initiation input), and when reverse gear is first engaged by the driver putting the transmission of the equipped vehicle into reverse, the rear backup camera captures image data and the video display screen displays real-time video images for the driver to view during the reversing maneuver (such as utilizing aspects of rear backup cameras and displays described in U.S. Pat. No. 10,336,255, which is hereby incorporated herein by reference in its entirety). Display of the trailer guidance indicators superimposed over the displayed real-time video images occurs only after a delay period has elapsed following shifting into reverse gear. During the delay period (commencing when the vehicle transmission is shifted into reverse gear), only video images captured by the rear back up camera are displayed by the video display screen, and the displayed video images are not obscured by any additional trailering indicating overlays.

The displayed images may include an indication of a jackknife zone or collision zone730. The jackknife zone and collision zones are trailer angles where the trailer is in danger of jackknifing or colliding with the towing vehicle respectively. As illustrated inFIG.8, these zones730may be highlighted (e.g., overlain with semi-transparent color layer) to indicate whether a current trailer angle222or a desired trailer angle720may cause a jackknife or a collision between the vehicle and the trailer.

In some examples, other views may be displayed by the display24to provide visual feedback of the current trailer angle222, the desired trailer angle212, and the jackknife/collision threshold angles. For example, a virtual elevated view (e.g., a bird's-eye view) may instead be displayed on the display24. The virtual elevated view may display the vehicle10, the trailer16at the current trailer angle222, and the immediate surroundings of the vehicle. The display24, in some examples, may display a model of the trailer16based on the current desired trailer angle212. That is, the display may display an avatar or representation of the trailer at desired trailer angle to provide visual feedback of the desired trailer angle212. As the user adjusts the user input200(e.g., moves the indicator302of the slider200a), the avatar may similarly adjust positions around the vehicle10to represent the adjusted desired trailer angle212.

Referring now toFIGS.10A and10B, a working example of the system12is illustrated. In this example, the current trailer angle222is indicated to be 0 degrees by the current trailer angle indicator710and the user desires to turn the trailer to the left of the vehicle (e.g., into a driveway at the left side of the road at which the vehicle and trailer are located). The user, using the user input200(e.g., the slider200a), indicates a desired trailer angle by sliding the indicator302to the left of the center point of the slider200a. In this example, the selected desired trailer angle212corresponds to negative 25 degrees, which may be indicated by the desired trailer angle indicator720(FIG.10A). That is, as the user actuates or adjusts the user input200, the display displays the current trailer angle indicator710reflective of the current trailer angle222and the desired trailer angle indicator720reflective of the desired trailer angle212. Based on the target steering angle232determined from the CTA222, the DTA212, and the current steering angle242(e.g., by the gap finder230), the EPS240may turn the steering wheel to the determined steering wheel angle232.

At this point, the driver may manually control the velocity of the vehicle (e.g., by controlling the accelerator pedal and the brake pedal) to begin moving the vehicle. As the vehicle moves, the current trailer angle222will change based on the current steering wheel angle (i.e., the target steering angle232). As the current trailer angle222changes, the current trailer angle indicator710displayed on the display will correspondingly change. As the vehicle continues to move, the current trailer angle222will approach the desired trailer angle until the current trailer angle222is equivalent to or aligned with the desired trailer angle212(FIG.10B). Once the current trailer angle is equal to the desired trailer angle212(FIG.10B), the system12may continually update the target steering angle232to the EPS240as the vehicle moves to maintain the CTA222at or near the DTA212. The system12may maintain the CTA222at or near the DTA212until the DTA212is changed via the user input200(e.g., moves the indicator302) or when the system is disabled or deactivated (e.g., via another user input, turning off the vehicle, etc.).

To continue the example, once the desired trailer angle121is equal to the current trailer angle222, the driver may continue to move the vehicle and the system will control the steering of the vehicle to maintain the current trailer angle. Once the turn of the trailer is almost complete (i.e., the trailer has turned into the driveway, but the vehicle is not yet aligned with the trailer), the driver may adjust the desired trailer angle to 0 degrees and continue to move the vehicle until the trailer straightens out behind the vehicle. Thus, the system12allows the driver to continuously update the desired trailer angle212via the user input200and the system will, in response, continuously update the target steering angle232to move the current trailer angle222toward the desired trailer angle212. The driver may update the desired trailer angle212at any point during the reversing maneuver while stopped or moving. For example, the driver may initially select a relatively sharp trailer angle and then, as the vehicle and trailer are moved rearward toward the sharper trailer angle, the driver may slide the input toward a less sharp trailer angle and eventually toward a zero degree trailer angle to back up the trailer to the targeted or selected destination. The driver may control how sharp the trailer turns by adjusting the desired trailer angle (e.g., a value near zero will result in a gentle turn while a value near −90 or +90 will result in a sharp turn). Throughout the backing up of the vehicle and trailer, the driver maintains control of the speed of the vehicle (i.e., the driver controls the acceleration and braking of the vehicle).

Referring now toFIGS.11A and11B, the system12does not restrict the user from setting the DTA212(e.g., via the user input200) to an angle within the jackknife and/or collision zones730. The system may instead alert the user when the desired trailer angle or DTA212is selected to be within a jackknife and/or collision zone730. For example, the system12may provide an intermittent audio tone (e.g., a beeping) and/or provide a warning on the display24. Alternatively, or additionally, the driver can be visually alerted when the desired trailer angle or DTA212is selected to be within a jackknife and/or collision zone730. For example, an electronically generated graphic overlay can be displayed superimposed over the real-time video images of the rearward scene being displayed on the screen of the video monitor that is viewable by the driver of the vehicle when trailering. As a further alternative and/or addition, a tactile or haptic alert (such as a vibrating seat element or the like) may be provided when the DTA212is selected to be within a jackknife and/or collision zone730. The system12may, during the reversing maneuver and as the current trailer angle is indicative of the trailer moving toward and/or into a jackknife zone, alert the user when the current trailer angle222has entered the jackknife and/or collision zones730. For example, the system12may generate a continuous audio tone and/or display a warning message to the user via the display24. A frequency of the continuous audio tone may be different than a frequency of the intermittent beeping when the desired trailer angle212is within the jackknife zone730. The system12may include a message to the driver that, in order to re-enable the system12(e.g., to begin controlling steering), the driver must manually drive forward to bring the current trailer angle222out of the jackknife zone730.

The advantages of the trailer guidance system include that there is no requirement or need for the system to calculate or determine a trajectory of the trailer along which the trailer will move to arrive at the target destination. Thus, the system avoids the costs and complexities associated with computing such trajectories or target paths along which the trailer is to be moved or associated with the ECU measuring any deviation of the measure of orientation of the trailer relative to the vehicle from an intended direction of travel of the trailer relative to the vehicle and without the need for consequential implementation of path correcting measures.

Optionally, and such as shown inFIGS.14-16, the vehicle and the trailer may both be equipped with multi-camera systems having multiple cameras to provide various bird's-eye/top down views of the areas surrounding the vehicle and/or trailer (including avatars representing the vehicle and trailer or portions thereof). As shown inFIG.14, the vehicle may include a front forward viewing vehicle camera, a left-side vehicle camera, a right-side vehicle camera and the rear backup camera, while the trailer may include a front forward viewing trailer camera, a left-side trailer camera, a right-side trailer camera and a rear trailer camera. The vehicle and the trailer may each be equipped with a surround view system ECU or a common ECU may be disposed at the vehicle (with the image data captured by the trailer cameras being provided to the common vehicle-based ECU). The system provides a complete 360 degree view around the vehicle and trailer. Dynamic guideline overlays may provide real-time feedback to the driver of the future positions of trailer and towing vehicle while the vehicle is backing up the trailer, such as via the trailer guidance system and responsive to the driver selection of an input device (knob or slider) that the driver uses to select the trailer backing up direction. The trailer360view provides better coverage around the trailer and provides enhanced reverse guidance and real time feedback to the driver when the driver uses the knob or slider input device to select the direction of trailer movement.

The system allows the driver to select various bird's eye/top down views, such as, for example, a bird's-eye/top down view of vehicle and trailer (FIG.15), or a zoomed in 360 degree surround view of the trailer (FIG.16), or a three-dimensional view of the vehicle and trailer and the surroundings. The system of the present invention includes providing ability of the driver sitting in the driver seat of the vehicle to, for example, select for display on the in-cabin video screen that is viewable by the driver, (i) a bird's-eye/top down view derived solely from image data captured by the multi-camera system of the vehicle, or (ii) a bird's-eye/top down view derived solely from image data captured by the multi-camera system of the trailer, or (iii) a bird's-eye/top down view derived solely from image data captured by the multi-camera system of the vehicle laterally tiled with a bird's-eye/top down view derived solely from image data captured by the multi-camera system of the trailer, such as in a side-by-side arrangement. The system may not need to duplicate the view captured by the front camera of the trailer and the view captured by the rear backup camera of the vehicle. The surround views and three dimensional views may be generated utilizing aspects of the systems described in U.S. Pat. Nos. 10,071,687; 9,900,522; 9,834,153; 9,762,880; 9,558,409; 9,446,713; 9,264,672 and/or 9,126,525, and/or U.S. Publication No. US-2015-0022664, which are hereby incorporated herein by reference in their entireties.

As shown inFIG.15, the system may provide a surround view display of images for viewing by the driver of the vehicle during the backing up maneuver of the vehicle and trailer. Video image data captured by some or all of at least eight cameras, with four vehicle cameras (front/left-side/right-side/rear) and up to four trailer cameras (front/left-side/right-side/rear) or more, may be image processed to generate a variety of bird's-eye/top down views that when displayed on the video screen in the interior cabin of the vehicle allow the driver when backing up the trailer to have a bird's-eye/top down view of areas surrounding the vehicle and trailer, including to the front of, to the sides of and rearward of the vehicle and to the front of, to the sides of and rearward of the trailer hitched to the vehicle.

For example, one vehicle/trailer bird's-eye/top down view may comprise a bird's-eye/top down view that shows areas adjacent to/surrounding the left side of the vehicle, the left side of the trailer, the right side of the vehicle, the right side of the trailer, the front of the vehicle, the front of the trailer, the rear of the vehicle and the rear of the trailer. Since the rear backup camera of the vehicle and the front camera of the trailer commonly view the trailer tongue region between the back of the towing vehicle and the front of the trailer body, any duplication in image data displayed and viewable by the driver in the vehicle viewing the video display is avoided so that the driver sees in bird's-eye/top down form what is between the vehicle and trailer and laterally thereof. Such bird's-eye/top down views allow the driver to better gauge the areas adjacent to/surrounding the vehicle/trailer combination, and thereby be able to appropriately intervene to avoid colliding with objects while driving over desirable areas via the trailer guidance system of the present invention. Thus, if the driver so observes such objects, the driver can take corrective action, such as easing or releasing the accelerator pedal of the vehicle to slow down or cease reversing and/or brake the vehicle and/or by exiting the trailer guidance system of the vehicle and assume manual steering control or by changing the selected input position or setting to cause the ECU of the trailer guidance system to control the electronic power steering system of the vehicle to change direction of backing up of the trailer to avoid obstacles, etc.

Optionally, image data captured by the towing vehicle cameras and the trailer cameras can be provided to a central ECU, that preferably is disposed at the towing vehicle, and the bird's-eye/top down views can be synthesized/generated in the central or common ECU using image processing techniques such as described in U.S. Pat. Nos. 10,071,687; 9,900,522; 9,834,153; 9,762,880; 9,558,409; 9,446,713; 9,264,672 and/or 9,126,525, and/or U.S. Publication No. US-2015-0022664, which are hereby incorporated herein by reference in their entireties. Image data captured by the trailer cameras optionally may be wirelessly communicated to be received at the towing vehicle and provided to the common ECU disposed in the towing vehicle. Alternatively, image data captured by the trailer cameras may be provided to the towing vehicle via a wired linkage from the trailer to the towing vehicle. Preferably, the common ECU also operates to provide the trailer guidance of the present invention. As an alternative to a common ECU, distributed ECUs may be used. For example, an ECU disposed at the towing vehicle can synthesize bird's-eye/top down views derived from image data captured by the four (or more) vehicle cameras, with this vehicle-based ECU preferably also running the trailer guidance of the present invention. A second ECU can be disposed at the trailer, and that trailer-based ECU is operable to generate bird's-eye/top down views derived from image data captured by the four (or more) trailer cameras.

Preferably, an HMI (such as a selector switch or touch control or the like) is provided in the towing vehicle and is selectable by the driver of the vehicle for display on the in-vehicle cabin video display screen of selected views selected from a menu of selectable bird's eye/top down views. For example, the driver can select for display a 360 degree bird's eye/top down view surrounding the towing vehicle and the trailer. Another selection can be a bird's eye/top down view derived at least from image data captured by the left side vehicle camera, the front vehicle camera and the right side vehicle camera (alternatively including the rear backup camera of the vehicle), and/or the driver may select a bird's eye/top down view derived at least from image data captured by the left side trailer camera, the rear trailer camera and the right side trailer camera (or alternatively including the front trailer camera). If both the vehicle surround view and the trailer surround view are selected, the video display may display the views laterally tiled in a side-by-side arrangement. Another menu selection can be a bird's eye/top down view derived from image data captured by the front vehicle camera, the left side vehicle camera, the left side trailer camera and the rear trailer camera. Another menu selection can be a bird's eye/top down view derived from image data captured by the front vehicle camera, the right side vehicle camera, the right side trailer camera and the rear trailer camera. Another menu selection can be a bird's eye/top down view derived from image data captured by the left side vehicle camera, the rear backup camera of the vehicle, the front camera of the trailer and the left side trailer camera, such bird's eye/top down view allowing the driver to see at least part of the region separating the towing vehicle from the trailer, and also see whether there is hazard of collision between the left-most rear corner of the towing vehicle and the front-most left corner of the trailer body. Another menu selection can be a bird's eye/top down view derived from image data captured by the right side vehicle camera, the rear backup camera of the vehicle, the front camera of the trailer and the right side trailer camera, such bird's eye/top down view allowing the driver to see at least part of the region separating the towing vehicle from the trailer, and also see whether there is hazard of collision between the right-most rear corner of the towing vehicle and the front-most right corner of the trailer body. Optionally, and less desirably, only the towing vehicle may be multi-camera equipped so that during a backing up event, various bird's-eye/top down views can be displayed for viewing by the driver that allows the driver to view areas surrounding the towing vehicle as well as at least partially seeing areas adjacent to the lateral sides of the trailer.

Note that optionally and preferably, image data captured by some (or all) of the vehicle cameras and/or by some or all of the trailer cameras can be image processed at an image processor(s) disposed at either (or both) of the vehicle and the trailer. Responsive to such image processing, objects or obstacles exterior either (or both) of the vehicle and the trailer can be detected, and the driver of the vehicle can be alerted to their presence (such by an audible alert or by a tactile/haptic/rumble alert or by a visual indicator or by a graphic overlay overlaying video images captured by some (or all) of the vehicle cameras and/or by some (or all) of the trailer cameras being displayed at an in-cabin video display screen viewable by the driver operating the vehicle to back up the trailer). The image processor preferably may comprise an image processing chip selected from the EYEQ family of image processing chips available from Mobileye Vision Technologies Ltd. of Jerusalem, Israel, and may include object detection software (such as the types described in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, which are hereby incorporated herein by reference in their entireties) that detect objects utilizing edge detection. The image processing chip preferably is part of the ECU used for trailer guidance.

Thus, the trailer guidance system may utilize a driver input that comprises a slider element that is laterally adjustable to position the slider element at a position along a slider range that correlates to a range of desired trailer angles (e.g., 0 to 60 degrees to the right and to the left of center and relative to a longitudinal axis of the vehicle), with the center position of the slider element correlating to a zero degree trailer angle (where a longitudinal axis of the trailer or trailer tongue is in line with the longitudinal axis of the vehicle. The driver thus can move the slider to the right or to the left to a particular position along its range that correlates to a desired or selected trailer angle to be achieved when backing up the vehicle and trailer. Responsive to the set or selected trailer angle, the ECU commands and controls the electronically controlled power steering of the vehicle to, as the vehicle and trailer are backed up, bring the current determined trailer angle to be the same as the desired or set trailer angle. When the determined trailer angle matches the desired trailer angle, the vehicle continues to back up and the ECU maintains the desired trailer angle for the backing up maneuver (absent further input to the driver input or HMI from the driver). For example, a driver could move the slider half-way to the left to select a 20 degree angle. While the driver backs up the vehicle and trailer, the driver can turn off the trailer guidance assist system. For example, the driver may position the slider at the center position or hit a “back up straight” button or input, whereby the ECU controls the power steering to bring the determined trailer angle to zero and then the vehicle and trailer can back up in a straight line. If the driver moves the slider during the backup maneuver, the ECU controls the power steering to bring the determined trailer angle to the new trailer angle setting.

Thus, in accordance with embodiments of the present invention, a driver of a vehicle desiring to use a vehicle equipped with a trailer towing hitch first connects a trailer tongue of a trailer to the trailer towing hitch of the vehicle to establish a pivoting joint or pivoting connection of the trailer tongue of the trailer to the trailer hitch of the vehicle that will allow operation and steering of the vehicle to tow the trailer when driving forward and to back up the trailer during a backing up maneuver. When connecting the trailer to the vehicle, the vehicular trailer guidance system asks and requires that the particular trailer now hitched to the vehicle be identified to the vehicular trailer guidance system, which the driver does by inputting an identification or name of the trailer to the vehicular trailer guidance system (such as via voice command or keypad entry or data touch input or alphanumeric individual selection of alphanumeric characters via the likes of a joystick/rotary knob or the like). With the particular trailer now identified to the system, the system determines whether the trailer is a previously identified trailer.

If the hitched trailer is not a previously identified/known trailer, the system initiates a trailer calibration procedure for that particular identified trailer. A preferred calibration procedure is disclosed in U.S. provisional applications, Ser. No. 62/705,968, filed Jul. 24, 2020, Ser. No. 62/705,967, filed Jul. 24, 2020, and/or Ser. No. 62/705,966, filed Jul. 24, 2020, which are hereby incorporated herein by reference in their entireties. Calibration under the preferred procedure includes the steps set forth inFIGS.17-21. Such a calibration process may be desired or required when, for example, a new trailer (if not previously calibrated) is hitched to the trailer, or for a previously calibrated trailer, when there is a change in shape of the trailer (installing after-market items on the trailer tongue or drawbar), when the hitch length gets changed (e.g., by 2 inches), or when there is an unsuccessful previous calibration(s).

The calibration procedure requires the driver, with the trailer hitched to the vehicle, to drive the vehicle straight forward for a desired minimum straight line driving distance, and then manually steer the vehicle to make a right hand (or left hand) turn. The system determines desired characteristics of the trailer during the calibration procedure (such characteristics including the trailer beam length [the distance between the trailer coupler (that attaches to the trailer hitch ball of the vehicle) at the end of the trailer tongue and the center of the trailer axle] and the collision angle [the trailer angle at which the truck and trailer would collide]). By following the preferred calibration procedure, physical measurement by the driver of physical characteristics of the particular trailer hitched to the vehicle (such as distance of the trailer tongue to the camera, location of the camera relative to the vehicle ball hitch, trailer beam length, etc.) is obviated.

Upon the system determining the desired characteristics, during calibration driving by the driver of the vehicle towing the trailer, the system stores such trailer characteristics/trailer data in a memory of the system (typically a memory chip on the ECU), under the identifier or name for the particular trailer input by the driver, and the system alerts/informs the driver that trailer calibration has been successfully completed. When that same trailer is next hitched to the vehicle, and upon the driver identifying that particular trailer when requested to by the control/ECU of the system of the present invention, the system searches the memory of the control/ECU for previously stored and calibrated identified trailers and recognizes that this particular identified trailer has already been calibrated to the vehicle, and so no further calibration for that particular trailer is need, and the backing-up trailer guidance by the system can commence.

With a recognized and identified particular trailer now being hitched to the vehicle, the vehicular trailer guidance system is operable to back up the trailer during a backing up maneuver of the vehicle with the trailer hitched thereto via the driver operating a three mode human machine interface (HMI) accessible by the driver while sitting in the driver's seat of the vehicle. The system is operable to steer the vehicle to properly back up the trailer via controlling the electronic power steering system of the vehicle, and not by the driver turning the steering wheel to steer the vehicle during the backing up maneuver of the trailer. Taking as an example, a vehicle used in the United States and other right hand drive countries (that drive on the right side of a two-way road), and bearing in mind that with the trailer tongue hitched to the trailer hitch of the vehicle, the vehicle's rear backup camera, while continuing to view at least a portion of the trailer tongue, is otherwise generally or partially or wholly obstructed in its rearward view by the body of the trailer itself, the driver of the vehicle (either looking at or using a left/driver side exterior mirror of the vehicle or looking out a left side/driver side window of the vehicle) determines a location/destination/area behind and to the left of the vehicle and trailer that the driver desires to back up the trailer towards/to. The driver, having first activated the vehicular trailer guidance system of the present invention via an appropriate input (if not already activated due to the required calibration for the hitched trailer), operates the three mode HMI to cause the electronic power steering system of the vehicle to back up the trailer toward the target/selected location/destination/area. The three mode HMI comprises a first or left mode HMI, a second or center or straight mode HMI and a third or right mode HMI.

For example, when desiring to back up the trailer towards/to a location rearward and to the left of the rear of the trailer, the driver slides a slider knob of a slider first mode HMI to the left, and with the degree of movement of the slider to the left (along a maximum range of left slider movement allowed) being commensurate with how far leftward/laterally sideward the driver desires to back up the trailer towards/to. Wherever along such a slider scale of the first mode HMI the driver slides the slider knob to (with the slider knob remaining at the selected location upon release of the slider knob by the driver), the system of the invention correlates that to a desired trailer angle to be achieved for backing up the trailer towards the target/selected location/destination/area that is left-sideward and rearward of the back of the trailer. The system operates/controls the electronic power steering system of the vehicle to bring the steering angle of the vehicle (as known and determined by the electronic power steering system of the vehicle) to come in line with and match the desired trailer angle commensurate with where the driver has leftward-moved the slider of the first mode left-backup HMI. The furthest leftward that the driver can move/slide the left slider corresponds with a maximum trailer angle that the system permits for a leftward trailer backing up event. Wherever the driver slides the slider input knob to along its permitted range/track of travel and rests there, the system of the invention maintains the trailer angle of the trailer relative to the longitudinal axis of the vehicle at the desired trailer angle.

Backing up of the vehicle and trailer to the right, using the third or right back up mode is similar to the left backing up maneuver, but of course with the driver using a right-side (in the USA, passenger side) exterior rearview mirror or by looking out a right-side side window of the vehicle to decide the target/selected location/destination/area where the driver desires to right-side backup the trailer towards/to.

At any time that the driver selects the second or straight back up mode HMI, the system of the invention determines that the driver wishes to back the trailer straight up (i.e., in a straight line rearward that is in line with or parallel to the longitudinal axis of the trailer at the time that the second mode HMI is selected or actuated). Thus, upon selection of the straight back up mode, the system of the invention controls the electronic power steering system of the vehicle to move the trailer angle to zero degrees and maintain it at zero degrees backing straight up.

The three mode HMI of the system of the invention can have several forms. Preferably, three separate user inputs are provided, each corresponding to (i) a left-side/left orientated input corresponding to the first or left back up mode, (ii) a separate center/straight input corresponding to the second or straight back up mode, or (iii) a right-side/right orientated input corresponding to the third or right back up mode, with all three inputs arranged generally horizontally at, for example, an instrument panel of the interior cabin of the vehicle, where they are generally and readily accessible by the driver's hand. By so orientating and arranging the three user inputs in parallel, the driver can cognitively and readily associate the left input with the left back up maneuver, the center input with the straight back up maneuver and the right input with the right back up maneuver. Since the three inputs are separate, the driver can readily select any one of them. Thus, for example, the left or first mode input can be a slider with a degree of leftward slide being commensurate with the degree of leftward backing up desired, the center input can be a button that, when pressed, causes straight line backing up, and when pressed again exits straight line backing up, and the right or third mode input can be a slider with a degree of rightward slide being commensurate with the degree of rightward backing up desired.

Of course other HMI configurations can be used. For example, a rotary knob can be used that, when rotated to the left, corresponds with the leftward or first mode, when rotated to the right, corresponds with the rightward or third mode, and with a separate central push button (or optionally a switch actuated by pushing the rotary knob itself) that, when the driver separately presses the central push button, causes the system to enter or exit the straight backup mode.

A rotatable driver interface, such as is described in U.S. Pat. No. 8,972,109, which is hereby incorporated herein by reference in its entirety, that includes a rotational control input device at a zero position and movable to opposing left backing up and right backing up positions relative to the zero position can be used with the system of the present invention, but is less desirable to be used than using the preferred separately driver accessible second mode of the three mode HMI of the system of the present invention. This is at least because having the likes of a rotary knob controlling a trailer backing up event that automatically swings back to/is biased back to the zero or straight line backing up position means that, once the driver releases the counterclockwise knob rotation or releases the clockwise knob rotation, the rotary knob returns to the center position and backing up of the trailer changes to straight line backing up, whether the driver so wants or not. By having the preferred three independent mode HMI of the present invention, the driver (for example) can move a slider switch to halfway to the left-permitted span of travel and then release the slider switch, whereupon the system of the invention will continue to back up the trailer at the trailer angle corresponding to where the first mode HMI slider is positioned and released by the driver. Then, in accordance with the present invention, when the driver wishes to change to straight line trailer backing up, the driver is free to operate/engage the separate second mode.

Thus, when the driver of the vehicle decides to back up the trailer to a target/selected location/destination/area located rearward and laterally sideward/leftward of the back of the vehicle, the driver moves the first mode/input to a position along its range of travel that the driver can associate with backing up towards such a desired target/selected location/destination/area. For example, a graphic overlay may be shown on the video screen overlaying video images captured by the rear backup camera, and this overlay enables or allows the driver to gauge where/how far laterally sideward to the left of the back of the trailer the driver wishes to back up the trailer towards and to.

Once the driver moves the HMI knob or slider on the slider scale to correspond to where he or she wishes to back up, the control/ECU receives an output from the first mode HMI (preferably provided via a vehicle communication bus or CAN bus of the vehicle) indicating to the ECU the selected slider position/trailer angle that the driver has selected to back up the trailer to where the driver-desired target/selected location/destination/area. Upon receiving that input from the HMI, the ECU, responsive at least in part to processing image data captured by the rear backup camera, determines the then current trailer angle, and the ECU receives from the electronic power steering system of the vehicle and/or from a steering wheel/steering position sensor of the vehicle the then current steering angle of the vehicle (that is typically determined by the angle the two front wheels of the vehicle are making relative to the longitudinal axis of the vehicle).

The ECU then controls the electronic power steering system of the vehicle, with the driver operating the accelerator pedal and/or brake pedal of the vehicle to propel the vehicle rearward and control the rearward speed of the vehicle, to bring the angle made by the tongue of the trailer to coincide with an angle commensurate with the position of the HMI slider along the range of travel of the slider selected by the driver. Upon the electronic power steering system of the vehicle having reversed the vehicle so that the then current trailer angle of the tongue of the trailer relative to the longitudinal axis of the vehicle coincides with the trailer angle desired for the trailer to back up towards the driver desired/selected/target destination/location/area, the ECU controls the electronic power steering system of the vehicle to steer/maintain that desired trailer angle, while the driver continues to back up the trailer using the accelerator pedal of the vehicle to propel the vehicle rearward (desirably at a low rearward travel velocity, such as below five mph) until the driver either disables the trailer guidance system of the vehicle or selects a different mode (e.g., the backing up straight mode or the backup up right mode) or moves or changes the backing up left mode's slider to a different location.

The system may utilize aspects of the trailering systems or trailer angle detection systems or trailer hitch systems described in U.S. Pat. Nos. 10,755,110; 10,638,025; 10,160,382; 10,086,870; 9,558,409; 9,446,713; 9,085,261 and/or 6,690,268, and/or U.S. Publication Nos. US-2020-0361397; US-2020-0356788; US-2020-0334475; US-2020-0017143; US-2019-0347825; US-2019-0118860; US-2019-0064831; US-2019-0042864; US-2019-0039649; US-2019-0143895; US-2019-0016264; US-2018-0276839; US-2018-0276838; US-2018-0253608; US-2018-0215382; US-2017-0254873 and/or US-2017-0217372, and/or U.S. patent application Ser. No. 17/115,826, filed Dec. 9, 2020 and published Jun. 10, 2021 as U.S. Publication No. US-2021-0170820, Ser. No. 17/247,220, filed Dec. 4, 2020 and published Jun. 10, 2021 as U.S. Publication No. US-2021-0170947, Ser. No. 16/948,253, filed Sep. 10, 2020 and published Mar. 18, 2021 as U.S. Publication No. US-2021-0078634, and/or Ser. No. 16/946,542, filed Jun. 26, 2020 and published Dec. 31 2020 as U.S. Publication No. US-2021-0406967, and/or U.S. provisional applications, Ser. No. 63/199,155, filed Dec. 10, 2020, Ser. No. 62/705,968, filed Jul. 24, 2020, Ser. No. 62/705,967, filed Jul. 24, 2020, and/or Ser. No. 62/705,966, filed Jul. 24, 2020, which are all hereby incorporated herein by reference in their entireties.

The system includes an image processor operable to process image data captured by the camera or cameras, such as for detecting objects or other vehicles or pedestrians or the like in the field of view of one or more of the cameras. For example, the image processor may comprise an image processing chip selected from the EYEQ family of image processing chips available from Mobileye Vision Technologies Ltd. of Jerusalem, Israel, and may include object detection software (such as the types described in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, which are hereby incorporated herein by reference in their entireties), and may analyze image data to detect vehicles and/or other objects. Responsive to such image processing, and when an object or other vehicle is detected, the system may generate an alert to the driver of the vehicle and/or may generate an overlay at the displayed image to highlight or enhance display of the detected object or vehicle, in order to enhance the driver's awareness of the detected object or vehicle or hazardous condition during a driving maneuver of the equipped vehicle.

The vehicle may include any type of sensor or sensors, such as imaging sensors or radar sensors or lidar sensors or ultrasonic sensors or the like. The imaging sensor or camera may capture image data for image processing and may comprise any suitable camera or sensing device, such as, for example, a two dimensional array of a plurality of photosensor elements arranged in at least 640 columns and 480 rows (at least a 640×480 imaging array, such as a megapixel imaging array or the like), with a respective lens focusing images onto respective portions of the array. The photosensor array may comprise a plurality of photosensor elements arranged in a photosensor array having rows and columns. Preferably, the imaging array has at least 300,000 photosensor elements or pixels, more preferably at least 500,000 photosensor elements or pixels and more preferably at least 1 million photosensor elements or pixels. The imaging array may capture color image data, such as via spectral filtering at the array, such as via an RGB (red, green and blue) filter or via a red/red complement filter or such as via an RCC (red, clear, clear) filter or the like. The logic and control circuit of the imaging sensor may function in any known manner, and the image processing and algorithmic processing may comprise any suitable means for processing the images and/or image data.

Optionally, the vision system may include a display for displaying images captured by one or more of the imaging sensors for viewing by the driver of the vehicle while the driver is normally operating the vehicle. Optionally, for example, the vision system may include a video display device, such as by utilizing aspects of the video display systems described in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755; 7,626,749; 7,581,859; 7,446,650; 7,338,177; 7,274,501; 7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187; 6,690,268; 7,370,983; 7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551; 5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,708,410; 5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,501; 6,222,460; 6,513,252 and/or 6,642,851, and/or U.S. Publication Nos. US-2014-0022390; US-2012-0162427; US-2006-0050018 and/or US-2006-0061008, which are all hereby incorporated herein by reference in their entireties.

Changes and modifications in the specifically described embodiments can be carried out without departing from the principles of the invention, which is intended to be limited only by the scope of the appended claims, as interpreted according to the principles of patent law including the doctrine of equivalents.