Patent Description:
Trailers are usually unpowered vehicles that are pulled by a powered tow vehicle. A trailer may be a utility trailer, a popup camper, a travel trailer, livestock trailer, flatbed trailer, enclosed car hauler, and boat trailer, among others. The tow vehicle may be a car, a crossover, a truck, a van, a sports-utility-vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to attach to the trailer and pull the trailer. The trailer may be attached to a powered vehicle using a trailer hitch. A receiver hitch mounts on the tow vehicle and connects to the trailer hitch to form a connection. The trailer hitch may be a ball and socket, a fifth wheel and gooseneck, or a trailer jack. Other attachment mechanisms may also be used. In addition to the mechanical connection between the trailer and the powered vehicle, in some examples, the trailer may be electrically connected to the tow vehicle. As such, the electrical connection allows the trailer to take the feed from the powered vehicle's rear light circuit, allowing the trailer to have taillights, turn signals, and brake lights that are in sync with the powered vehicle's lights.

Some trailers may be equipped with aftermarket trailer cameras that are mounted by the driver. The mounted trailer cameras may either be wired or wireless cameras and can communicate with the tow vehicle. Many vehicle-trailer applications, such as, but not limited to, trailer hitch assist, trailer reverse assist, rely on camera calibration parameters of cameras supported by the vehicle and the trailer. Therefore, it is desirable to have a system that is capable of providing automatic camera calibration parameters of one or more aftermarket cameras supported by the trailer for use by vehicle-trailer applications. <CIT> describes a method for calibrating a camera device of a vehicle, wherein the calibration can be in a vehicle longitudinal direction or vehicle trans-verse direction. <CIT> describes a trailer assist system for a vehicle including a plurality of sensors disposed at a vehicle and trailer. XP055757381 describes calibration of a vehicle camera system with divergent fields-of-view in an urban environment. <CIT> describes a method of calibrating a computer-based vision system onboard a craft.

One aspect of the disclosure provides a method of calibrating extrinsic parameters of a trailer camera. The trailer camera is supported by a trailer that is attached to a tow vehicle. The method includes determining, at data processing hardware (e.g., controller executing the calibration algorithm), a three-dimensional feature map from one or more vehicle images received from a camera supported by the tow vehicle. In some examples, determining the three-dimensional feature map includes executing at least one of: a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, or a Structure from Motion (SfM) algorithm. The camera may be positioned on a front portion of the vehicle to capture a front environment of the vehicle. In some examples, the one or more vehicle images may be received from more than one camera positioned to capture a front and side environment of the vehicle. The method includes identifying, at the data processing hardware, reference points within the three-dimensional feature map. The method also includes detecting, at the data processing hardware, the reference points within one or more trailer images received from the trailer camera after the vehicle and the trailer moved a predefined distance in the forward direction. The method includes determining, at the data processing hardware, a trailer camera location of the trailer camera relative to the three-dimensional feature map. Additionally, the method includes determining, at the data processing hardware, a trailer reference point based on the trailer camera location. Finally, the method includes determining, at the data processing hardware, extrinsic parameters of the trailer camera relative to the trailer reference point.

Implementations of this aspect of the disclosure may include one or more of the following optional features. In some implementations, the method includes transmitting the extrinsic parameters of the trailer camera to one or more vehicle systems causing the vehicle system to execute a behavior using the extrinsic parameters. The extrinsic parameters may define the position of a center of the trailer camera and a heading of the trailer camera. In some implementations, the trailer reference point is at a predefined distance from the trailer camera. The trailer reference point may overlap with the trailer camera location.

The method may also include associating an identifier with each one of the reference points identified in the one or more vehicle images; where detecting the reference points within one or more trailer images received from the trailer camera includes determining the identifier associated with each one of the reference points.

In some implementations, the method further includes, before detecting, at the data processing hardware, the reference points within one or more trailer images received from the trailer camera: transmitting instructions to a user interface and receiving images from the trailer camera after the vehicle has traveled a predefined distance. The instruction causing a user display to prompt a driver of the vehicle to drive the vehicle in forward direction.

Another aspect of the disclosure provides an example arrangement of operations for a method of calibrating extrinsic parameters of a trailer camera. The trailer camera is supported by a trailer attached to a tow vehicle. The method includes determining, at data processing hardware (e.g., controller executing the calibration algorithm), a three-dimensional feature map from one or more vehicle images received from a camera supported by the tow vehicle. The method includes identifying, at the data processing hardware, reference points within the three-dimensional feature map. The method includes determining, at the data processing hardware, a vehicle pose relative to a first origin point within the three-dimensional feature map. The method also includes detecting, at the data processing hardware, the reference points within one or more trailer images received from the trailer camera. The method includes determining, at the data processing hardware, a trailer camera location of the trailer camera relative to the three-dimensional feature map. Additionally, the method includes determining, at the data processing hardware, a first trailer camera pose of the trailer camera relative to the vehicle pose within the three-dimensional feature map. The method includes determining, at the data processing hardware, a trailer reference point based on the trailer camera location. The method includes determining, at the data processing hardware, a trailer reference pose of the trailer reference point relative to the trailer camera location. Additionally, the method includes determining, at the data processing hardware, a second trailer camera pose of the trailer camera relative to the trailer reference pose. The method includes determining, at the data processing hardware, extrinsic parameters of the trailer camera relative to the trailer reference pose.

Implementations of this aspect of the disclosure may include one or more of the following optional features. In some implementations, the trailer reference point is at a predefined distance from the trailer camera. The trailer reference point may overlap with the trailer camera location. The extrinsic parameters may define the position of a center of the trailer camera and a heading of the trailer camera.

In some examples, the method also includes associating an identifier with each one of the reference points identified in the one or more vehicle images, where detecting the reference points within one or more trailer images received from the trailer camera includes determining the identifier associated with each one of the reference points.

The method may also include transmitting the extrinsic parameters of the trailer camera to one or more vehicle systems causing the vehicle system to execute a behavior using the extrinsic parameters.

In some implementations, determining a three-dimensional feature map from one or more vehicle images received from a camera supported by the tow vehicle includes executing at least one of: a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, or a Structure from Motion (SfM) algorithm.

Before detecting, at the data processing hardware, the reference points within one or more trailer images received from the trailer camera, the method may include transmitting instructions to a user interface, the instruction causing a user display to prompt a driver of the vehicle to drive the vehicle in a forward direction, and receiving images from the trailer camera after the vehicle has traveled a predefined distance.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. These operations include the methods described above.

A tow vehicle, such as, but not limited to a car, a crossover, a truck, a van, a sports-utility-vehicle (SUV), and a recreational vehicle (RV) may be configured to tow a trailer. The tow vehicle connects to the trailer by way of a trailer hitch. It is desirable to have a system capable of calibrating aftermarket cameras supported by the trailer.

Referring to <FIG>, <FIG> and <FIG>, in some implementations, a vehicle-trailer system <NUM> includes a tow vehicle <NUM> having a tow ball <NUM> supported by a vehicle hitch bar <NUM>. The vehicle-trailer system <NUM> also includes a trailer <NUM> having a trailer hitch coupler <NUM> supported by a trailer hitch bar <NUM>. The vehicle tow ball <NUM> is coupled to the trailer hitch coupler <NUM>. The tow vehicle <NUM> may include a drive system <NUM> that maneuvers the tow vehicle <NUM> across a road surface based on drive commands having x, y, and z components, for example. As shown, the drive system <NUM> includes a front right wheel <NUM>, 112a, a front left wheel <NUM>, 112b, a rear right wheel <NUM>, 112c, and a rear left wheel <NUM>, 112d. The drive system <NUM> may include other wheel configurations as well. The drive system <NUM> may also include a brake system (not shown) that includes brakes associated with each wheel <NUM>, 112a-d, and an acceleration system (not shown) that is configured to adjust a speed and direction of the tow vehicle <NUM>. In addition, the drive system <NUM> may include a suspension system (not shown) that includes tires associates with each wheel <NUM>, 112a-d, tire air, springs, shock absorbers, and linkages that connect the tow vehicle <NUM> to its wheels <NUM>, 112a-d and allows relative motion between the tow vehicle <NUM> and the wheels <NUM>, 112a-d.

The tow vehicle <NUM> may move across the road surface by various combinations of movements relative to three mutually perpendicular axes defined by the tow vehicle <NUM>: a transverse axis XV, a fore-aft axis YV, and a central vertical axis ZV. The transverse axis XV extends between a right-side and a left-side of the tow vehicle <NUM>. A forward drive direction along the fore-aft axis YV is designated as FV, also referred to as a forward motion. In addition, an aft or rearward drive direction along the fore-aft direction YV is designated as RV, also referred to as rearward motion. In some examples, the tow vehicle <NUM> includes a suspension system (not shown), which when adjusted causes the tow vehicle <NUM> to tilt about the XV axis and or the YV axis, or move along the central vertical axis ZV.

Moreover, the trailer <NUM> follows the tow vehicle <NUM> across the road surface by various combinations of movements relative to three mutually perpendicular axes defined by the trailer <NUM>: a trailer transverse axis XT, a trailer fore-aft axis YT, and a trailer central vertical axis ZT. The trailer transverse axis XT extends between a right-side and a left-side of the trailer <NUM> along a trailer turning axle <NUM>. In some examples, the trailer <NUM> includes a front axle (not shown) and rear axle <NUM>. In this case, the trailer transverse axis XT extends between a right-side and a left-side of the trailer <NUM> along a midpoint of the front and rear axle (i.e., a virtual turning axle). A forward drive direction along the trailer fore-aft axis YT is designated as FT, also referred to as a forward motion. In addition, a trailer aft or rearward drive direction along the fore-aft direction YT is designated as RT, also referred to as rearward motion. Therefore, movement of the vehicle-trailer system <NUM> includes movement of the tow vehicle <NUM> along its transverse axis XV, fore-aft axis YV, and central vertical axis ZV, and movement of the trailer <NUM> along its trailer transverse axis XT, trailer fore-aft axis YT, and trailer central vertical axis ZT. Therefore, when the tow vehicle <NUM> makes a turn as it moves in the forward direction FV, then the trailer <NUM> follows along. While turning, the tow vehicle <NUM> and the trailer <NUM> form a trailer angle α being an angle between the vehicle fore-aft axis YV and the trailer fore-aft axis YT.

The tow vehicle <NUM> may include a user interface <NUM>. The user interface <NUM> may include a display <NUM>, a knob, and a button, which are used as input mechanisms. In some examples, the display <NUM> may show the knob and the button. While in other examples, the knob and the button are a knob button combination. In some examples, the user interface <NUM> receives one or more driver commands from the driver via one or more input mechanisms or a touch screen display <NUM> and/or displays one or more notifications to the driver. In some examples, the display <NUM> displays an image <NUM> of an environment of the vehicle-trailer system <NUM>.

The vehicle-trailer system <NUM> includes a sensor system <NUM> that provides sensor system data <NUM> of the vehicle-trailer system <NUM> and its surroundings for aiding the driver while driving. The sensor system <NUM> may include different types of sensors that may be used separately or with one another to create a perception of the environment of the vehicle-trailer system <NUM> that is used for the tow vehicle <NUM> to drive and aid the driver in make intelligent decisions based on objects and obstacles detected by the sensor system <NUM>. The sensor system <NUM> may include the one or more cameras <NUM>, 132a-f supported by the vehicle-trailer system <NUM>. In some implementations, the tow vehicle <NUM> includes a front vehicle camera 132a (i.e., a first camera) that is mounted to provide a view of a front-driving path for the tow vehicle <NUM>, or in other words, the front vehicle camera 132a captures images <NUM> of a front environment of the tow vehicle <NUM>. In some examples, the sensor system <NUM> also includes side vehicle cameras 132c, 132d (i.e., third camera and fourth camera) each mounted to provide a side images <NUM> of the side environment of the tow vehicle <NUM>. The vehicle side cameras 132c, 132d may each be mounted on a side view mirror of the tow vehicle <NUM>, a side door, or a side frame. The sensor system <NUM> also includes one or more cameras 132d-f supported by the trailer <NUM>. For example, a trailer rear camera 132d (i.e., fourth camera) is mounted to provide a view of a rear environment of the trailer <NUM>. Additionally, side trailer cameras 132e, 132f (i.e., fifth camera and sixth camera) are mounted to each provide a side image <NUM> of the side environment of the trailer <NUM>. The trailer cameras 132d-e may be connected to the vehicle <NUM> (e.g., a vehicle controller <NUM>) via a wire connection or wirelessly. As shown, the vehicle-trailer system <NUM> includes six cameras <NUM>, three cameras 132a-c supported by the tow vehicle <NUM> and three cameras 132d-e supported by the trailer <NUM>; however, the vehicle <NUM> may include at least one or more cameras <NUM> and the trailer may include at least one or more cameras <NUM>, where each camera <NUM> is positioned to capture an environment of the vehicle-trailer system <NUM>. Each camera <NUM> may include intrinsic parameters (e.g., focal length, image sensor format, and principal point) and extrinsic parameters (e.g., the extrinsic parameters define the position of the camera center and the heading of the camera in with respect to a reference point).

The sensor system <NUM> may also include other sensors <NUM> that detect the vehicle motion, i.e., speed, angular speed, position, etc. The other sensors <NUM> may include an inertial measurement unit (IMU) configured to measure the vehicle's linear acceleration (using one or more accelerometers) and rotational rate (using one or more gyroscopes). In some examples, the IMU also determines a heading reference of the tow vehicle <NUM>. Therefore, the IMU determines the pitch, roll, and yaw of the tow vehicle <NUM>. The other sensors <NUM> may also include, but are not limited to, radar, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), ultrasonic, HFL (High Resolution 3D Flash LIDAR), etc..

The user interface <NUM> and the sensor system <NUM> are in communication with a vehicle controller <NUM>. The vehicle controller <NUM> includes a computing device (or data processing hardware) <NUM> (e.g., central processing unit having one or more computing processors) in communication with non-transitory memory or hardware memory <NUM> (e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s)). The controller may be supported by the tow vehicle <NUM>, the trailer <NUM>, or both the tow vehicle <NUM> and the trailer <NUM>. In some example, the controller <NUM> executes a calibration algorithm <NUM> to calibrate one or more trailer cameras <NUM>, 132d-f. As shown, the vehicle controller <NUM> is supported by the tow vehicle <NUM>; however, the vehicle controller <NUM> may be separate from the tow vehicle <NUM> and in communication with the tow vehicle <NUM> via a network (not shown).

In some implementations, the calibration algorithm <NUM> is configured to simplify a calibration process of the trailer cameras 132d-f attached to the trailer <NUM> by the driver or the trailer cameras 132d-f that are previously installed on the trailer <NUM> but not calibrated. Since the trailer cameras 132d-f may be either connected to the controller via a wire or wirelessly, it may be difficult for the driver to calibrate the extrinsic parameters (e.g., location and rotation with respect to a reference point) of the trailer cameras 132d-f. As such, the calibration algorithm <NUM> provides an automatic method to calibrate the cameras 132d-f which only requires a driver to drive in a forward direction.

The controller <NUM> receives intrinsic parameters (e.g., focal length, image sensor format, principal point, and distortion parameters) of each one of the vehicle cameras 132a-c and each one of the trailer cameras 132d-f from the cameras <NUM> since the intrinsic parameters are known by each camera <NUM>, 132a-f. As for the extrinsic parameters of the vehicle cameras 132a-c, since these cameras 132a-c are already installed on the vehicle <NUM>, their location and rotation are known, and therefore, the extrinsic parameters of the vehicle cameras 132a-c are known. However, the extrinsic parameters of the trailer cameras 132d-f are not known since these cameras 132d-f are placed on the trailer <NUM> by the driver. Therefore, the calibration algorithm <NUM> automatically calibrates the extrinsic parameters <NUM> of each of the trailer cameras 132d-f relative to a trailer reference point <NUM>.

Referring to <FIG>, the calibration algorithm <NUM> includes three stages <NUM>, <NUM>, <NUM>. During the first stage <NUM>, the calibration algorithm <NUM> builds a three-dimensional feature map <NUM> based on images <NUM> captured by the vehicle cameras 132a-c and determines a vehicle pose <NUM> (e.g., position and orientation) relative to a vehicle reference point <NUM> and based on the three-dimensional feature map <NUM>. The vehicle reference point <NUM> may be a predetermined point within the vehicle <NUM>. In some examples, the vehicle reference point <NUM> is at the intersection point of the vehicle transverse axis XV, the vehicle fore-aft axis YV, and the vehicle central vertical axis ZV. The calibration algorithm <NUM> may use a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, and/or a Structure from Motion (SfM) algorithm to determine the three-dimensional feature map. The VO, SLAM and SfM frameworks are well established theories and allow the calibration algorithm <NUM> to localize one or more reference points <NUM> in real-time in a self-generated 3D point cloud map generated based on the received images <NUM>. The VO method extracts image feature points <NUM> and tracks the extracted image feature points <NUM> in the image sequence. Examples of the feature points <NUM> may include, but are not limited to, edges, corners or blobs of objects within the images <NUM>. The VO method may also use pixel intensity in the image sequence directly as visual input. The SLAM method constructs or updates a map of an unknown environment while simultaneously keeping track of one or more targets or image feature points <NUM> (e.g., reference points). In other words, the SLAM method uses the received images <NUM> as the only source of external information to construct a representation of the environment including the image feature points <NUM>. The SfM method estimates the 3D structure of the objects or image feature points <NUM> based on the received images <NUM> (i.e., 2D images).

Once the calibration algorithm <NUM> determines the three-dimensional feature map <NUM>, then the calibration algorithm <NUM> determines a vehicle pose <NUM> relative to a point of origin within the three-dimensional feature map <NUM>.

During the second stage <NUM>, the calibration algorithm <NUM> re-localizes the trailer cameras 132d-f relative to the three-dimensional feature map <NUM>. In other words, the calibration algorithm <NUM> determines a trailer camera location <NUM> associated with each trailer camera 132d-f, where the trailer camera location <NUM> is relative to the three-dimensional feature map <NUM>. In some examples, the calibration algorithm <NUM> instructs the display <NUM> to prompt the driver to driver the vehicle-trailer system <NUM> in a forward direction FV, FT. During the forward move of the vehicle-trailer system <NUM>, the trailer cameras 132d-f capture images <NUM> and the calibration algorithm <NUM> analyses the captured trailer images <NUM> to identify the one or more image feature <NUM> identified by the vehicle images <NUM> at the first stage <NUM>. Once the calibration algorithm <NUM> identifies one or more of the one or more image features <NUM>, then the calibration algorithm <NUM> determines a trailer camera pose <NUM> of each trailer camera 132d-f relative to the three-dimensional feature map <NUM>. Each trailer camera pose <NUM> includes an (x, y, z) position and an orientation with a point of origin defined within the three-dimensional feature map <NUM>.

During the third stage <NUM>, the calibration algorithm <NUM> estimates the trailer camera extrinsic parameters <NUM> relative to a trailer reference point <NUM> based on the camera poses <NUM> determined during the second stage <NUM>. The trailer reference point <NUM> may be any point within the trailer <NUM>, such as a camera location, the center of the camera location or any predetermined location within the trailer <NUM>.

In some implementations, the calibration algorithm <NUM> optimizes the trailer camera extrinsic parameters <NUM> and the pose of trailer reference point <NUM> by minimizing the sum of all reprojection errors, while keeping the coordinates of all feature points <NUM> of the three-dimensional map <NUM> fixed. Reprojection errors provide a qualitative measure of accuracy. A reprojection error is an image distance between a projected point and a measured one. The reprojection error is used to determine how closely an estimate of a three-dimensional trailer reference point <NUM> recreates the point's actual projection.

Once the third stage <NUM> is completed, the calibration algorithm <NUM> communicates or transmits the trailer camera extrinsic parameters <NUM> and the pose of trailer reference point <NUM> to one or more vehicle-trailer system that rely on the trailer camera extrinsic parameters <NUM> and the pose of trailer reference point <NUM> to make system decisions. For example, the calibration algorithm <NUM> communicates or transmits the trailer camera extrinsic parameters <NUM> and the pose of trailer reference point <NUM> to a path planning system, a trailer reverse assist system, a trailer hitch assist system, or any other system that relies on the trailer camera extrinsic parameters <NUM> and the pose of trailer reference point <NUM>.

The calibration algorithm <NUM> executes an automatic calibration method that reduces the complexity of operations for the driver and leverages vehicle cameras 132a-c to calibrate the trailer cameras 132d-f. The calibration algorithm <NUM> enables new features for the trailer <NUM>.

<FIG> provides an example arrangement of operations for a method <NUM> of calibrating extrinsic parameters <NUM> of a trailer camera 132d, 132e, 132f using the system of <FIG>. The trailer camera 132d, 132e, 132f is supported by a trailer <NUM> that is attached to a tow vehicle <NUM>. At block <NUM>, the method <NUM> includes determining, at data processing hardware (e.g., controller <NUM> executing the calibration algorithm <NUM>), a three-dimensional feature map <NUM> from one or more vehicle images <NUM> received from a camera 132a, 132b, 132c supported by the tow vehicle <NUM>. In some examples, determining the three-dimensional feature map <NUM> includes executing at least one of: a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, or a Structure from Motion (SfM) algorithm. The camera 132a, 132b, 132c may be positioned on a front portion of the vehicle <NUM> to capture a front environment of the vehicle <NUM>. In some examples, the one or more vehicle images <NUM> may be received from more than one camera 132a, 132b, 132c positioned to capture a front and side environment of the vehicle <NUM>. At block <NUM>, the method <NUM> includes identifying, at the data processing hardware, reference points <NUM> within the three-dimensional feature map <NUM>. At block <NUM>, the method <NUM> includes detecting, at the data processing hardware <NUM>, the reference points <NUM> within one or more trailer images <NUM> received from the trailer camera 132d, 132e, 132f after the vehicle <NUM> and the trailer <NUM> moved a predefined distance in the forward direction FV, FT. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a trailer camera location <NUM> of the trailer camera 132d, 132e, 132f relative to the three-dimensional feature map <NUM>. Additionally, at block <NUM>, the method <NUM> includes determining, at the data processing hardware, a trailer reference point <NUM> based on the trailer camera location <NUM>. Finally, at block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, extrinsic parameters <NUM> of the trailer camera 132d, 132e, 132f relative to the trailer reference point <NUM>.

In some implementations, the method <NUM> includes transmitting the extrinsic parameters <NUM> of the trailer camera 132d, 132e, 132f to one or more vehicle systems causing the vehicle system to execute a behavior using the extrinsic parameters <NUM>. The extrinsic parameters <NUM> may define the position of a center of the trailer camera 132d, 132e, 132f and a heading of the trailer camera 132d, 132e, 132f. In some implementations, the trailer reference point <NUM> is at a predefined distance from the trailer camera 132d, 132e, 132f. The trailer reference point <NUM> may overlap with the trailer camera location <NUM>.

The method <NUM> may also include associating an identifier with each one of the reference points <NUM> identified in the one or more vehicle images <NUM>; where detecting the reference points <NUM> within one or more trailer images <NUM> received from the trailer camera 132d, 132e, 132f includes determining the identifier associated with each one of the reference points <NUM>.

In some implementations, the method <NUM> further includes, before detecting, at the data processing hardware <NUM>, the reference points <NUM> within one or more trailer images <NUM> received from the trailer camera 132d, 132e, 132f: transmitting instructions to a user interface <NUM> and receiving images <NUM> from the trailer camera 132d, 132e, 132f after the vehicle <NUM> has traveled a predefined distance. The predefined distance may cause the vehicle <NUM> and the trailer <NUM> to be aligned such that the trailer angle is zero. The instruction causing a user display <NUM> to prompt a driver of the vehicle <NUM> to drive the vehicle <NUM> in forward direction.

<FIG> provides an example arrangement of operations for a method <NUM> of calibrating extrinsic parameters <NUM> of a trailer camera 132d, 132e, 132f using the system of <FIG>. The trailer camera 132d, 132e, 132f is supported by a trailer <NUM> attached to a tow vehicle <NUM>. At block <NUM>, the method <NUM> includes determining, at data processing hardware (e.g., controller <NUM> executing the calibration algorithm <NUM>), a three-dimensional feature map <NUM> from one or more vehicle images <NUM> received from a camera 132a, 132b, 132c supported by the tow vehicle <NUM>. At block <NUM>, the method <NUM> includes identifying, at the data processing hardware <NUM>, reference points <NUM> within the three-dimensional feature map <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a vehicle pose <NUM> relative to a first origin point within the three-dimensional feature map <NUM>. At block <NUM>, the method <NUM> includes detecting, at the data processing hardware <NUM>, the reference points <NUM> within one or more trailer images <NUM> received from the trailer camera 132d, 132e, 132f. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a trailer camera location <NUM> of the trailer camera 132d, 132e, 132f relative to the three-dimensional feature map <NUM>. Additionally, at block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a first trailer camera pose <NUM> of the trailer camera 132d, 132e, 132f relative to the vehicle pose <NUM> within the three-dimensional feature map <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a trailer reference point <NUM> based on the trailer camera location <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a trailer reference pose <NUM> of the trailer reference point <NUM> relative to the trailer camera location <NUM>. Additionally, at block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, a second trailer camera pose <NUM> of the trailer camera 132d, 132e, 132f relative to the trailer reference pose <NUM>. At block <NUM>, the method <NUM> includes determining, at the data processing hardware <NUM>, extrinsic parameters <NUM> of the trailer camera 132d, 132e, 132f relative to the trailer reference pose <NUM>.

In some implementations, the trailer reference point <NUM> is at a predefined distance from the trailer camera 132d, 132e, 132f. The trailer reference point <NUM> may overlap with the trailer camera location <NUM>. The extrinsic parameters <NUM> may define the position of a center of the trailer camera 132d, 132e, 132f and a heading of the trailer camera 132d, 132e, 132f.

In some examples, the method <NUM> also includes associating an identifier with each one of the reference points <NUM> identified in the one or more vehicle images <NUM>, where detecting the reference points <NUM> within one or more trailer images <NUM> received from the trailer camera 132d, 132e, 132f includes determining the identifier associated with each one of the reference points <NUM>.

The method <NUM> may also include transmitting the extrinsic parameters <NUM> of the trailer camera 132d, 132e, 132f to one or more vehicle systems causing the vehicle system to execute a behavior using the extrinsic parameters <NUM>.

In some implementations, determining a three-dimensional feature map <NUM> from one or more vehicle images <NUM> received from a camera 132a, 132b, 132c supported by the tow vehicle <NUM> includes executing at least one of: a Visual Odometry (VO) algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, or a Structure from Motion (SfM) algorithm.

Before detecting, at the data processing hardware <NUM>, the reference points <NUM> within one or more trailer images <NUM> received from the trailer camera 132d, 132e, 132f, the method <NUM> may include transmitting instructions to a user interface <NUM>, the instruction causing a user display to prompt a driver of the vehicle <NUM> to drive the vehicle <NUM> in a forward direction FV, FT, and receiving images <NUM> from the trailer camera 132d, 132e, 132f after the vehicle <NUM> has traveled a predefined distance.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

Claim 1:
A method for calibrating extrinsic parameters (<NUM>) of a trailer camera (132d, 132e, 132f), the trailer camera (132d, 132e, 132f) supported by a trailer (<NUM>) attached to a tow vehicle (<NUM>), the method comprising:
determining, at data processing hardware (<NUM>), a three-dimensional feature map (<NUM>) from one or more vehicle images (<NUM>) received from a camera (132a, 132b, 132c) supported by the tow vehicle (<NUM>);
identifying, at the data processing hardware, reference points (<NUM>) within the three-dimensional feature map (<NUM>);
detecting, at the data processing hardware, the reference points (<NUM>) within one or more trailer images (<NUM>) received from the trailer camera (132d, 132e, 132f) after the vehicle (<NUM>) and the trailer (<NUM>) moved a predefined distance in the forward direction;
determining, at the data processing hardware, a trailer camera location (<NUM>) of the trailer camera (132d, 132e, 132f) relative to the three-dimensional feature map (<NUM>);
determining, at the data processing hardware, a trailer reference point (<NUM>) based on the trailer camera location (<NUM>); and
determining, at the data processing hardware, extrinsic parameters (<NUM>) of the trailer camera (132d, 132e, 132f) relative to the trailer reference point (<NUM>).