Patent ID: 12233858

DETAILED DESCRIPTION

This disclosure teaches a trailer monitoring system for a vehicle equipped to tow a trailer. The trailer monitoring system is configured to detect one or more irregularities of the trailer during towing, for example, an imminent collision of the trailer with a nearby object such as a nearby vehicle, departure of the trailer from the lane in which the vehicle is traveling, and/or swaying of the trailer within the lane in which the vehicle is traveling or across multiple lanes. The trailer monitoring system may detect an irregularity by receiving, from one or more monocular cameras mounted to the vehicle, one or more monocular camera images of at least a portion of the trailer. Based on the monocular camera image(s), the trailer monitoring system can generate one or more depth maps and identify the irregularity based on the depth map(s).

A representative passenger vehicle100is shown inFIG.1towing a representative trailer102. In this description, uses of “front,” “forward” and the like, and uses of “rear,” “rearward” and the like, refer to the longitudinal directions of the vehicle100. “Front,” “forward” and the like refer to the front (fore) of the vehicle100, while “rear,” “rearward” and the like refer to the back (aft) of the vehicle100. Uses of “side,” “sideways,” “transverse” and the like refer to the lateral directions of the vehicle100, with “driver's side” and the like referring to the left side of the vehicle100, and “passenger side” and the like referring to the right side of the vehicle100. The vehicle100may be any suitable type of vehicle100. As shown, the vehicle100is a pickup truck, but the vehicle100can be any other type of vehicle, such as a sedan, an SUV, a motorhome, etc. The vehicle100is configured to tow the trailer102. As shown, the trailer102is a storage trailer, but the trailer102can be any other type of trailer, such as a camper, a boat trailer, a flatbed trailer, etc.

The vehicle100includes an exterior and a number of interior compartments. The compartments include a passenger compartment, an engine compartment (and, in the illustrated pickup truck configuration of the vehicle100, an open-topped bed for carrying cargo). Among other things, the vehicle100may include seats, a dash assembly, an instrument panel, controls and the like housed in the passenger compartment. Additionally, the vehicle100may include an engine, a motor, a transmission, and the like, as well as other powertrain components, such as wheels, housed in the engine compartment and elsewhere in the vehicle100. The wheels support the remainder of the vehicle100on the ground. One, some or all of the wheels are powered by the remainder of the powertrain components to drive the vehicle100along the ground.

The vehicle100includes one or more vehicle systems104operable to perform vehicle functions. In addition to the vehicle systems104, the vehicle100includes a sensor system110as well as one or more processors120, a memory122, and a control module124to which the vehicle systems104and the sensor system110are communicatively connected. The sensor system110is operable to detect information about the vehicle100. The processors120, the memory122, and the control module124together serve as one or more computing devices118whose control module124is employable to orchestrate the operation of the vehicle100, in whole or in part. Specifically, the control module124operates the vehicle systems104based on information about the vehicle100. Accordingly, as a prerequisite to operating the vehicle systems104, the control module124gathers information about the vehicle100, including the information about the vehicle100detected by the sensor system110. The control module124then evaluates the information about the vehicle100and operates the vehicle systems104based on its evaluation.

The vehicle systems104are part of, mounted to, or otherwise supported by the body. The vehicle systems104may be housed, in whole or in part, in any combination of the passenger compartment, the engine compartment, or elsewhere in the vehicle100. Each vehicle system104includes one or more vehicle elements. On behalf of the vehicle system104to which it belongs, each vehicle element is operable to perform, in whole or in part, any combination of vehicle functions with which the vehicle system104is associated. It will be understood that the vehicle elements, as well as the vehicle system104to which they belong, may but need not be mutually distinct.

The vehicle systems104include an energy system106and a propulsion system108. The energy system106and the propulsion system108are connected to one another. Moreover, the drivetrain is mechanically connected to the propulsion system108. The propulsion system108and the drivetrain together serve as a powertrain for the vehicle100. The energy system106is operable to perform one or more energy functions, including but not limited to storing and otherwise handling energy. The propulsion system108is operable to perform one or more propulsion functions using energy from the energy system106, including but not limited to powering the wheels.

As a part of the sensor system110, the vehicle100includes one or more vehicle sensors112and one or more environment sensors114. The vehicle sensor(s)112monitor the vehicle100in real-time. The vehicle sensor(s)112, on behalf of the sensor system110, are operable to detect information about the vehicle100, including information about user requests and information about the operation of the vehicle100. For example, the vehicle sensor(s)112can be configured to detect and/or acquire data about various operating parameters of the vehicle100. For example, the vehicle sensor(s)112can include one or more speedometers, one or more gyroscopes, one or more accelerometers, one or more inertial measurement units (IMUs), one or more wheel sensors, one or more steering angle sensors, one or more controller area network (CAN) sensors, and the like. Relatedly, among information about the operation of the vehicle100, the sensor system110is operable to detect the location and motion of the vehicle100, including its speed, acceleration, orientation, rotation, direction, and the like, the movement of the wheels, the steering angle, and the operational statuses of one, some, or all of the vehicle systems104.

The environment sensor(s)114can be configured to detect, determine, assess, monitor, measure, quantify, acquire, and/or sense data or information about the external environment in which the vehicle100is located or one or more portions thereof. The environment sensor(s)114can include one or more exterior cameras and one or more exterior sensors, such as temperature sensors, weather sensors, LIDAR, RADAR, etc. The exterior cameras can include one or more monocular cameras116. The environment sensor(s)114can be located on an exterior of the vehicle100or can be located in any other suitable location on the vehicle100. Using the environment sensor(s)114, the vehicle systems104can determine information about the external environment of the vehicle100. For example, the vehicle systems104can detect one or more objects in the external environment of the vehicle100.

The vehicle systems104, the sensor system110, the processors120, the memory122, and the control module124may be leveraged to implement a monocular depth estimation (MDE) system126. In the vehicle100, the vehicle systems104, the sensor system110, the processors120, the memory122, and the control module124leveraged to implement the MDE system126may be part of one or more other control systems typical of vehicles or may be dedicated to the MDE system126. The MDE system126will be described with reference to a representative monocular camera image200as shown inFIG.2A, a depth map202ofFIG.2B, a depth map300ofFIG.3, depth maps400,402, and404ofFIGS.4A-4C, and depth maps500,502, and504as shown inFIGS.5A-5C. As described with respect toFIGS.2B-5C, the MDE system126may be configured to generate depth map(s) of at least a portion of the external environment of the vehicle100based on information received from the sensor system110. More specifically, the MDE system126may be configured to generate depth map(s) based, at least in part, on information received by the monocular camera(s)116. The depth map(s) may then be used as an input to other vehicle systems104, for example, a trailer monitoring system128, as described in further detail below.

As mentioned above, the vehicle100may include one or more monocular cameras116. The monocular camera(s)116may be mounted to the exterior of the vehicle100at the front of the vehicle100, at the rear of the vehicle100, and/or at any other location on the vehicle100. For example, the monocular camera(s)116can be mounted to the rear of the vehicle100and/or one or more side view mirrors130(FIG.1) of the vehicle100(for example, a left side view mirror130A and a right side view mirror130B) and can have a field of view of at least a portion of the trailer102. The monocular camera(s)116are configured to capture one or more monocular camera images of the trailer102. Referring now toFIG.2A, an example of a monocular camera image200is shown. The monocular camera image200shows a view of a front of the trailer102. The monocular camera image200may be a color image typical of other types of vehicle-mounted cameras. Referring now toFIG.2B, an example of a depth map202is shown. The depth map202is generated based on the monocular camera image200and is a monochrome image in which the pixel values of the depth map202are proportional to the distance between the monocular camera116and the object in the monocular camera image200(e.g., the trailer102).

In addition to the MDE system126, the vehicle systems104, the sensor system110, the processor(s)120, the memory122, and the control module124may be leveraged to implement a trailer monitoring system128based on the MDE system126. In the vehicle100, the vehicle systems104, the sensor system110, the processor(s)120, the memory122, and the control module124leveraged to implement the trailer monitoring system128may be part of one or more other control systems typical of vehicles or may be dedicated to the trailer monitoring system128. The trailer monitoring system128may be configured to identify one or more trailer irregularities based only on the depth map(s). This may be advantageous over other methods of identifying trailer irregularities because reliance only on the depth map(s) eliminates the need for prior knowledge of the dimensions, weight, aerodynamics, etc. of the trailer102and/or the kinematics between the vehicle100and the trailer102.

Based on the depth map(s), the trailer monitoring system128may be configured to identify one or more contours of the trailer102. More specifically, based on each depth map202, a point cloud may be generated including an array of points that each correspond to the distance from the trailer102to the monocular camera116. Using a direct transformation, the point cloud can be used to detect the contours. Referring back toFIG.1, the contours can include one or more faces, for example, a front face132corresponding to the front of the trailer102, a left face134corresponding to the left side of the trailer102, and a right face136corresponding to the right side of the trailer102. The contours can also include one or more edges, for example, two front side edges and two rear side edges. The two front side edges can include a front left side edge138and a front right side edge140. The two rear side edges can include a rear left side edge142and a rear right side edge144. The edges can also include one or more top edges. The top edges can include a top left side edge146and a top right side edge148. The edges can also include one or more bottom edges. The bottom edges can include a front bottom edge150, a left side bottom edge152, and a right side bottom edge154. Using the contours detected in the depth map(s), the trailer monitoring system128may be configured to detect one or more trailer irregularities during towing.

In some instances, the trailer irregularity can be identified using a single depth map. For example, referring toFIG.3, the trailer irregularity can be an imminent collision between the trailer102and an object312near the trailer102. The object312may be a nearby vehicle located in a lane adjacent the vehicle100and the trailer102, as shown, or any other object. In some instances, the object can be a fixed feature of the environment, such as one or more curbs, light poles, traffic lights, road signs, etc. The trailer monitoring system128can identify an imminent collision between the trailer102and the object312by comparing the depth of one or more of the contours to the depth of the object312. When the depth of a contour is similar to the depth of the object312, an imminent collision may be identified. For example, when the trailer102departs the lane towards the adjacent right lane, and there is another vehicle traveling in the adjacent right lane, the trailer monitoring system128can compare the depth of the front right side edge140and/or the depth of the rear right side edge144to the depth of the object312, and, if the depths are the same or similar, the trailer monitoring system128can identify an imminent collision with the object312. In another example, when the trailer102departs the lane towards an adjacent left lane, and there is another vehicle traveling in the adjacent left lane, the trailer monitoring system128can compare the depth of the front left side edge138and/or the depth of the rear left side edge142to the depth of the other vehicle, and, if the depths are the same, the trailer monitoring system128can identify an imminent collision between the trailer102and the other vehicle. In some instances, the object312may be the vehicle100itself, and the imminent collision may be an imminent collision between the trailer102and the vehicle100itself. This may occur when the vehicle100is making a sharp turn.

In some instances, the trailer irregularity may be identified by comparing the depth of one or more of the contours over a plurality of consecutive or otherwise successive depth maps. For example, the monocular camera(s)116may be configured to capture a monocular camera image every second, for example, and thus, generate a depth map every second. Using consecutive or otherwise successive depth maps, the trailer irregularity can be identified by detecting a change in the depth of one or more of the contours over time as the trailer102is moving. Referring toFIGS.4A-4C, the trailer irregularity can be departure of the trailer102from the lane in which the vehicle100is traveling (lane departure), assuming the vehicle100itself is not switching lanes. The trailer monitoring system128can identify lane departure when the depth of one or more of the contours decreases over successive depth maps400,402, and404. For example, as shown inFIGS.4A-4C, when the trailer102departs the lane such that the trailer102moves into the adjacent right lane, the depth of each of the front face132, the right face136, the front right side edge140, the right side bottom edge154, and the rear right side edge144all decrease over the successive depth maps400,402, and404, and the trailer monitoring system128can identify that the trailer102has departed the lane and at least a portion of the trailer102is in the adjacent right lane. More specifically, referring toFIG.4A, which shows a first successive depth map400, the trailer102is shown traveling in the same lane as the vehicle100, and is not departing the lane.FIG.4B, which shows a second successive depth map402, shows the trailer102moving into the adjacent lane. InFIG.4B, the depth of the front face132, the right face136, the front right side edge140, the right side bottom edge154, and/or the rear right side edge144are less than they are inFIG.4A, and thus, the trailer monitoring system128can identify lane departure.FIG.4C, which shows a third successive depth map404, shows the trailer102moving further into the adjacent lane. The depth of the front face132, the right face136, the front right side edge140, the right side bottom edge154, and/or the rear right side edge144are less than they are inFIG.4B, and thus, the trailer monitoring system128can identify lane departure.

Referring now toFIGS.5A-5C, in some instances, the trailer irregularity can be swaying of the trailer102(trailer sway). The trailer monitoring system128can identify trailer sway when the depth of one or more of the contours fluctuates (e.g., increases and decreases repeatedly) over successive depth maps500,502, and504. For example, the trailer monitoring system128can identify trailer sway when the depth of the front face132fluctuates, when the depth of the front right side edge140fluctuates, and/or when the depth of the front left side edge138fluctuates. For example, referring toFIG.5B, the depth of the left face134, the left front side edge138, the rear left side edge142, and/or the left side bottom edge152may fluctuate when the trailer is swaying toward the left. In another example, referring toFIG.5C, the depth of the right face136, the front right side edge140, the rear right side edge144, and/or the right side bottom edge154may fluctuate when the trailer is swaying to the right. When the vehicle100includes a monocular camera116mounted to the left side view mirror130A of the vehicle100, the trailer monitoring system128can identify trailer sway when the depth of the right side of the trailer102fluctuates, for example, when the depth of the right face136fluctuates, when the depth of the top right side edge148fluctuates, and/or when the depth of the rear right side edge144fluctuates. When the vehicle100includes a monocular camera116mounted to the right side view mirror130B of the vehicle100, the trailer monitoring system128can identify trailer sway when the depth of the left side of the trailer102fluctuates, for example, when the depth of the left face134fluctuates, when the depth of the top left side edge146fluctuates, and/or when the depth of the rear left side edge142fluctuates.

In any of the above-described examples, upon the detection of trailer irregularity, the trailer monitoring system128may be configured to issue a warning or make a corrective action by taking control of the vehicle100. For example, when the trailer irregularity is trailer sway, the trailer monitoring system128can issue a warning to the driver and/or take control of the steering and/or speed of the vehicle100to reduce the trailer sway. When the trailer irregularity is lane departure, the trailer monitoring system128can issue a warning to the driver, take control of the steering and/or speed of the vehicle100to bring the trailer102back in the lane, and/or issue a warning to nearby vehicles, such as by honking or flashing the lights of the vehicle100and/or the trailer102. When the trailer irregularity is an imminent collision, the trailer monitoring system128can issue a warning to the driver, take control of the steering and/or speed of the vehicle100to prevent the collision, and/or issue a warning to nearby vehicles, such as by honking or flashing the lights of the vehicle100and/or the trailer102.

Referring now toFIG.6, an example of the MDE system126is shown. The MDE system126may be configured to receive an input600and generate an output650. The input600may be the monocular camera image200. The monocular camera image200can be a color image taken by a monocular camera116. The output650can be the identification of one or more contours of the trailer102. The MDE system126includes a monocular depth estimation (MDE) module610, a road segmentation and extraction (RSE) module620, a feature extraction module630, and one or more processor(s)640. The processor(s)640may be the processor(s)120ofFIG.1or any other suitable processor(s). The MDE module610, the RSE module620, and/or the feature extraction module630can be components of the processor(s)640or may be components of one or more other processors. The MDE module610is configured to receive the input600(i.e., the monocular camera image200) and generate a depth map202using machine learning or any other suitable method. As described above, the depth map202is a grayscale image in which each pixel value is proportional to the distance to the monocular camera116. The RSE module620is configured to receive the monocular camera image200and/or the depth map202and detect, segment out, and extract the part of the monocular camera image200and/or the depth map202corresponding to the road. The feature extraction module630may receive the input600and may be configured to detect features (e.g., one or more contours of the trailer102) in the monocular camera image200. The processor(s)640may function as a decision system based on the input600(i.e., the monocular camera image200), the depth map202, the road, and/or the features to generate the output650. The output650may be the detection of one or more contours of the trailer102.

With reference once again toFIG.1, as noted above, the processor(s)120, the memory122, and the control module124together serve as the computing device(s)118whose control module124orchestrates the operation of the vehicle100, including but not limited to the operation of the vehicle systems104. The control module124may be a dedicated control module for the trailer monitoring system128and/or the MDE system126. Relatedly, as part of a central control system, the vehicle100may include a global control unit (GCU) to which the control module124is communicatively connected. Alternatively, the control module124may be a global control module. Relatedly, as part of a central control system, the vehicle100may include a global control unit (GCU) to which the control module124belongs. Although the vehicle100as shown includes one control module124, it will be understood that this disclosure is applicable in principle to otherwise similar vehicles including multiple control modules. Moreover, although the control module124is shown as part of the vehicle100, it will be understood that the control module124may be located offboard the vehicle100.

The processor(s)120may be any components configured to execute any of the processes described herein or any form of instructions to carry out such processes or cause such processes to be performed. The processor(s)120may be implemented with one or more general-purpose or special-purpose processors. Examples of suitable processors include microprocessors, microcontrollers, digital signal processors, or other forms of circuitry that execute software. Other examples of suitable processors include, without limitation, central processing units (CPUs), array processors, vector processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), application specific integrated circuits (ASICs), programmable logic circuitry, or controllers. The processor(s)120may include at least one hardware circuit (e.g., an integrated circuit) configured to carry out instructions contained in program code. In arrangements where there are multiple processors, the processors may work independently from each other or in combination with one another. Moreover, although the processor(s)120are shown as part of the vehicle100, it will be understood that the processor(s)120may be located offboard the vehicle100.

The memory122is a non-transitory computer readable medium. The memory122may include volatile or nonvolatile memory, or both. Examples of suitable memory include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination of these. The memory122includes stored instructions in program code. Such instructions are executable by the processor(s)120or the control module124. The memory122may be part of the processor(s)120or the control module124or may be communicatively connected to the processor(s)120or the control module124. Generally speaking, the control module124includes instructions that may be executed by the processor(s)120. The control module124may be implemented as computer readable program code that, when executed by the processor(s)120, executes one or more of the processes described herein. Such computer readable program code may be stored in the memory122. The control module124may be part of the processor(s)120or may be communicatively connected to the processor(s)120.

Now that the various potential systems, devices, elements, and/or components have been described, various methods, including various possible steps of such method, will now be described. The methods described may be applicable to the arrangements described above, but it is to be understood that the methods can be carried out with other suitable systems and arrangements. Moreover, the methods may include other steps not shown here, and the methods are not limited to including every step shown. The blocks illustrated here as part of the methods are not limited to the particular chronological order. Indeed, some of the blocks may be performed in a different order than what is shown and/or at least some of the blocks shown can occur simultaneously.

Referring toFIG.7, an example of a method700for generating a depth map is shown. The method700may begin in operation702. In operation704, the method700may include receiving a monocular camera image200from a sensor system110of a vehicle100. In operation706, the method700may include generating a depth map202based on the monocular camera image200. In operation708, the method may include detecting, segmenting out, and extracting a part of the depth map202corresponding to a surface upon which the vehicle100and the trailer102are traveling. In operation710, the method700may include detecting and extracting features from the depth map202. The method700may be used to detect one or more contours of the trailer102. Referring now toFIG.8, an example of a method800of operating the trailer monitoring system128is shown. The method800may begin in operation802. In operation804, the method800may include receiving, from at least one monocular camera116mounted to the vehicle100, at least one monocular camera image200of at least a portion of the trailer102. In operation806, the method800may include generating at least one depth map202based on the at least one monocular camera image200. In operation808, the method800may include identifying one or more contours of the trailer102based on the depth map202. In operation810, the method800may include identifying at least one trailer irregularity based on the depth of the one or more contours. If a trailer irregularity is identified, the method800can proceed to operation812and/or operation814. In operation812, the method800includes issuing a warning to a driver of the vehicle100and/or a nearby vehicle when the trailer irregularity is identified. In operation814, the method800includes taking control of the steering of the vehicle100and/or the speed of the vehicle100when the trailer irregularity is identified.

While recited characteristics and conditions of the invention have been described in connection with certain embodiments, it is to be understood that the invention is not to be limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as is permitted under the law.