Method for autonomously parking a motor vehicle

A system is provided that includes a computer including a processor and a memory. The memory includes instructions such that the processor is programmed to: receive an image depicting a parking spot, determine a length of the parking spot based on a classified endpoint of the parking spot, compare the length to an average length, and determine an endpoint of the parking spot when the length is less than the average length, wherein the determined endpoint is distal to the classified endpoint.

FIELD

The disclosure relates generally to autonomous driver assistance systems for motor vehicles, and more particularly to autonomous driver assistance systems for parking a motor vehicle.

BACKGROUND

Smart car technologies such as free-ranging on grid navigation, as well as parking guidance and information systems, aid in the prevention of human error when drivers operate a vehicle. Such technologies have been used to improve navigation of roadways, and to augment the parking abilities of motor vehicle drivers while the drivers are present within the motor vehicle. For example, rear view camera systems and impact alert systems have been developed to assist the operator of the motor vehicle while parking to avoid collisions. In addition, autonomous parking systems have been developed that autonomously park the motor vehicle in a parallel parking spot once the operator of the motor vehicle has positioned the motor vehicle in a predefined location proximate the parking spot.

While these systems are useful for their intended purpose, they require that the operator of the motor vehicle locate the parking spot and drive to the parking spot. Thus, there is a need in the art for improved smart car technologies that utilize preexisting infrastructure to autonomously park a motor vehicle. Moreover, there is a need to implement automatic parking systems in motor vehicles that do not increase cost, and which also increase the accuracy and robustness of parking systems.

SUMMARY

A system is provided that includes a computer including a processor and a memory. The memory includes instructions such that the processor is programmed to: receive an image depicting a parking spot, determine a length of the parking spot based on a classified endpoint of the parking spot, compare the length to an average length, and determine an endpoint of the parking spot when the length is less than the average length, wherein the determined endpoint is distal to the classified endpoint.

In other aspects, the processor is further programmed to actuate a vehicle system of a vehicle using the determined endpoint and the classified endpoint.

In other aspects, the processor is further programmed to classify the classified endpoint via a convolutional neural network.

In other aspects, the convolutional neural network comprises at least one of a single shot detector, a You-Only-Look-Once network, or a Fast-RCNN network.

In other aspects, the system includes a sensor configured to capture images representative of a vehicle environment.

In other aspects, the sensor is configured to capture Red-Green-Blue (RGB) images.

In other aspects, the classified endpoint and the determined endpoint comprise an x-coordinate and a y-coordinate within the image.

In other features, the image comprises a stitched image.

In other features, the processor is further programmed to stitch the stitched image based on a plurality of Red-Green-Blue images received from a plurality of sensors.

A method is provided that includes receiving an image depicting a parking spot, determining a length of the parking spot based on a classified endpoint of the parking spot, comparing the length to an average length, determining an endpoint of the parking spot when the length is less than the average length, wherein the determined endpoint is distal to the classified endpoint, and realigning the determined endpoint with an endpoint of an adjacent parking spot such that the parking spot is oriented with the adjacent parking spot.

In other aspects, the method includes actuating a vehicle system of a vehicle using the determined endpoint and the classified endpoint.

In other aspects, the classified endpoint is classified via a convolutional neural network.

In other aspects, the convolutional neural network comprises at least one of a single shot detector, a You-Only-Look-Once network, or a Fast-RCNN network.

In other aspects, the image is captured by a sensor configured to capture images representative of a vehicle environment.

In other aspects, the sensor is configured to capture Red-Green-Blue (RGB) images.

In other aspects, the classified endpoint and the determined endpoint comprise an x-coordinate and a y-coordinate within the image.

In other features, the image comprises a stitched image.

In other features, the method includes stitching the stitched image based on a plurality of Red-Green-Blue images received from a plurality of sensors.

A system is disclosed that includes a computer including a processor and a memory. The memory includes instructions such that the processor is programmed to: receive a stitched image depicting a parking spot, the stitched image based on a plurality of images captured by a plurality of sensors, determine a length of the parking spot based on a classified endpoint of the parking spot, compare the length to an average length, determine an endpoint of the parking spot when the length is less than the average length, wherein the endpoint is distal to the endpoint, realign the determined endpoint with an endpoint of an adjacent parking spot such that the parking spot is oriented with the adjacent parking spot, and actuate a vehicle system of a vehicle using the endpoint and the classified endpoint.

In other aspects, the classified endpoint is classified via a convolutional neural network.

Further aspects, examples, and advantages will become apparent by reference to the following description and appended drawings wherein like reference numbers refer to the same component, element or feature.

DETAILED DESCRIPTION

With reference toFIG.1, an autonomous park assisting system according to the principles of the present disclosure is indicated by reference number10. The autonomous park assisting system10is used with an exemplary motor vehicle12and an exemplary mobile device14. The motor vehicle12is illustrated as a passenger vehicle, however, the motor vehicle12may be a truck, sport utility vehicle, van, motor home, or any other type of vehicle without departing from the scope of the present disclosure. The mobile device14is preferably a mobile phone, however, the mobile device14may be a mobile computer, laptop, tablet, smart watch, or any other device in wireless communication with the motor vehicle12. The autonomous valet system10runs an autonomous valet method or application, as will be described in greater detail below.

The autonomous park assisting system10is operable to autonomously park and un-park the motor vehicle12. The autonomous park assisting system10may have various configurations without departing from the scope of the present disclosure but generally includes a sensor sub-system16and a communication sub-system18each in communication with a controller20. The controller20communicates with a vehicle control system22. The sensor sub-system16includes a plurality of sensors24A-D mounted along the periphery of the motor vehicle12. In the example provided, the sensors24A through24D are located at the front, left, right, and rear of the motor vehicle12, respectively, to provide 360 degrees of overlapping coverage. However, it should be appreciated that the sensor sub-system16may have any number of sensors24without departing from the scope of the disclosure. Each of the sensors24A-D is operable to collect or sense information in a predefined area surrounding the motor vehicle12. Information from the sensors24A through24D is communicated to the controller20. In a preferred embodiment, the sensors24A through24D are cameras that collect images and/or video data. For example, the sensors24A through24D may be infra-red cameras, RGB cameras, dual (side-by-side) cameras, time-of-flight cameras, or log profile cameras.

The communication sub-system18includes a receiver/transmitter operable to receive and/or transmit wireless data to the mobile device14. The wireless data is communicated to the controller20. In addition, the communication sub-system18may communicate with other vehicles (vehicle-to-vehicle communication), infrastructure such as a parking lot (vehicle-to-infrastructure), and may receive Global Positioning System (GPS) data.

The controller20is a non-generalized, electronic control device having a preprogrammed digital computer or processor, memory or non-transitory computer readable medium used to store data such as control logic, instructions, image data, lookup tables, etc., and a plurality of input/output peripherals or ports. The processor is configured to execute the control logic or instructions. The controller20may have additional processors or additional integrated circuits in communication with the processor, such as perception logic circuits for analyzing the sensor data.

The controller20may optionally communicate with a human machine interface (HMI)26. The HMI26is disposed within the cabin of the motor vehicle12and is preferably a touch screen accessible by an operator of the motor vehicle12. However, the HMI26may be any haptic, verbal, or gesture control system without departing from the scope of the present disclosure. The HMI26may be used to activate and control the autonomous park assisting system10. Additionally, the mobile device14may be used to activate and control the autonomous park assisting system10.

The vehicle control system22includes any systems that implement the autonomous park assisting functions which include parking the motor vehicle12. For example, the vehicle control system22may include a braking control system, throttle control system, steering control system, body control system, etc. The vehicle control system22may also include any advanced driver assistance system (ADAS) functions that automate, adapt, or enhance vehicle systems in order to increase vehicle safety and/or operator driving performance. For example, the vehicle control system22may include ADAS technologies that alert the driver to potential problems or to avoid collisions by implementing safeguards, such as autonomously controlling the motor vehicle12. The vehicle control system22may also include ADAS features that enhance certain systems, such as automated lighting, adaptive cruise control, automated braking, or improved blind spot elimination using camera technology. Finally, it should be appreciated that the vehicle control system22may be part of the autonomous valet system10without departing from the scope of the present disclosure.

With reference toFIG.2, an exemplary parking area is indicated by reference number30. The parking area30includes a plurality of parking spots32. It should be appreciated that the parking area30may have any configuration, may be a parking structure, and may have any number of parking spots32without departing from the scope of the present disclosure. Each parking spot32may be defined by corresponding parking spot markings34. In an example embodiment, a parking spot32A may be defined by a first parking spot marking34A, a second parking spot marking34B, and a third parking spot marking34C. Each sensor24A through24D can capture images within a corresponding field-of-view (FOV) exterior to the vehicle12of the respective sensors24A through24D. As the vehicle12is traveling within the parking area30, one or more sensors24A through24D may capture images including one or more markings34defining one or more parking spots32. For example, the motor vehicle12may be located in the parking area30by positioning the motor vehicle12in a predefined starting location or parking spot or by GPS coordinates. At step56the motor vehicle12communicates with the parking area infrastructure to receive a map of the parking area30. The map may be defined as a Cartesian coordinate system with x and y coordinates.

With reference toFIG.3, each sensor24A through24D provides respective images to an image stitching module40. The image stitching module40can be executed by the controller20. During operation, the sensor24A provides an image having a first FOV exterior to the vehicle12to the image stitching module40. In this example, the sensor24B provides an image having a second FOV exterior to the vehicle12to the image stitching module40. The sensor24C and the sensor24D also provide an image having a third FOV and a fourth FOV, respectively, exterior to the vehicle12to the image stitching module40.

The image stitching module40receives images from each of the sensors24A-D and generates a stitched image using the received images. The stitched image can depict objects captured from each of the sensors24A through24D in a single image. Stitched images may depict a panoramic view, which may be referred to as a surround view, a top view, or a bird's eye view. The image stitching module40can apply conventional image stitching processes to the received images to connect captured images to generate a single stitched image depicting a panoramic view around the vehicle12.

The image stitching module40outputs the generated stitched image to a detection module42. The detection module42can be executed by the controller20. The detection module42uses the stitched image to classify endpoints48, such as endpoints48A,48B,48C,48D corresponding to one or more parking spots32as shown inFIG.5. For example, the detection module42classifies endpoints48as an edge of the parking spot markings34A,34B. The detection module42can use a neural network, such as a convolutional neural network (CNN)50or any other suitable neural network, to classify endpoints48corresponding to the one or more parking spots using the parking spot markings34. It is understood that the detection module42may also use suitable image processing techniques to classify endpoints48of the parking spots. The endpoints48can represent two-dimensional coordinates, e.g., x- and y-coordinates, within the stitched image representing edges of the parking spot markings34A,34B,34C,34D. Within a stitched image, an endpoint48of a parking spot marking34may be defined as a subset of pixels within the stitched image that represent non-parking spot markings that are adjacent to parking spot markings34A,34B,34C,34D. For example, the parking spot markings34may be determined when pixel hue, e.g., color, changes from one hue, e.g., white representing parking spot markings34, to another hue, e.g., gray representing areas of the parking lot mot marked.

An example embodiment of the CNN50is illustrated inFIG.4. The CNN50is configured as a single shot detector (SSD) in the example provided. However, the CNN50may be a You-Only-Look-Once (YOLO) network or a Fast-RCNN without departing from the scope of the present disclosure. The stitched image is communicated to the CNN50and is trained to predict endpoints48for parking spot markings34from the stitched image.

With reference toFIG.4, the CNN50generally includes an input layer54, a number of convolution and pooling layers (CPUs)56, a fully connected layer58, and a non-maximum suppression layer60, each stacked together with each other. The input layer54loads the raw input data of the stitched image52to be processed by the CNN50. The stitched image is usually defined by the image width, height, and the number of channels per pixel. In Red/Green/Blue (RGB) input images, the number of channels is three to hold the individual RGB values.

The CPUs56each contain convolution layers and pooling layers. Convolution layers detect the presence of specific features or patterns in the original data by applying a convolution operation between a filter (the weights of the network which have been previously trained, as described above) and the stitched image. These features and patterns are given a confidence vote and used to identify endpoints48A,48B representing an edge, e.g., a parking spot marking edge, of the parking spot markings34A,34B, respectively. The parking lot lane markings34are categorized to generate a number of feature maps. The first CPUs56learn to represent data in a very simple form such as horizontal and vertical lines and simple blobs of colors. The following CPUs56capture more complex shapes such as circles, rectangles, triangles, and the like. The subsequent layers of the CPUs56detect complex combinations of features from the previous layer or layers to form a more meaningful representation such as wheels, parking spot markings, faces, grids, and the like. The output of the convolution and activation layer for a given filter is a feature map. The pooling layers represent the process of reducing the spatial size of the stitched image. Examples include max pooling and average pooling. For example, if a resolution of the input image to a pooling layer is 640×540, then the resolution of the output image or feature map is 320×270. The outputs, including feature maps, from the input layer54and each CPL56are communicated to the fully connected layer58.

The fully connected layer58receives the filtered images, i.e. the feature maps, from the input layer54and the fully connected layer58and translates these into votes or confidence levels for edges of the parking spot lane markings34. The fully connected layer58generates multiple parking spot lane marking edge predictions, each having a confidence level associated with strength of the prediction.

The non-maximum suppression layer60receives the multiple lane bounding box predictions and generates multiple final parking spot lane marking edge predictions. The multiple final parking spot lane marking edge prediction can have an aggregate high confidence value, for example of between 98% and 100%, that the parking spot lane marking edges are properly classified and localized.

An estimation module44receives the output of the CNN50as input, and the estimation module44generates parking spot estimations using the output of the CNN50. The estimation module44estimates a length L (shown as L1through L3for respective parking spots32A through32C) for each parking spot32based on corresponding endpoints48. While described within the context of calculating a length of a parking spot, it is understood that techniques described herein can be applied to calculating a width for a parking spot.

As the vehicle12traverses parking area30, the sensors24A through24D capture additional images of the surrounding environment. As the vehicle12approaches the parking spots32A through32C, the sensors24A through24D provide additional images to the detection module42to classify endpoints48A through48D for visible parking spots32A through32C.

As shown inFIG.5, based on a relative vehicle12position within the parking area30, the sensors24A through24D may capture images that only allow the detection module42to classify endpoints48for parking spot32A that are proximal to the vehicle12, e.g., endpoints48A and48B. The estimation module44can initially determine a valid parking spot for parking spot32A based on length L1and/or L2of parking spots32B,32C. The estimation module44can determine the lengths L1and/or L2by estimating a distance between endpoints48A and48C of the corresponding parking spots32B,32C. The length L1and/or L2may correspond to a length of the parking spot marking34A. In an example implementation, the estimation module44can estimate L1and/or L2by determining a ratio that calculates a number of pixels per given metric, e.g., inches, feet, meters, etc.

Based on a position of the vehicle12relative to the parking spot32A since the estimation module44has not received endpoints48C and48D for parking spot32A from the determination module42. The estimation module44determines the distal pseudo endpoints by applying a predefined default length to endpoints48A and48B. A pseudo length, e.g., a length similar to L1and/or L2, for parking spot34A can be initially set to the predefined default length. In an example implementation, the predefined default length is preset by a vehicle manufacturer or a vehicle supplier. The estimation module44can then calculate a mean length “Lm” by determining an arithmetic mean for the lengths L1, L2, and the pseudo length according to equation 1:
Lm=(sum of total number of lengths)/(total number of lengths)  Eqn. 1

As the vehicle traverses the parking area30, the determination module42can determine endpoints48C and48D using the techniques described above. Based on the determination, the determined endpoints48C and48D may differ from the distal pseudo endpoints calculated using the predefined default length. Using the determined endpoints48C and48D, the estimation module44can calculate length L3based on the pixel coordinates of endpoints34A,34B,34C, and/or34D.

The estimation module44compares the length L3to the mean length Lm to determine whether the length L3is equal to the mean length Lm. If the length L3is less than the mean length Lm, the estimation module44re-calculates the distal endpoints48C and48D as discussed in greater detail below.

The estimation module44can re-calculate the endpoints48C and48D by calculating vectors values using x- and y-coordinates for the endpoints48C and48D of parking spot32A according to equations 2 and 3:
vecx=xP1−xP2Eqn. 2
vecy=yP1−yP2Eqn. 3

where vecxis a vector representing a difference between coordinate xP1and coordinate xP2, xP1represents an x-coordinate value for endpoint48A of parking spot32C, xP2represents an x-coordinate value for endpoint48B of parking spot32A, vecyis a vector representing a difference between coordinate yP1and coordinate yP2, yP1represents an y-coordinate value for endpoint48A of parking spot32A, and yP2represents an y-coordinate value for endpoint48B of parking spot32A.

where dirVec[1] is a first directional vector, dirVec[2] represents a second directional vector, and θ a value representing an angle measurement. In one or more implementations, the value of θ can be determined via a rotation matrix of one or more sensors24A through24D. The rotation matrix can represent an external parameter determined using a given known calibrator. The value of θ can change according to an orientation of the parking slot32A. For example, the value θ may be negative for a counter-clockwise orientation and may be positive for a clockwise orientation.

where xP3represents an x-coordinate value for endpoint48C, yP3represents an y-coordinate value for endpoint48C, xP4represents an x-coordinate value for endpoint48D, and yP4represents an y-coordinate value for endpoint48D.

In some implementations, the value of θ can be set to 90° for perpendicular and parallel parking spots, and the value of θ can be set to a standard angle value, i.e., 45°, 60°, etc., for slanted parking spots. After calculating the endpoints48C and48D, the length L3for parking spot32A is determined by calculating a difference between endpoints48A and48C or endpoints48B and48D, e.g., the difference between the x-coordinates and the difference between the y-coordinates for each endpoint.

Referring toFIG.6, in some parking environments, the autonomous park assisting system10may not be able to estimate one or more parking spots within a parking area30due to improper parking spot markings. For example, as shown inFIG.6, parking markings delineating parking spot50may not be parallel to an adjacent parking spot. In these situations, the autonomous park assisting system realigns the parking spot50such that the parking spot50is parallel to the adjacent parking spot.

The estimation module44determines whether two parking slots are adjacent to one another. Referring toFIG.6the estimation module44can determine whether parking spot50is adjacent to parking spot52calculating a y-axis difference d1and d2according to equations 10 and 11:
d1=yP1−yQ1Eqn. 11
d2=yP2−yQ2Eqn. 12

where d1represents a difference between yP1and yQ1, where d2represents a difference between yP2and yQ2, yP1represents a y-coordinate value for endpoint52A, yQ1represents a y-coordinate value for endpoint50B, yP2represents a y-coordinate value for endpoint52C, and yQ2represents a y-coordinate value for endpoint50D.

The y-axis difference can be calculated by determining a difference between d1and d2. If the difference is less than a predetermined distance threshold, the estimation module44determines the parking spots50and52are adjacent. The coordinates for endpoints50B and50D are set equal to coordinates for endpoints52A and52C.

where xP1represents an x-coordinate for endpoint52A, xP2represents an x-coordinate for endpoint52C, yP2represents a y-coordinate for endpoint52A, and yP1, represents a y-coordinate for endpoint52C.

Similar to the process described above with respect to Equations 2 through 5, the estimation module44calculates endpoints48C and48D by calculating vectors values using x- and y-coordinates for the endpoints48C and48D according to Equations 14 and 15 and calculating a first and a second direction vector using vecxand vecyaccording to equations 16 and 17:
vecx=xP1−xP2Eqn. 14
vecy=yP1−yP2Eqn. 15
dirVec[1]=(vecx*cos θ)+(vecy*(−sin θ))  Eqn. 16
dirVec[2]=(vecx*sin θ)+(vecy*cos θ)Eqn.17

where xQ4′ represents an x-coordinate value for endpoint50C′, yQ4′ represents a y-coordinate value for endpoint50C′, where xQ3′ represents an x-coordinate value for endpoint50A′, and yQ3′ represents a y-coordinate value for endpoint50A′. The updated coordinates represent realigned coordinates such that the parking spot50is oriented in the same direction as parking spot52, e.g., the endpoints50A′ and50C′ are parallel with endpoints52A and52C.

As with Equations 4 and 5, the estimation module44uses the rotation matrix to calculate Equations 16 and 17. In an example implementation, the value for θ can be set to 90° based on the configuration of the parking spots50and52.

The CNN50may be trained via a supervised training process or a self-supervised training process. In an example implementation,FIGS.7A and7Billustrate an example process for training the CNN50in accordance with one or more implementations of the present disclosure. As shown inFIG.10A; during an initial training phase, the CNN50receives a set of labeled training images (e.g., training images70and training labels72). The training images70may depict objects within a stitched image. The training labels72may comprise object labels, object types, and/or labeled endpoints relative to one or more parking spot markings. After the initial training phase, at a supervised training phase, a set of training images74are input to the CNN50. The CNN50generates outputs indicative of the endpoints for each of the training images74.

FIG.7Billustrates an example of generating output for one training image74, such as a non-labeled training image, of the training images74. Based on the initial training, the CNN50outputs a vector representation76of the parking spot endpoints. The vector representation76is compared to the around-truth data78. The CNN50updates network parameters based on the comparison to the ground-truth78. For example, the network parameters, e.g., weights associated with neurons of the CNN50, may be updated via backpropagation.

After training, the CNN50may be used to generate output80representing classified parking spot endpoints based on the received stitched images82as shown inFIG.7C.

FIG.8is a flowchart of an exemplary process800for estimating values for a parking spot within a parking area, such as a parking area30. Blocks of the process800can be executed by the controller20. The process800begins at block805in which one or more images depicting a portion of a parking area is received from one or more sensors, such as the sensors24A through24D. At block810, the image stitching module40stitches the one or more images together. At block815, the detection module42classifies endpoints of a parking spot using the stitched image. For example, the detection module42can use the CNN50to determine one or more endpoints of the parking spot.

At block820, the estimation module44determines a length for the one or more parking spots based on the classified endpoints. At block825, the estimation module44calculates a mean average based on the lengths. At block830, the estimation module determines whether a length for a parking spot is less than the mean length. If the length is less than the mean length, the estimation module44provides the coordinates for a parking spot to the vehicle control system22at block840.

If the length is not less than the mean length, the estimation module44calculates the endpoints for the parking spot at block845. The estimation module44can provide the coordinates for the parking spot to the vehicle control system22at block840. At block850, the vehicle control system22can actuate the vehicle12using the coordinates. For example, the vehicle control system22can be actuated to cause the vehicle12to park into a parking spot based on the coordinates of the calculated endpoints. The process800then ends.

Memory may include a computer readable medium (also referred to as a processor readable medium) that includes any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random-access memory (DRAM), which typically constitutes a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to a processor of an ECU. Common forms of computer readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

In some examples, system elements may be implemented as computer readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.), stored on computer readable media associated therewith (e.g., disks, memories, etc.). A computer program product may comprise such instructions stored on computer readable media for carrying out the functions described herein.

With regard to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes may be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps may be performed simultaneously, that other steps may be added, or that certain steps described herein may be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claims.