Patent Description:
Document "<NPL>, discloses a highly fused convolutional network (HFCN) based segmentation method for parking slot and lane markings based on a panoramic surround view dataset.

Document <CIT> discloses a parking stall visual identifying system including a parking stall image capture module being acquired to a parking bit image in parking lot, the parking stall image recognition processing module of identifying processing is successively carried out by frame to the parking stall video image received, the parking stall tracing module of identification is tracked to the target parking position that parking stall image recognition processing module identifies and parking stall display module that coordinate information real-time display under world coordinate system of the target parking position and target parking position that transmit parking stall tracing module comes out.

Document <CIT> discloses a kind of parking stall condition detection method and systems, image creation shape Matching Model under the parking lot idle state that wherein method passes through acquisition, simultaneously, according to the image information on parking stall contour line extraction parking stall, discrimination threshold of several differentiation features as actual conditions is therefrom obtained; The image in practical parking scene is registrated using the shape Matching Model being pre-created after obtaining the image in practical parking scene; The image on one parking stall of regional choice in image after registration, the corresponding discrimination threshold of differentiation feature for obtaining the image are compared; The particular state of current parking stall is determined according to comparison result. As corresponding discrimination threshold, i.e. the corresponding discrimination threshold in a parking stall efficiently avoids in actually detected the differentiation feature that the present invention is gone out by the correspondence parking stall measure of the parking field picture under idle state to greatly improve the accuracy of detection.

Document <CIT> discloses an automatic parking positioning method, which comprises the steps of obtaining acceleration information and angular speed information of a vehicle, obtaining wheel speed information of the vehicle, and obtaining first position information and first course information of the vehicle under a first coordinate system; acquiring point cloud data of the vehicle in a second coordinate system to obtain second position information and second course information; the first position information, the first course information, the second position information and the second course information are subjected to fusion processing through a first algorithm to obtain first position and attitude information of the vehicle in a first coordinate system; acquiring image data of a parking space to be parked, and constructing an equation; determining coordinates of each corner point of the parking space in a third coordinate system, and determining second position and attitude information of the vehicle in the third coordinate system; converting the first posture information of the first coordinate system into third posture information of a third coordinate system; and performing fusion processing on the second attitude information and the third attitude information to determine target position information and target course information of the vehicle.

An automatic parking system needs to perceive the position of a parking space in a world coordinate system. Since the parking spaces are generally drawn with a parking space marking line on a road surface in a parking area, a very popular method among various solutions is to use a computer vision algorithm to detect parking spaces on a panoramic aerial surround-view image of a vehicle. In the field of computer vision, a deep learning algorithm is more and more popular because it outperforms conventional vision methods in terms of robustness. In an automatic parking system, a method for detecting a parking space includes: using a segmentation neural network in a vehicle surround view to calculate pixels belonging to parking lines, calculating straight-line equations and endpoints and points of intersection of straight lines for each parking line, matching the detected straight lines, the straight-line endpoints, and the points of intersection of the straight lines with each other through a prior rule to obtain an area of parking place in the vehicle surround view, and then mapping into world coordinates. In this method, on the one hand, the neural network is required to obtain a high-precision segmentation result of a parking space segmentation line, i.e., to obtain a high-precision parking-line segmentation image; on the other hand, it is necessary to calculate accurate straight-line equations and straight-line endpoints for parking lines.

Currently, although there are many neural network frameworks that may be used to obtain relatively accurate segmentation results, it is still very challenging to calculate accurate straight-line equations and straight-line endpoints for parking lines. There are currently some algorithms that may be used to obtain linear equations and endpoints, but it is difficult for most of the algorithms to set parameters in a changing environment. Furthermore, the mutual influence of multiple lines may also affect the accuracy of the calculation.

Therefore, there is a need for an improved technical solution to overcome the above shortcomings of the existing solutions.

According to an aspect of the present disclosure, a method for an automatic parking system is provided. The method may include: obtaining a parking-line segmentation image through a segmentation neural network; preprocessing the parking-line segmentation image to obtain at least one parking-line skeleton image; and calculating straight-line equations and straight-line endpoints for parking lines in the parking-line segmentation image based on the at least one parking-line skeleton image. The calculating straight-line equations and straight-line endpoints for parking lines in the parking-line segmentation image based on the at least one parking-line skeleton image includes: performing, for each parking-line skeleton image, the following operations: A: dividing the parking-line skeleton image into multiple patches; B: generating multiple straight-line equations based on the multiple patches, wherein each straight-line equation is formed by fitting the pixels with first color in each patch; C: selecting an optimal straight-line equation from the multiple straight-line equations, wherein the optimal straight-line equation represents the straight line matching the largest quantity of pixels with first color in the parking-line skeleton image; D: calculating the corresponding endpoints of the optimal straight-line equation in the parking-line skeleton image; and E: deleting the pixels with first color corresponding to the optimal straight-line equation from the parking-line skeleton image; repeating steps B-E in a loop until the number of loops reaches a straight-line quantity threshold or the quantity of remaining pixels with first color in the parking-line skeleton image is less than a pixel quantity threshold.

According to one or more embodiments, the preprocessing the parking-line segmentation image to obtain at least one parking-line skeleton image may include: extracting pixels belonging to parking lines from the parking-line segmentation image and displaying the pixels belonging to parking lines with first color, and displaying the remaining pixels not belonging to parking lines with second color, thereby generating an extracted parking-line segmentation image; and performing skeleton extraction on the extracted parking-line segmentation image to extract skeletons of the parking lines, thereby generating the at least one parking-line skeleton image.

According to one or more embodiments, the preprocessing the parking-line segmentation image to obtain at least one parking-line skeleton image may include: extracting pixels belonging to parking lines from the parking-line segmentation image and displaying the pixels belonging to parking lines with first color, and displaying the remaining pixels not belonging to parking lines with second color, thereby generating an extracted parking-line segmentation image; dilating the extracted parking-line segmentation image to generate a dilated parking-line segmentation image; and performing skeleton extraction on the dilated parking-line segmentation image to extract skeletons of the parking lines, thereby generating the at least one parking-line skeleton image.

According to one or more embodiments, the preprocessing the parking-line segmentation image to obtain at least one parking-line skeleton image may include: extracting pixels belonging to parking lines from the parking-line segmentation image and displaying the pixels belonging to parking lines with first color, and displaying the remaining pixels not belonging to parking lines with second color, thereby generating an extracted parking-line segmentation image; dilating the extracted parking-line segmentation image to generate a dilated parking-line segmentation image; determining at least one connection region based on the dilated parking-line segmentation image, wherein each connection region corresponds to a region in the dilated parking-line segmentation image, each connection region only includes pixels of parking lines connected to each other, and the pixels of the parking lines belonging to different connection regions are separated from each other; dividing the dilated parking-line segmentation image according to the connection regions to generate sub-images respectively corresponding to different connection regions; and performing skeleton extraction on each sub-image to generate respective parking-line skeleton images corresponding to the sub-images.

According to one or more embodiments, the generating multiple straight-line equations based on the multiple patches may include: performing, for each patch, the following operations in parallel: getting a patch; obtaining coordinates of the pixels with first color in the patch; and fitting a straight-line equation based on the obtained pixels with first color.

According to one or more embodiments, the method may further include: performing, for each patch, the following operations in parallel: determining whether the ratio of the quantity of pixels with first color matched by the straight-line equation to the quantity of all pixels with first color in the patch is greater than a ratio threshold, wherein the pixel with first color in the patch whose distance from the straight line represented by the straight-line equation is less than a distance threshold is considered to be the pixel with first color matched by the straight-line equation; transferring the straight-line equation to a coordinate system of the parking-line skeleton image to represent a straight line in the parking-line skeleton image, and in response to the ratio of the quantity of pixels with first color matched by the straight-line equation to the quantity of all pixels with first color in the patch being greater than the ratio threshold, taking the number of pixels with first color whose distances from the straight line are less than the distance threshold in the parking-line skeleton image as a value of a comparison parameter; and setting the value of the comparison parameter to <NUM> in response to the ratio of the quantity of pixels with first color matched by the straight-line equation to the quantity of all pixels with first color in the patch being equal to or less than the ratio threshold.

According to one or more embodiments, the method may further include: comparing multiple comparison parameter values calculated for multiple patches; and selecting the straight-line equation corresponding to the largest comparison parameter value among the multiple comparison parameter values as the optimal straight-line equation.

According to one or more embodiments, the method may further include: filtering the generated parking-line skeleton image to generate a filtered parking-line skeleton image.

According to another aspect of the present disclosure, an apparatus for an automatic parking system is provided, including: a memory and a processor. The memory is configured to store computer-executable instructions. The processor is configured to execute the instructions to implement the method described above.

The present disclosure may be better understood by reading the following description of nonlimiting implementations with reference to the accompanying drawings, in which:.

It should be understood that the following description of the embodiments is given for purposes of illustration only and not limitation. The division of examples in functional blocks, modules or units shown in the figures should not be construed as implying that these functional blocks, modules or units must be implemented as physically separate units. The functional blocks, modules or units shown or described may be implemented as separate units, circuits, chips, functional blocks, modules, or circuit elements. One or more functional blocks or units may also be implemented in a common circuit, chip, circuit element, or unit.

The use of singular terms (for example, but not limited to, "a") is not intended to limit the number of items. Relational terms, for example but not limited to, "top", "bottom", "left", "right", "upper", "lower", "down", "up", "side", "first", "second" ("third", etc.), "entry", "exit", etc. are used in the written description for clarity in specific reference to the drawings and are not intended to limit the scope of the present disclosure or the appended claims, unless otherwise noted. The terms "couple," "coupling," "being coupled," "coupled," "coupler", and similar terms are used broadly herein and may include any method or device for fixing, bonding, adhering, fastening, attaching, associating, inserting, forming thereon or therein, communicating with, or otherwise directly or indirectly mechanically, magnetically, electrically, chemically, and operatively associated with an intermediate element and one or more members, or may also include, but is not limited to, one member being integrally formed with another member in a unified manner. Coupling may occur in any direction, including rotationally. The terms "including" and "such as" are illustrative rather than restrictive, and the word "may" means "may, but not necessarily", unless stated otherwise. Although any other language is used in the present disclosure, the embodiments shown in the figures are examples given for purposes of illustration and explanation and are not the only embodiments of the subject matter herein.

The present disclosure provides a robust and fast method and system for an automatic parking system. The method and system calculate straight-line equations and straight-line endpoints for parking lines based on a parking-line segmentation image of a panoramic aerial surround-view image of a vehicle. The method provided by the present disclosure can obtain reliable and robust straight-line equations, so that an accurate straight-line equation can be detected even when the splicing distortion of parking lines is serious and the detection result of a segmentation network is irregular, thereby providing reliable information for subsequent detection of a parking space. Meanwhile, the method provided by the present disclosure may adopt a parallel calculation mechanism, which provides a favorable guarantee for the calculation speed of an algorithm.

A calculation method and apparatus of various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. <FIG> shows a flow chart of a method according to one or more embodiments of an implementation of the present disclosure. As shown in <FIG>, at S102, a parking-line segmentation image is obtained. Specifically, a parking-line segmentation image may be obtained through a segmentation neural network. At S104, the parking-line segmentation image is preprocessed to obtain at least one parking-line skeleton image. At S106, straight-line equations and straight-line endpoints are calculated for parking lines in the parking-line segmentation image based on the at least one parking-line skeleton image. The method shown in <FIG> can obtain reliable and effective intermediate data and related auxiliary information by preprocessing a parking-line segmentation image output from a segmentation neural network, thereby calculating straight-line equations more reliably and avoiding the influence of some noise.

<FIG> shows a flow chart of a method for preprocessing a parking-line segmentation image according to one or more embodiments of an implementation of the present disclosure. Those skilled in the art may understand with reference to the following description of <FIG> that the steps in a dotted box in <FIG> may be used as preferred steps.

In some embodiments, the preprocessing method may include: extracting, at S202, parking lines from the parking-line segmentation image, thereby generating an extracted parking-line segmentation image. In some examples, pixels belonging to parking lines are extracted from the parking-line segmentation image and the pixels belonging to parking lines are displayed with first color (for example, white), and the remaining pixels not belonging to parking lines are displayed with second color (for example, black), thereby generating an extracted parking-line segmentation image. After performing the processing at S202, the method proceeds directly to S210. At S210, extraction is performed on the extracted parking-line segmentation image to extract skeletons of the parking lines, thereby generating the parking-line skeleton image.

In some other embodiments, the preprocessing method may include: extracting parking lines from the parking-line segmentation image, thereby generating an extracted parking-line segmentation image (S202). However, after performing the processing at S202, the method proceeds to S204. At S204, the extracted parking-line segmentation image is first dilated to generate a dilated parking-line segmentation image. Then, the method proceeds to S210. At S210, extraction is performed on the dilated parking-line segmentation image to extract skeletons of the parking lines, thereby generating the parking-line skeleton image.

Furthermore, in some other embodiments, the preprocessing method may include: extracting, at S202, parking lines from the parking-line segmentation image, thereby generating an extracted parking-line segmentation image. After performing the processing at S202, the method proceeds to S204. At S204, the extracted parking-line segmentation image is first dilated to generate a dilated parking-line segmentation image. Then, the method proceeds to S206. At S206, at least one connection region is determined based on the dilated parking-line segmentation image. Each connection region corresponds to a region in the dilated parking-line segmentation image, each connection region only includes pixels of parking lines connected to each other, and the pixels of the parking lines belonging to different connection regions are separated from each other. Then, the method continues to proceed to S208. At S208, the dilated parking-line segmentation image is divided according to the connection regions to generate sub-images respectively corresponding to different connection regions. Finally, the method proceeds to S210. At S210, extraction is performed on each sub-image to generate respective parking-line skeleton images corresponding to the sub-images.

In addition, in some examples, the preprocessing method according to the above various embodiments may further include step S212. At S212, the generated parking-line skeleton image of parking lines is filtered to generate a filtered parking-line skeleton image.

In order to be more intuitive, each step in the preprocessing method according to the above various embodiments is described in further detail below with reference to <FIG>. <FIG> shows a parking-line segmentation image obtained by, for example, a segmentation neural network, taking a panoramic aerial surround-view image of a vehicle in a garage as an example. The left image in <FIG> is the panoramic aerial surround-view image of the vehicle, and the right image is the parking-line segmentation image. The parking-line segmentation image is obtained by classifying pixels in the panoramic aerial surround-view image shown in the left image by the segmentation neural network.

In some examples, the extraction processing performed at S202 is to extract the pixels detected by the neural network that belong to the parking lines (which may also be referred to as parking space segmentation lines), and filter out the pixels that do not belong to the parking lines. In the extracted parking-line segmentation image, these two kinds of pixels are represented by two colors. For example, the pixels belonging to parking lines are displayed with white color, and the pixels not belonging to parking lines are displayed with black color. The extracted parking-line segmentation image is shown in <FIG>.

In some embodiments, preferably, the extracted parking-line segmentation image may be dilated at S204, and the generated dilated parking-line segmentation image is shown in <FIG>. It will be appreciated from the above that in some embodiments, skeleton extraction may be directly performed on the dilated parking-line segmentation image, thereby generating the parking-line skeleton image. Further, the generated parking-line skeleton image may also be filtered. In order to save space, no further description is given here in conjunction with the schematic diagram of image processing.

<FIG> shows a schematic diagram of a result of determining connection regions performed at S206. For visual representation, <FIG> distinctively shows different connection regions found in the dilated parking-line segmentation image with different gray scales. It can be seen that the connection region is a region including parking lines connected together, and parking lines belonging to different connection regions are not connected to each other. In some examples, different connection regions may also be classified by pixels (for example, pixel coordinates). The purpose of determining at least one connection region is to calculate the pixels of the parking lines connected together as a single image, so as to avoid the situation where the pixels in different connection regions affect each other and the calculation result is inaccurate. It will be appreciated that <FIG> are only for helping understanding, and the distribution of parking lines and the number of connection regions determined therefrom are not limited by the above examples. In an actual scenario, if the obtained panoramic aerial surround-view images of the vehicle are different, the generated parking-line segmentation images are different, and the number of qualified connection regions may also be different.

In some embodiments, preferably, at S208, the parking lines belonging to different connection regions may be further divided to form respective sub-images corresponding to different connection regions. For example, the dilated parking-line segmentation image shown in <FIG> will be divided into different sub-images according to different connection regions, such as sub_img1, sub_img2, sub_img3, and sub_img4, as shown in <FIG>.

It will be appreciated that the extraction processing performed at S210 may be based on different types of parking-line segmentation images generated in the above embodiments, such as parking-line segmentation images generated only through the extraction step, parking-line segmentation images generated through the dilation step, and parking-line segmentation images corresponding to sub-images obtained after further division according to connection regions. In order to save space, only the sub-images shown in <FIG> are used as an example for the following description.

At S210, skeleton extraction is performed on the parking-line segmentation image to extract skeletons of the parking lines, thereby generating the parking-line skeleton image. <FIG> exemplarily shows a result of the skeleton extraction performed on each sub-image at S210. The middle column of <FIG> shows the corresponding sub-images sub_img1, sub _img2, sub_img3, and sub_img4 shown in <FIG>, and the rightmost column is a result of skeleton extraction of corresponding sub-images.

Due to the inaccurate output results of seams or segmentation networks, there will be many isolated pixel blocks at the periphery of straight lines after the skeleton extraction. For illustration, <FIG> shows a result of partial enlargement of a skeleton image corresponding to sub_img4. It can be seen that there are many isolated pixel blocks at the periphery of the straight lines after the skeleton extraction. Therefore, preferably, at S220, each parking-line skeleton image may also be filtered, so as to delete the small isolated pixel blocks caused by the skeleton extraction. For example, these small isolated pixel blocks may be deleted by calculating the quantity of white pixels in the connection regions. The filtered parking-line skeleton image corresponding to sub _image4 is shown in <FIG>. In the right partial enlarged image, it can be seen that the isolated pixel blocks have been deleted.

After the preprocessing process in one or more of the embodiments described above, the parking-line skeleton image generated after the preprocessing may be finally obtained. As described above, those skilled in the art will appreciate that the parking-line skeleton image obtained after the preprocessing of the parking-line segmentation image in the above various embodiments may be applied to the method for calculating straight-line equations and straight-line endpoints that will be described hereinafter. In order to save space, the following calculation method will only take the parking-line skeleton image obtained after preprocessing the parking-line segmentation image shown in <FIG> as an example. <FIG> shows a parking-line skeleton image obtained after the preprocessing processes S202-S212 of the parking-line segmentation image, for example, including the filtered parking-line skeleton images sub_img1_f1, sub_img1_f2, sub_img1_f3, and sub_img1_f4.

So far, the preprocessing process of a panoramic surround aerial-view segmentation image of a vehicle ends. The preprocessing process described in each of the above embodiments has the following advantages.

Firstly, extracting the parking-line skeleton image based on the parking-line segmentation image will be more conducive to calculating straight-line equations therefor. Furthermore, filtering the parking-line skeleton image is further conducive to avoiding the influence of noise and improving the robustness.

Secondly, since the panoramic aerial surround-view image of the vehicle is spliced from multiple pictures, the splicing often leads to picture distortion, such as striated cracks shown in <FIG>. Due to the insufficient accuracy of the segmentation network, the parking lines cannot be detected completely, which may result in subsequent inaccurate calculation of the straight-line equations. The dilation processing helps to eliminate the striated cracks formed by splicing, and also to repair the incomplete detection of the parking-lines by the segmentation network, so as to prepare for the next step of skeleton extraction. <FIG> shows an effect comparison between striated cracks at the splicing of the parking-line segmentation image under two situations of dilation and no dilation. In <FIG>, the original parking-line segmentation image is illustrated in the left, and the dilated parking-line segmentation image is illustrated in the right. It can be seen that the dilated parking-line segmentation image has a better effect. Further, <FIG> shows a result comparison of performing skeleton extraction on a parking-line segmentation image under two situations of dilation and no dilation. It can be seen that the straight line obtained after the dilation processing (the left part) is cleaner, there is less noise at the periphery of the straight line, and the existence of these noise will affect the later calculation accuracy.

Thirdly, after the skeleton extraction, some isolated pixels or pixel blocks may be formed at the periphery of the skeletonized straight line (see <FIG>). These isolated pixels or pixel blocks may be deleted by filtering to reduce the influence of noise.

Fourthly, the division according to connection regions make the straight line be calculated independently in different connection regions, so as to avoid the influence of the pixels in other connection regions on the calculation accuracy.

Fifthly, the two steps of skeleton extraction for sub-images of different connection regions and filtering of the parking-line skeleton image after the skeleton extraction are both independent calculations for multiple sub-images. Therefore, parallel calculation (such as opencl and multithreading) may be used to accelerate the algorithm.

The following will explain how to calculate the straight-line equations and endpoints for the parking lines based on the preprocessed parking-line skeleton image. At present, there are many methods for detecting straight lines from images, such as Hough straight line detection, LSD, and other methods. Although some methods also have strong robustness, these methods still have the following problems. <NUM>) Some algorithms need to set parameters, but since the detection results in different environments are too sensitive to parameters, it is difficult to set the parameters uniformly. <NUM>) In an image, a positional relationship between straight lines will affect the solution accuracy. For example, <FIG> shows a situation where the presence of intersecting straight lines may affect the accuracy. As shown in <FIG>, a picture having a straight line to be detected is in the left. It can be clearly seen that the left picture contains two straight lines, but some current algorithms may obtain through calculation an additional straight line at a point of intersection, as shown in the right picture.

The method for calculating straight-line equations and straight-line endpoints provided by the present disclosure will avoid the above problems, thereby making the calculation more accurate. The calculation method of the present disclosure includes: dividing each obtained parking-line skeleton image sub_img_f (for example, sub_img1_f1, sub_img1_f2, sub_img1_f3, and sub_img1_f4 shown in <FIG>) into several patches, fitting a straight-line equation representing a straight line for pixels (for example, white pixels) belonging to the straight line in each patch, and after fitting one straight-line equation for each of the patches, selecting a straight-line equation that matches best in a competitive manner (for example, selecting a straight-line equation that matches the most pixels in sub_image_f). Then, the pixels corresponding to the optimal straight-line equation are deleted from sub_image_f. After that, a second straight line is found from sub_image_f in the same way, and so on, until all the straight lines in the parking-line skeleton image sub_image_f are found, that is, the straight-line equations corresponding to all the straight lines in the parking-line skeleton image sub_image_f are calculated.

<FIG> shows a method for calculating straight-line equations and straight-line endpoints for parking lines based on a preprocessed parking-line segmentation image according to one or more embodiments of the present disclosure.

As shown in <FIG>, at S1502, the parking-line skeleton image is divided into multiple patches. When selecting the size of the patches, the following conditions may be satisfied: the size of the obtained patches through division should be such that the obtained patches through division can capture a segment on each straight line in the parking-line skeleton image (for example, sub _image_f), and no other straight lines are included in these patches. Since parking lines are arranged generally in order and have a fixed size, the size selection of the patches has a large range to satisfy this condition (the influence of the size selection will be described later in conjunction with <FIG>).

At S1504, multiple straight-line equations may be generated based on the multiple patches. Each straight-line equation is formed by fitting the pixels (for example, white pixels) with first color in each patch.

At S1506, an optimal straight-line equation may be selected from the generated multiple straight-line equations in a competitive manner. For example, the straight-line equation matching the largest quantity of white pixels in the parking-line skeleton image is considered to be the optimal straight-line equation among the multiple straight-line equations. In other words, the quantity of white pixels belonging to the straight line represented by the optimal straight-line equation is the largest. At S1508, the corresponding endpoints of the optimal straight-line equation in the parking-line skeleton image may be calculated. Then, at S1510, the white pixels corresponding to the optimal straight-line equation may be deleted from the parking-line skeleton image.

At S1512, "determination <NUM>" is executed, that is, it is determined whether the number of executions from step S1504 to step S1510 reaches a straight-line quantity threshold. The straight-line quantity threshold represents a maximum quantity of straight lines included in a connection region image (for example, sub-image sub_img). The straight-line quantity threshold may be preset according to system requirements and actual experience. Taking <NUM> sub-images obtained in <FIG> each including a connection region as an example (for example, sub_img1, sub_img2, sub_img3, and sub_img4), it can be seen that sub_img3 includes the most straight lines. If the occluded part is included, sub_img3 includes a total of six straight lines, as shown in <FIG>. For example, in this example, the straight-line quantity threshold may be determined from multiple pictures in this way to be <NUM>. It will be appreciated that, the selection of the straight-line quantity threshold is only described here by taking each sub-image shown in <FIG> as an example, rather than making a specific limit on the size of the threshold value thereof. In fact, every time a straight line is found, pixels belonging to the straight line are deleted in the method of the present disclosure. Therefore, the straight-line quantity threshold may be set arbitrarily, and there is no need to worry that the straight-line quantity threshold will be set too large.

Returning to <FIG>, if it is determined at S1512 that the number of executions from step S1504 to step S1510 has reached the straight-line quantity threshold, that is, the result of "determination <NUM>" is "Yes", the method ends. If it is determined at S1512 that the number of executions from step S1504 to step S1510 has not reached the straight-line quantity threshold, that is, the result of "determination <NUM>" is "No", the method proceeds to S1514.

At S1514, "determination <NUM>" is executed, that is, it is determined whether the quantity of remaining white pixels in the parking-line skeleton image is less than a pixel quantity threshold after deleting the pixels with first color (for example, white) corresponding to the optimal straight-line equation from the parking-line skeleton image (S1510). In some embodiments, the pixel quantity threshold may be preset according to the calculation accuracy requirement. For example, a preferred range for the pixel quantity threshold may be set to <NUM>-<NUM>.

If it is determined at S1514 that the quantity of remaining white pixels in the parking-line skeleton image has been less than the pixel quantity threshold after deleting the pixels corresponding to the optimal straight-line equation from the parking-line skeleton image, that is, the result of "determination <NUM>" is "Yes", the method ends. At this moment, it is considered that all straight lines in the parking-line skeleton image have been found. If it is determined at S1514 that the quantity of remaining white pixels in the parking-line skeleton image is greater than or equal to the pixel quantity threshold, that is, the result of " determination <NUM>" is "No", the method proceeds to S1504, and the steps of S1504-S1510 continue to be performed until the number of loops of steps S1504-S1510 reaches the straight-line quantity threshold or the quantity of remaining white pixels in the parking-line skeleton image is less than the pixel quantity threshold.

In the above method for calculating straight-line equations and straight-line endpoints based on a preprocessed parking-line segmentation image according to one or more embodiments, since a method for further dividing the preprocessed parking-line segmentation image (for example, sub_image_f) into several equal patches to obtain straight-line equations is adopted, the complicated process of adjusting parameters in the existing calculation methods can be avoided. Furthermore, in the method of one or more embodiments of the present disclosure, if white pixels included in one selected patch are not sufficient to reflect the feature of a straight line, the calculated straight-line equation cannot obtain a maximum matching value and will be eliminated. Therefore, selecting a straight-line equation matching the largest quantity of pixels in a competitive manner can ensure that the obtained straight-line equation is the straight-line equation with the best degree of fitting. Furthermore, in the method of one or more embodiments of the present disclosure, by deleting the pixels belonging to the found straight line, the influence of these pixels on subsequent calculation can be avoided, thereby improving the robustness of the calculation results.

<FIG> shows, in more detail, a method for calculating straight-line equations and straight-line endpoints for parking lines based on a preprocessed parking-line segmentation image according to one or more embodiments of the present disclosure. <FIG> schematically illustrates the overall method in the form of an exemplary algorithmic flow chart.

The upper left corner of <FIG> schematically shows a parking-line skeleton image represented by sub _img_f, where the image sub_img_f is divided into n equal patches. Each patch is referred to, for example, by p[idx], where idx is greater than or equal to <NUM> and less than or equal to n. The size of the patches depends on the value of n. As mentioned above, when selecting the size of the patches, the following conditions should be satisfied: the size of the obtained patches through division should be such that the obtained patches through division can capture a segment on each straight line in the parking-line skeleton image (for example, sub_image_f), and no other straight lines are included in these patches. This point is further explained below with reference to <FIG>. In order to make the example more clearly displayed, <FIG> shows six straight lines included therein, straight lines <NUM>-<NUM>, by taking the dilated parking-line segmentation image sub_img3 (see <FIG>) corresponding to the parking-line skeleton image sub_image_f3 as an example. The straight line numbered <NUM> may be understood as a hidden straight line caused by occlusion.

Taking sub_img_f3 as an example, <FIG> respectively shows the problems caused by the situation where the size of a window for patch selection is appropriate, and the extreme situations of being too large and too small. <FIG> is the image sub_image_f3. The horizontal and vertical straight lines constituting grids in 18b are dividing lines used to cut into patches, and each grid represents a patch. It can be seen that equations corresponding to straight lines <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> in <FIG> may be obtained from white pixels included in patches numbered ①, ②, ③, ④, and ⑤, respectively. Since these patches include corresponding straight-line segments and do not include other straight-line segments, the division size of the patches shown in <FIG> is appropriate. The division size of the patches in <FIG> is too large, and only the patches numbered ③ and ⑤ meet the requirements and may be used to calculate the equation corresponding to straight line <NUM> in <FIG>. The remaining patches either do not include straight lines, or include multiple straight lines, as a result, wrong straight-line equations will be obtained through fitting. <FIG> shows a situation where a patch of a small size (for example, 4x3 pixel area) is enlarged. Since the size of the patch is too small, the pixels in the patch include too few white pixels. Too few white pixels cannot represent the features of a straight line, and therefore, a correct straight-line equation cannot be obtained through fitting based on too few white pixels. In an actual environment, since parking lines are arranged generally in order and have a fixed size, the size selection of the patches has a large range to satisfy this condition. The present disclosure does not limit the size of the patches. Those skilled in the art can make debugging selections according to the accuracy of the system and the accuracy requirements for the calculation results.

Returning to <FIG>, an algorithm flow of the entire method will be described below for the divided multiple patches. After the algorithm starts, an initial value of the number of loops is set to <NUM>, and two determinations: "determination <NUM>" and "determination <NUM>" are then performed sequentially. The determination criteria for these two determination processes are the same as those in the method described above with reference to <FIG>. That is, the determination process in "determination <NUM>" includes determining whether the number of loops of the entire algorithm reaches a preset straight-line quantity threshold. In one or more embodiments shown in <FIG>, the number of loops includes, for example, the number of executions from steps S1 to S8. The determination process in "determination <NUM>" includes: determining whether the quantity of remaining white pixels in the parking-line skeleton image (after the execution of step S8) is less than a preset pixel quantity threshold. In other words, there are two conditions for the end of the algorithm flow. One condition is that the number of loops reaches the preset threshold, and the other condition is that the quantity of remaining white pixels in the parking-line skeleton image is less than the threshold. It any one of the two conditions is satisfied, it means that all the straight lines in the parking-line skeleton image have been found, and the algorithm ends.

In the large box in the middle of <FIG>, a part of parallel calculation for each patch is shown. That is, the calculation for each patch may be independent. For example, parallel calculation methods (such as OpenCL) may be used to accelerate the operation.

The following takes a certain patch as an example to introduce the method for calculating straight-line equations and endpoints for each patch. For example, at S1, a patch p[idx] is obtained. At S2, X and Y coordinates of all pixels with first color (for example, white) in the patch p[idx] are obtained. At S3, based on all the obtained white pixels, a straight-line equation representing a straight line is fitted. For example, methods such as least squares may be used to fit the straight-line equation.

Then, "determination <NUM>" is executed to determine whether the fitted straight-line equation matches at least the white pixels of a certain ratio threshold in the patch p[idx]. In other words, it is determined whether the ratio of the quantity of pixels belonging to the straight-line equation (that is, the quantity of pixels matched by the straight-line equation) to the quantity of all the current white pixels in the entire patch p[idx] is greater than a ratio threshold which may be, for example, a percentage value. Preferably, the ratio threshold may range from <NUM>% to <NUM>%. The white pixel in the patch p[idx] whose distance from the straight line represented by the straight-line equation is less than a distance threshold is considered to be the pixel belonging to the straight line represented by the straight-line equation, that is, the pixel matched by the straight-line equation. In some examples, the distance threshold may be preset to, for example, a few pixel units. The purpose of setting "determination <NUM>" is that it is correct for the straight-line equation to match the vast majority of white pixels in the patch; otherwise, it is considered that the white pixels included in the patch do not have the features of a straight line (multiple straight lines may be included, or an area where a straight line is not appropriate is selected, such as a slightly deformed area). In other words, the calculated straight-line equation has the ability to match a whole straight line in the parking-line skeleton image (sub_image_f) only if it basically matches all the pixels in the patch.

If the result of determination at "determination <NUM>" is "Yes" (that is, the ratio of the quantity of pixels belonging to the straight line represented by the straight-line equation to the quantity of all the current white pixels in the entire patch p[idx] is greater than the ratio threshold), the algorithm proceeds to S4. <NUM>, the straight-line equation is transferred to a coordinate system of the parking-line skeleton image from a coordinate system of patches (that is, transferred from a patch coordinate system to a sub_img_f coordinate system) to represent a straight line in the parking-line skeleton image.

The algorithm then proceeds to S5. At S5, the quantity of the white pixels in the parking-line skeleton image whose distances from the straight line represented by the straight-line equation obtained in step S4 in the parking-line skeleton image are less than a distance threshold is stored as a value of a comparison parameter. That is, in the coordinate system of the parking-line skeleton image, the quantity of current white pixels whose distances from the straight line represented by the straight-line equation are less than the threshold distance is counted, and the quantity is used as a value of a comparison parameter (for example, cnt[idx]).

If the result of determination at "determination <NUM>" is "No" (that is, the ratio of the quantity of pixels belonging to the straight line represented by the straight-line equation to the quantity of all the current white pixels in the entire patch p[idx] is less than or equal to the ratio threshold), the algorithm proceeds to S4. <NUM>, the value of the comparison parameter is set to <NUM>.

After the above calculations from S1-S5, the comparison parameter values such as cnt[idx](idx=<NUM>, <NUM>,. n) may be calculated in parallel for the patches. Then, at S6, the comparison parameter values calculated for the patches are compared, and the straight-line equation corresponding to the largest comparison parameter value among all the comparison parameter values is selected as an optimal straight-line equation. At S7, the corresponding endpoints of the optimal straight-line equation in the parking-line skeleton image (sub_img_f) are calculated. Then, at S8, the pixels corresponding to the optimal straight-line equation are deleted from the parking-line skeleton image. So far, the algorithm is completed for one loop, and determination <NUM> is executed after adding <NUM> to the number of loops. The above process is repeated until the end condition is satisfied, and then the algorithm ends.

One or more embodiments of another implementation of the present disclosure may provide an apparatus for an automatic parking system. The apparatus may include a memory and a processor. The memory may be configured to store computer-executable instructions. The processor may be configured to execute the computer-executable instructions to implement the methods described above herein. For example, the processor may be implemented as a microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), discrete logic, or a combination of these and/or other types of circuits or logic. For example, the memory may be a DRAM, a SRAM, a flash memory, or other types of memory. Parameters (for example, conditions and thresholds) and other data structures may be stored and managed separately, may be combined into a single memory or database, or may be logically and physically organized in many different ways. Programs and instruction sets may be parts of a single program, or separate programs, or distributed across a plurality of memories and processors.

The description of the implementations has been presented for the purposes of illustration and description. The implementations may be appropriately modified and changed according to the above description or these modifications and changes may be obtained by practicing the method. For example, unless otherwise indicated, one or more of the methods described may be performed by a suitable device and/or a combination of devices. The method may be performed by using one or more logic devices (for example, processors) in combination with one or more additional hardware elements (such as storage devices, memories, hardware network interfaces/antennas, switches, actuators, clock circuits, etc.) to execute stored instructions. The method described and associated actions may also be executed in parallel and/or simultaneously in various orders other than the order described in this application. The system described is illustrative in nature, and may include additional elements and/or omit elements.

Claim 1:
A method for an automatic parking system, comprising:
obtaining a parking-line segmentation image through a segmentation neural network;
preprocessing the parking-line segmentation image to obtain at least one parking-line skeleton image; and
calculating straight-line equations and straight-line endpoints for parking lines in the parking-line segmentation image based on the at least one parking-line skeleton image, wherein
the calculating straight-line equations and straight-line endpoints for parking lines in the parking-line segmentation image based on the at least one parking-line skeleton image comprises: performing, for each parking-line skeleton image, the following operations:
A: dividing the parking-line skeleton image into multiple patches;
B: generating multiple straight-line equations based on the multiple patches, wherein each straight-line equation is formed by fitting the pixels with first color in each patch;
C: selecting an optimal straight-line equation from the multiple straight-line equations, wherein the optimal straight-line equation represents the straight line matching the largest quantity of pixels with first color in the parking-line skeleton image;
D: calculating the corresponding endpoints of the optimal straight-line equation in the parking-line skeleton image; and
E: deleting the pixels with first color corresponding to the optimal straight-line equation from the parking-line skeleton image;
repeating steps B-E in a loop until the number of loops reaches a straight-line quantity threshold or the quantity of remaining pixels with first color in the parking-line skeleton image is less than a pixel quantity threshold.