Patent Description:
Innovations in electronics and technology have made it possible to incorporate a variety of advanced features on automotive vehicles. Various sensing technologies have been developed for detecting objects or monitoring the surroundings in a vicinity or pathway of a vehicle. Such systems are useful for parking assist, lane departure detection and cruise control adjustment features, for example.

More recently, automated vehicle features have become possible to allow for autonomous or semi-autonomous vehicle control. Sensors for such systems may incorporate cameras, ultrasonic sensors, LIDAR (light detection and ranging) detectors or radar detectors for determining when an object or another vehicle is in the pathway of or otherwise near the vehicle. Depending on the particular implementation, information from such a sensor may be used for automating at least a portion of the vehicle control or providing an indication to a driver regarding the conditions around the vehicle.

While such information is useful, it is not obtained without challenges. For example, the information from a camera detector can require relatively large amounts of processing capacity and time to make useful determinations. The same is true of other types of sensors or detectors. One challenge those skilled in the art are trying to overcome is how to handle information from such sensors or detectors in an efficient manner within the capabilities of the types of processors that are economical to include on vehicles.

Object detection systems are known from documents <CIT> and <CIT>.

This object is solved by the independent claims. The invention is defined in the claims. An illustrative object detection system includes a camera having a field of view. The camera provides an output comprising information regarding potential objects within the field of view. A processor is configured to select a portion of the camera output based on information from at least one other type of detector that indicates a potential object in the selected portion. The processor determines an Objectness of the selected portion based on information in the camera output regarding the selected portion.

The processor is configured to locate a plurality of segments in the selected portion, divide at least one of the segments into patches, and determine the Objectness of each of the patches, respectively. The processor may be configured to determine a total Objectness of an entire at least one of the segments, and the processor may be configured to determine the Objectness of the at least one of the segments based on the Objectness of each of the patches and the total Objectness. The processor is configured to divide up the selected portion into segments, and the processor may be configured to arrange the segments based on an Objectness of the respective segments.

The segments may include at least one segment having a first geometry and at least one other segment having a second, different geometry. At least the first geometry may correspond to a distribution of data points from the at least one other type of detector within the at least one segment.

The processor is configured to ignore other parts of the camera output that do not include information from the at least one other type of detector indicating a potential object in the other parts of the camera output.

The object detection system comprises the at least one other type of detector, wherein the at least one other type of detector comprises one of a radar detector or a LIDAR detector.

The processor is configured to recognize a clustered set of data points from the at least one other type of detector, and the processor may be configured to select at least one clustered set of data points as the selected portion.

The at least one other type of detector may provide a LIDAR output having an intensity; and the processor may determine the Objectness from at least one of the camera output and the intensity of the LIDAR output. The camera output may comprise a plurality of images, the plurality of images may be in a time-based sequence, the processor may be configured to use the plurality of images to determine motion cues corresponding to movement of a potential object in the selected portion, and the processor may be configured to use the motion cues when determining the Objectness of the selected portion.

The processor may be configured to determine a respective Objectness of a plurality of segments of the selected portion, the processor is configured to rank the Objectness of each of the segments, and the processor may be configured to select a highest ranked Objectness to identify a location of a potential object. The processor may be configured to provide an object location estimation within an identified area of the selected portion of the camera output.

An illustrative inventive method of detecting at least one potential object includes selecting a portion of a camera output based on information from at least one other type of detector that indicates a potential object in the selected portion and determining an Objectness of the selected portion based on information in the camera output regarding the selected portion.

The method includes dividing up the selected portion into segments and determining the Objectness of each of the segments, respectively. The method may include dividing at least one of the segments into a plurality of patches, determining a total Objectness of the entire at least one of the segments, determining an Objectness of each of the patches, and determining the Objectness of the at least one of the segments based on the Objectness of each of the patches and the total Objectness.

Dividing up the selected portion into segments may comprise configuring respective geometries of the segments based on information from the at least one other type of detector, wherein the segments may include at least one segment having a first geometry and at least one other segment having a second, different geometry, and at least the first geometry may correspond to a distribution of data points from the at least one other type of detector within the at least one segment.

The method includes ignoring other parts of the camera output that do not include information from the at least one other type of detector indicating a potential object in the other parts of the camera output.

Selecting the portion of the camera output may comprise recognizing a clustered set of data points from the at least one other type of detector.

The camera output may comprise a plurality of images in a time-based sequence and the method may comprise using the plurality of images to determine movement cues of a potential object in the selected portion, and using the movement cues to determine the Objectness of the selected portion.

The method includes determining a respective Objectness of a plurality of segments of the selected portion, ranking the Objectness of each of the segments, and selecting a highest ranked Objectness to identify a location of a potential object.

The method may include providing an object location estimation within an identified area of the selected portion of the camera output.

The at least one other type of detector may provides a LIDAR output having an intensity, and determining the Objectness may comprise using at least one of the camera (<NUM>) output and the intensity of the LIDAR output.

Further features and advantages will appear more clearly on a reading of the following detailed description of at least one disclosed embodiment, which is given by way of non-limiting example only and with reference to the accompanying drawings.

Embodiments of this invention provide an ability to process information from a camera-based detector in an efficient manner. Information from at least one other type of detector is used for selecting a portion of a camera output and object detection is based on determining an Objectness of that selected portion.

<FIG> schematically illustrates a vehicle <NUM> including a camera-based detector device <NUM>. This example includes a camera <NUM> and a processor <NUM>. The camera <NUM> has a field of view <NUM> within which the camera <NUM> is capable of providing information regarding the environment near or in a pathway of the vehicle <NUM>. The processor <NUM> is configured to use information output from the camera <NUM> to determine whether one or more objects is within the environment corresponding to the field of view <NUM>. In other words, the processor <NUM> is configured to determine whether an object is within the camera's field of view <NUM>.

In an example embodiment, the processor <NUM> includes at least one computing device, such as a microprocessor. The computing device is configured or otherwise programmed to make object detection determinations consistent with those described below. The processor <NUM> in some examples includes on-board memory and in other examples the processor <NUM> is configured to communicate with a remotely located memory using known wireless communication techniques. The processor <NUM> may be a dedicated device that is a portion of the camera-based detector device <NUM> or may be a portion of another processor or controller located on the vehicle <NUM>.

At least one other type of detector <NUM> is provided on the vehicle <NUM>. In some embodiments, the detector <NUM> comprises a LIDAR detector. In other embodiments, the detector <NUM> comprises a RADAR detector. The type of information or data provided by the detector <NUM> is different than that provided by the camera <NUM>. The detector <NUM> has a field of view (not illustrated) that at least partially overlaps with the field of view <NUM> of the camera <NUM>.

Reference to the camera's field of view and the camera's output within this description should be considered synonymous or interchangeable unless the context requires a different interpretation. For example, when the processor <NUM> is described as selecting a portion of the camera output or selecting a portion of the camera's field of view, that should be understood to refer to the processor <NUM> utilizing information from the camera <NUM> corresponding to an image or other output from the camera <NUM> that indicates the contents of the environment within the camera's field of view <NUM>.

<FIG> is a flowchart diagram <NUM> that summarizes an example approach of detecting an object, which may include identifying an object's presence, type, location, movement or a combination of these. The technique summarized in <FIG> begins at <NUM> where the processor <NUM> obtains information regarding potential objects in the camera field of view <NUM> from the at least one other type of detector <NUM>. At <NUM>, the processor <NUM> selects at least one area within the camera field of view containing a possible object based on the information obtained from the other type of sensor <NUM>. In an example embodiment, the processor <NUM> is configured to recognize clustered data from the detector <NUM> that corresponds to a potential object location within the camera field of view <NUM>. At <NUM>, the processor <NUM> determines an Objectness of the selected area using information from the output of the camera <NUM>. Known Objectness determining techniques are used in some example embodiments.

In most cases camera information is used to compute Objectness. In another embodiment, LiDAR or Radar information can be used. For example, LiDAR provides intensity detection in addition to point cloud. Objectness can be computed from LiDAR intensity such as averaging the LiDAR intensity in a patch. In another example, Radar Doppler information can be used to define motion Objectness.

There are a variety of known Objectness determination techniques that may be used for determining the Objectness of the selected portion. Those skilled in the art who have the benefit of this description will be able to select an appropriate Objectness determination technique to meet their particular needs. For example, a known fast Fourier transform technique, a Walsh Hadamard transform technique, a standard deviation filter technique, a local co-occurrence matrix technique or a global color spatial-distribution technique may be used. Further, the Objectness determination made by the processor <NUM> can be based on a combination of known Objectness measuring techniques.

<FIG> schematically illustrates information obtained by the processor <NUM> regarding an output, such as an image, <NUM> from the camera <NUM>. A plurality of data <NUM>, such as detection points, are based on an output from the detector <NUM>. The data <NUM> correspond to detection by the detector <NUM> that may indicate an object within the field of view of the detector <NUM> and the corresponding portion of the camera field of view.

As can be appreciated from the illustration, the entire camera output or image <NUM> does not contain data corresponding to an output from the detector <NUM>. One feature of this example embodiment is that the processor <NUM> need not consider the entire camera output <NUM>. Instead, portions or areas of the camera output <NUM> that do not contain information corresponding to an output from the detector <NUM> are ignored by the processor <NUM> when determining an Objectness of the camera output.

Instead, the processor <NUM> is configured to select one or more portions of the camera output <NUM> that include information from the detector <NUM> regarding a potential object in such a portion of the camera output. The processor <NUM> has information or programming that relates positions from the output of the detector <NUM> to positions or areas within the camera output <NUM>. In <FIG>, portions <NUM> each include a sufficient number of data points or indications corresponding to information from the detector <NUM> that may indicate the presence of an object. The portions <NUM> in this example are considered clusters of such information or data points from the detector <NUM>. The manner in which the clusters <NUM> are determined by the processor <NUM> or another processor associated with the detector <NUM> may vary depending on the particular embodiment. An example cluster determination technique is described below, which is particularly useful with LIDAR-based detectors. On the other hand, for an embodiment using Radar the detection region may be created from selecting a region around the radar detection as it is sparse compared to LiDAR. The size of the region can be set based on the range of the detection.

<FIG> schematically illustrates an example technique in which the processor <NUM> selects a portion <NUM> of the camera output <NUM> and determines an Objectness of that portion <NUM>. The processor <NUM> divides the portion <NUM> into a plurality of windows or segments and determines an Objectness of each of those segments. The processor <NUM> in this example also determines a total Objectness for the entire portion <NUM>. In some examples, the Objectness determination or object detection output from the processor <NUM> is based on a combination of the respective Objectness of the individual segments of the portion <NUM> and the total Objectness determination regarding the entire portion <NUM>.

The processor <NUM> in this example arranges different segments within the portion <NUM> so that the different segments have different geometries. For example, a first segment <NUM> has a rectangular geometry that appears relatively tall and narrow in the illustration. The data points or information <NUM> within that segment <NUM> have a spatial distribution such that a relatively long and narrow rectangular segment geometry fits well or corresponds to the arrangement or spatial orientation of that data or information from the detector <NUM>. Other segments <NUM>, <NUM> and <NUM> are also rectangular but closer to a square shape because the data points or information <NUM> within each of those windows or segments fits better with a square shaped segment.

In some examples, the processor <NUM> divides the portion <NUM> into equally sized and similarly configured segments for purposes of determining an Objectness of each of those segments.

Objectness for each window or segment is determined in some embodiments by dividing each of the segments <NUM>-<NUM> into multiple small windows or patches. In some embodiments the patches comprise superpixels. An example configuration of patches <NUM> for the example segment <NUM> is schematically shown in <FIG>. Although rectangular patches <NUM> are illustrated in a <NUM> x <NUM> matrix arrangement that configuration is an example for discussion purposes and other tessellations can be used.

When at least one of the segments is divided into patches <NUM>, the processor <NUM> determines an Objectness score for each patch <NUM> based on the distinctiveness of the respective patches <NUM> with respect to the surrounding patches. The parameters for calculating the score are based on one or more known techniques in this embodiment such as saliency, Multiscale PCA (Principle Component Analysis), Fast Fourier Transform, Walsh Hadamard Transform, Local Co -Occurrence Matrix, HOG, edge density, etc. Objectness can be defined from a combination of the scores at both the patch and full segment or window level. For example, the Objectness of each patch <NUM> provides one measure or determination while the total Objectness of the entire segment <NUM> provides another measure. The processor <NUM> in some embodiments determines the Objectness of each segment based on the total Objectness of the segment and the Objectness of each of the patches within that segment. Multiple Objectness determinations at the patch and segment or level can be combined as needed. Most of the previous work on Objectness has focused only on the total Objectness of a single image.

The Objectness determination is based on information from the output of the camera <NUM>. The processor <NUM> uses known image processing and Objectness determining techniques in this example. The Objectness determination is not based on the data or information from the detector <NUM>. Instead, the information from the detector <NUM> is used for locating the portions of the camera output that are more likely to contain an object than other portions of the camera output. Once those portions have been identified, the processor <NUM> is able to focus in on selected portions of the camera output for purposes of making an Objectness determination. Utilizing information from at least one other type of detector to direct or focus the Objectness determinations by the processor <NUM> reduces the computational load on the processor <NUM>. The processor <NUM> does not need to process or analyze the entire image or camera output <NUM>. The disclosed example technique increases processing speed and reduces processing complexity without sacrificing accuracy of object detection, location, identification, or tracking.

The output from the detector <NUM> does not need to be part of the Objectness determination based on the camera output but it may be used in combination with the Objectness determination to provide additional information regarding a detected object. For example, the detector output may provide more detailed location or three-dimensional information regarding a detected object.

The processor <NUM> in some examples is configured to rank the Objectness of the various segments <NUM>-<NUM> within a selected portion <NUM>. The segment or segments having a higher rank are selected by the processor <NUM> to identify or locate an object within the output or field of view of the camera <NUM>.

In some examples, the processor <NUM> utilizes a series of camera outputs that are related in a time sequence. For example, the camera <NUM> provides a sequence of images over time. The processor <NUM> utilizes the disclosed example technique for selecting portions of each of those images based on information from the detector <NUM> and makes Objectness determinations regarding those portions. Over time the processor <NUM> is able to track the position or movement of an object detected within the output of the camera <NUM>.

In some embodiments, the processor <NUM> is configured to consider multiple frame Objectness and add motion cues to the Objectness measure. For this case, a motion cue is computed from a sequence of images and the Objectness measure is defined as Objectness = motion Objectness + segment Objectness + patch Objectness. Although summation is used in this example, other groupings can be used to combine the different measures.

As mentioned above, different techniques may be used in different embodiments for processing information from the detector <NUM> to allow the processor <NUM> to select appropriate portions of the camera output within which to perform an Objectness determination for detecting an object. For embodiments that include a LIDAR detector as the detector <NUM>, a clustering technique is useful. One example clustering technique includes segmenting a point-cloud from a LIDAR detector.

The following paragraphs describe an example methodology for the segmentation of a point-cloud received by a <NUM>° coverage LIDAR sensor. First, a deterministic iterative multiple plane fitting technique named Ground Plane Fitting (GPF) is presented for the fast extraction of the ground points. Next is a point-cloud clustering methodology named Scan Line Run (SLR) which is directed to algorithms for connected components labeling in binary images from a LIDAR. Each paragraph is conceptually divided in three sections including a brief reasoning behind the algorithm selection along with the definition of new terms, the overview of the algorithm according to the pseudo-code diagrams, and discussion of algorithm implementation details.

Cloud points that belong to the ground surface constitute the majority of a typical point-cloud from a LIDAR, and their removal significantly reduces the number of points involved in the proceeding computations. The identification and extraction of ground-points is suitable for this application for two main reasons: (i) the ground-points are easily identifiable since they are associated with planes, which are primitive geometrical objects with a simple mathematical models; and (ii) it is acceptable to assume that points of the point-cloud with the lowest height values are most likely to belong to the ground surface. This prior knowledge is used to dictate a set of points for the initiation of the algorithm and eliminate the random selection seen in typical plane-fit techniques such as the RANdom Sample Consensus (RANSAC), resulting in much faster convergence.

Generally, a single plane model is insufficient for the representation of the real ground surface as the ground points do not form a perfect plane and the LIDAR measurements introduce significant noise for long distance measurements. It has been observed that in most instances the ground surface exhibits changes in slope which need to be detected. The proposed ground plane fitting technique extends its applicability to such instances of the ground surface by dividing the point-cloud into segments along the x-axis (direction of travel of the vehicle), and applying the ground plane fitting algorithm in each one of those segments.

As depicted in the main loop of Algorithm <NUM>, for each of the point-cloud segments the ground plane fitting starts by deterministically extracting a set of seed points with low height values which are then used to estimate the initial plane model of the ground surface. Each point in the point-cloud segment P is evaluated against the estimated plane model and produces the distance from the point to its orthogonal projection on the candidate plane. This distance is compared to a user defined threshold Thdist, which decides whether the point belongs to the ground surface or not. The points belonging to the ground surface are used as seeds for the refined estimation of a new plane model and the process repeats for Niter number of times. Finally, the ground points resulting from this algorithm for each of the point-cloud segments can be concatenated and provide the entire ground plane.

Algorithm <NUM>: Pseudocode of the ground plane fitting methodology for one segment of the point-cloud. Results: Pg are points belonging to ground surface; Png are points not belonging to ground surface. <IMG>
<IMG>.

The approach for the selection of initial seed points introduces the lowest point representative (LPR), a point defined as the average of the NLPR lowest height value points of the point-cloud. The LPR guarantees that noisy measurements will not affect the plane estimation step. Once the LPR has been computed, it is treated as the lowest height value point of the point-cloud P and the points inside the height threshold Thseeds are used as the initial seeds for the plane model estimation.

For the estimation of the plane, a simple linear model is used: <MAT> which can be rewritten as <MAT> where with N = Trans [a b c] and X = Trans[x y z], and solve for the normal N through the covariance matrix C ∈ R(3x3) as computed by the set of seed points S ∈ R(<NUM>), <MAT> where sm ∈ R(<NUM>) is the mean of all si ∈ S.

The covariance matrix C captures the dispersion of the seed points and its three singular vectors that can be computed by its singular value decomposition (SVD), describe the three main directions of this dispersion. Since the plane is a flat surface, the normal N, which is perpendicular to the plane, indicates the direction with the least variance and is captured by the singular vector corresponding to the smallest singular value. After the acquisition of N, d is directly computed from Eq. <NUM> by substituting X with S which is a good representative for the points belonging to the plane.

The remaining points Png that do not belong to the ground surface need to form or be organized into clusters to be used in higher level post processing schemes. The goal is for each point that is an element of Png (pk ∈ Png) to acquire a label '<NUM>' that is indicative of a cluster identity while using simple mechanisms that will ensure the fast running time and low complexity of the process.

In the case of <NUM>° LIDAR sensor data, the multi-layer structure of the 3D point-cloud strongly resembles the row-wise structure of 2D images with the main differences being the non-uniform number of elements in each layer and the circular shape of each layer. The methodology treats the 3D points as pixels of an image and adapts a two-run connected component labeling technique from binary images [<NPL>] to produce a real time 3D clustering algorithm.

A layer of points that are produced from the same LIDAR ring is named a scan-line. Within each scan-line, elements of the scan-line are organized in vectors of contiguous point runs. As used herein, a run is defined as the number of consecutive non-ground points in a scan line that has the same label. That is, the elements within a run share the same label and are the main building blocks of the clusters.

According to Algorithm <NUM> and without loss of generality, it is assumed that the point-cloud Png is traversed in a raster counterclockwise fashion starting from the top scan-line. The runs of the first scan-line are formed and each receives its own newLabel which is inherited or used for all of the point-elements in the scan-line. The runs of the first scan-line then become the runsAbove and are used to propagate their labels to the runs in the subsequent scan-line. The label is propagated to a new run when the distance between a point of the new run and its nearest neighbor in the prior scan-line above is less than Thmerge. When many points in the same run have nearest neighbors with different inheritable labels, the selected or winning label is the smallest one. On the other hand, when no appropriate nearest neighbors can be found for any of the points in the run, it receives a newLabel. The above are performed in a single pass though the point-cloud and when this is done, a second pass is performed for the final update of the point's labels and the extraction of the clusters.

Algorithm <NUM>: Pseudocode of the scan line run clustering. Results: labels are labels of the non ground points. <IMG>
<IMG>.

The following example with reference to accompanying <FIG> covers the main instances of the methodology with the ground-points indicated by white points (numbered <NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>) and non-ground-points indicated by gray points (numbered <NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>). The gray points (numbers <NUM>,<NUM>,<NUM>,<NUM>, <NUM>, <NUM>, <NUM>) are non-ground points not yet visited. In <FIG> the first scan-line is initialized with two runs so non-ground-points numbered <NUM>, <NUM> are assigned to run #<NUM> (<NUM> inside a triangle) and non-ground-points numbered <NUM>, <NUM> are assigned to run#<NUM>, where the assignment to a run is indicated by a newLabel. <FIG> demonstrates the assignment of a newLabel and the propagation of two labels. In particular, the nearest non-ground neighbor of <NUM> is <NUM> and their distance is greater than Thmerge. In this case, labelsToMerge is empty and point <NUM> represents a new cluster. On the other hand, the nearest non-ground neighbor of <NUM> is <NUM> with their distance smaller than Thmerge, which makes label <NUM> to propagate over to point <NUM>. Similarly, points <NUM> and <NUM> are both close to their respective neighbors <NUM> and <NUM>, and based on the non-empty labelsToMerge, label <NUM> is assigning to them. Next, the final scan-line is considered in <FIG> where one run is present. Points <NUM> and <NUM> have neighbors <NUM> and <NUM> which belong to different clusters and are both appropriate to propagate their label. According to the algorithmic logic the smallest of the two labels (namely label <NUM>) is inherited. Nevertheless, as indicated in <FIG>, the merging of the two labels <NUM> and <NUM> is noted and handled accordingly by the label equivalence resolving technique which is discussed below.

The outline of the algorithm is straight forward, but for an efficient implementation of proposed solutions on (i) how to create runs, (ii) how to look for the nearest neighbor, and (iii) how to resolve label conflicts when merging two or more connected components.

Under the assumption that the points in a scan-line are evenly distributed along the whole scan-line, a smart indexing methodology is utilized that overcomes the problem of the uneven number of elements in the different scan-lines and significantly reduces the number of queries for the nearest neighbor. Assume that each scanline has Ni number of points and that each point owns two indices; one global INDg which represents its position in the whole point-cloud, and one local INDl that identifies the point inside the scanline. One can easily alternate between the indices of the scan-line K by: <MAT>.

Given a point index in scan-line i with local index INDli it is possible to directly find the local index of a neighbor INDlj in the close vicinity of the actual nearest neighbor in the above scan-line j by: <MAT> as well as computing its global index from Eq. <NUM>.

Depending on the distribution of the points inside the scan line, the index might not indicate the nearest neighbor but a close enough point. In this case, it may be necessary to search through a number of its surrounding points for the nearest neighbor, but this number is far smaller than considering the whole scan-line.

In a run, identifying potential neighbors and searching through their surroundings for the best match results in a large overhead that undermines the performance of the algorithm. Bearing this in mind, the proposed solution is to find the nearest neighbors of the first and last points of a run via the smart indexing, form a kdtree structure with all the non-ground points within that range, and use this to search for nearest neighbors.

Two visual examples of the smart indexing can be seen in Figures 4A and 4B. In <FIG>, although the number of points in the two scanlines is quite different, the randomly selected points with local indices <NUM>, <NUM>, <NUM>, and <NUM> in the outer scan-line are indicated as the nearest neighbors of the points with local indices <NUM>, <NUM>, <NUM>, and <NUM> respectively in the inner scan-line. In addition, in <FIG> the distribution of points is highly uneven but smart indexing still succeeds to indicate appropriate neighbors. These cases are common to the first few scanlines when some of their laser beams never return, because of absorption or very high distance. In rare cases where the number of points between consecutive scan-lines is vastly different or a significant portion of the scan-line is missing, the smart indexing will most likely fail. In these cases, the naive solution where the whole scan-line is considered as potential nearest neighbors still produces good results.

iii) The methodology to resolve label merging conflicts is being introduced in [<NPL>] where all the details for the implementation and deep understanding are provided. Following, a brief presentation of the essentials along with a simple example is given.

The conflicts arise when two or more different labeled components need to merge and the solution is given by adding their labels l in the same set S. This way, one connected component is represented by the smallest l in its respective S and a sophisticated collection of three vectors is used to capture their hierarchies and connections. All three vectors have the size of the number of total labels that have been created during the first pass through the point-cloud. Each entry of the first vector "next" stores the next l in its S and the entry for the last l in the S is -<NUM>. Next, the vector "tail" stores the index to the last l of the S. The last vector "rtable" has the assistive role of reporting what the final label of each l would be at any given moment. At the end of the first pass, rtable is used as the look-up table for the final labelling.

Referring now to the example formed by Figures 2A, 2B, and 2C from the point-view of the three vectors. In Figure 2A two labels are created (<NUM> and <NUM>) and the label <NUM>,<NUM> entries are filled. Each of the two sets has only one element thus next entries are both -<NUM>, tail entries show the index of the last element in the S which is <NUM> and <NUM> respectively for the two S, and rtable shows the final representative label. Next, in <FIG> the l3 is created and the vectors are filled the same as before. Finally, in <FIG> the S1 and S2 merge which means that the first entry of next will point to the index of the next element in S1, the tail for both elements in S1 is the same and points at the index of the last element of the set, and rtable is updated to properly depict the final labels.

<FIG> illustrates a non-limiting example of an object-detection system <NUM>, hereafter referred to as the system <NUM>, which is suitable for use on an automated vehicle, a host-vehicle <NUM> for example. As used herein, the term `automated vehicle' is not meant to suggest that fully automated or autonomous operation of the host-vehicle <NUM> is required. It is contemplated that the teachings presented herein are applicable to instances where the host-vehicle <NUM> is entirely manually operated by a human-operator (not shown) except for some small level of automation such as merely providing a warning to the operator of the presence of an object <NUM> and/or automatically operating the brakes of the host-vehicle <NUM> to prevent a collision with the object <NUM>. That is, the host-vehicle <NUM> may be operable in an automated-mode <NUM> which may be a fully autonomous type of operation where the human-operator does little more than designate a destination, and/or the host-vehicle <NUM> may be operable in a manual-mode <NUM> where the human-operator generally controls the steering, accelerator, and brakes of the host-vehicle <NUM>.

The system <NUM> includes a LIDAR <NUM> used to detect a point-cloud <NUM>, see also <FIG>. The point-cloud <NUM> may be indicative of the object <NUM> being proximate to the host-vehicle <NUM>. As shown in <FIG> and many of the figures, the point-cloud <NUM> is organized into a plurality of scan-lines <NUM>. While <FIG> show only three instances of the scan-lines <NUM>, this is only to simplify the drawings. That is, it is recognized that the point-cloud <NUM> from a typical commercially available example of the LIDAR <NUM> will provide a point-cloud with many more scan-lines, sixty-four scan-lines for example.

The system <NUM> includes a controller <NUM> in communication with the LIDAR <NUM>. The controller <NUM> may include a processor (not specifically shown) such as a microprocessor or other control circuitry such as analog and/or digital control circuitry including an application specific integrated circuit (ASIC) for processing data as should be evident to those in the art. The controller <NUM> may include memory (not specifically shown), including non-volatile memory, such as electrically erasable programmable read-only memory (EEPROM) for storing one or more routines, thresholds, and captured data. The one or more routines may be executed by the processor to perform steps for determining the presence and location of the object <NUM> based on signals received by the controller <NUM> from the LIDAR <NUM> as described herein.

The controller <NUM> is configured to classify each detected point in the point-cloud as a ground-point <NUM> or a non-ground-point <NUM>. Several methods have been proposed to distinguish the ground-points <NUM> from the non-ground-points <NUM>, as will be recognized by those in the art.

The controller <NUM> is further configured to define runs <NUM> of non-ground-points <NUM>. Each run <NUM> is characterized as a collection of one or multiple instances of adjacent non-ground-points in an instance of the scan-line <NUM> that is separated from a subsequent run <NUM> of one or more non-ground-points <NUM> by at least one instance of a ground-point <NUM>. That is, each instance of a run <NUM> is defined by one or more instance of the non-ground-points <NUM> that are next to each other (i.e. adjacent to each other) without an intervening instance of a ground-point <NUM>.

The controller <NUM> is further configured to define a cluster <NUM> of non-ground-points associated with the object <NUM>. If multiple objects are present in the field-of-view of the LIDAR <NUM>, there may be multiple instances of point-clouds <NUM> in the point-cloud <NUM>. A cluster <NUM> may be characterized by or include a first run 32A (<FIG>) from a first scan-line 24A being associated with a second run 32B from a second scan-line 24B when a first point 22A from the first run 32A is displaced less than a distance-threshold <NUM> (see 'Thmerge' above) from a second point 22B from the second run 32B.

Accordingly, an object-detection system (the system <NUM>), a controller <NUM> for the system <NUM>, and a method of operating the system <NUM> is provided. The process of organizing the non-ground-points <NUM> into runs <NUM>, and then associating nearby runs <NUM> into clusters <NUM> makes for an efficient way to process the point-cloud data from the LIDAR <NUM>.

Claim 1:
A method of detecting at least one potential object, the method comprising:
obtaining information regarding potential objects from a detector (<NUM>) provided on a vehicle (<NUM>), the detector comprising one of a radar detector or a LIDAR detector, the detector (<NUM>) having a field of view that at least partially overlaps with a field of view (<NUM>) of a camera (<NUM>) of the vehicle, and the information being data points corresponding to potential objects in the field of view (<NUM>) of the camera (<NUM>);
recognizing clustered data points from the detector (<NUM>) that corresponds to a potential object location within the field of view (<NUM>) of the camera (<NUM>) based on the obtained information from the detector (<NUM>);
receiving an image (<NUM>) from the camera (<NUM>) different than the detector (<NUM>);
selecting a portion (<NUM>) of the image (<NUM>) that corresponds to the potential object location;
performing an objectness determination only on the selected portion (<NUM>) of the image (<NUM>) to determine an objectness of the selected portion (<NUM>) and refraining from performing the objectness determination on portions of the image (<NUM>) other than the selected portion (<NUM>) by ignoring the portions of the image (<NUM>) other than the selected portion (<NUM>); and
detecting the at least one potential object within the selected portion (<NUM>) based on the determined objectness,
characterized in that the performing the objectness determination only on the selected portion (<NUM>) comprises:
dividing up the selected portion (<NUM>) into segments (<NUM>-<NUM>); and
performing an objectness determination on each of the segments (<NUM>-<NUM>);
further comprising ranking the objectness of each of the segments (<NUM>-<NUM>) based on the respective determined objectnesses of the segments (<NUM>-<NUM>) and detecting the at least one potential object based on a highest ranking objectness.