METHOD AND DEVICE FOR TRACKING OBJECTS DETECTED THROUGH LIDAR POINTS

A method of tracking objects detected through light detection and ranging (LiDAR) points can include, when two or more objects are moved in a previous frame and classified as one object in a current frame, clustering LiDAR points in the current frame into a plurality of clusters, finding center points of the plurality of clusters in the current frame, matching center points of the two or more objects in the previous frame with the center points of the plurality of clusters in the current frame, and updating positions of the center points of the two or more objects according to the matching in the current frame.

BACKGROUND

1. Field of the Invention

The present invention relates to a method and device for tracking objects detected through light detection and ranging (LiDAR) points, and more specifically, to a method and device for tracking objects detected through LiDAR points that are robust to combination and separation for continuous accurate tracking of objects.

2. Discussion of Related Art

Light detection and ranging (LiDAR) sensors are sensors that use light in the form of pulsed laser to generate maps of objects and the surrounding environment thereof. LiDAR sensors may be used in various fields of autonomous vehicles, mobile robots, and the like.

FIG.1illustrates frames for describing a method of tracking objects detected through LiDAR points according to the related art.

Referring toFIG.1, LiDAR points that may be generated through a LiDAR sensor may be recognized in an Nthframe which is a current frame1, wherein N is a natural number.

The LiDAR points may be segmented to recognize objects in the Nthframe which is the current frame1. Conventional well-known methods (for example, model fitting, boundary-based, graph-based, region-based, and attributes-based methods) may be used to segment the LiDAR points (S1). When the LIDAR points are segmented, a three-dimensional (3D) bounding boxes are marked in a current frame3. One segment may be recognized as one object.

In order to determine to which object of the current frame3an object (for example, an object A) of a previous frame5corresponds, the current frame3in which the LiDAR points are segmented overlaps an (N−1)thframe, which is the previous frame5(S2).

In order to determine to which object of the current frame3an object (for example, the object A) of the previous frame5corresponds, a segment tracking algorithm may be applied to a current frame7overlapping the previous frame5(S3).

As a result of applying the segment tracking algorithm, objects may be annotated in a current frame9. For example, in the current frame9, objects may be annotated with letters “A” “B,” “C,” and “D.”

After the annotating in the current frame9, types of objects (for example, vehicles or pedestrians) may be determined (S4).

FIG.2illustrates frames for describing a method of tracking objects detected through LiDAR points according to the related art.

Referring toFIGS.1and2, in a previous frame5, objects may be annotated with letters “A,” “B,” “C,” and “D.” In the previous frame5, arrows indicate trajectories. Although a bounding box inFIG.2is expressed as a two-dimensional (2D) bounding box, it should be actually understood that the bounding box is a 3D bounding box.

In a current frame7, the current frame7overlap the previous frame5(S2).

Dotted bounding boxes represent bounding boxes of objects in the previous frame5. In the current frame7, objects have not yet been annotated.

A segment tracking algorithm may be applied to annotate the objects in the current frame7(S3). Similarity scores may be used as conventional segment tracking algorithms. The similarity score means that the bounding boxes of the objects in the previous frame5are compared with the bounding boxes of the objects in the current frame7, and a degree of similarity therebetween is expressed as a score.

Korean Patent Publication No. 10-2022-0041485 (Apr. 1, 2022), disclosed is a technique in which a correlation index between a current representative point and a previous representative point of each of a plurality of segment boxes is calculated, and objects in a previous frame5are matched with objects in a current frame7according to the correlation index.

After the segment tracking algorithm is applied, objects in a current frame9may be annotated.

FIG.3illustrates frames for describing a method of tracking objects detected through LiDAR points according to the related art.FIG.3is similar toFIG.2.

Referring toFIGS.1and3, in order to calculate a similarity score, objects “A,” “B,” and “C” in a previous frame5are compared with objects1,2, and3in a current frame7. For example, the object1in the current frame7is compared with the objects “A,” “B,” and “C” in the previous frame5. The object2in the current frame7is compared with the objects “A,” “B,” and “C” in the previous frame5. The object3in the current frame7is compared with the objects “A,” “B,” and “C” in the previous frame5. In the current frame7,1,2, and3are reference numbers assigned to describe the similarity score.

According to the similarity score, the objects1,2, and3in a current frame9may be annotated with letters “A,” “B,” and “C.”

FIG.4illustrates frames for describing a method of tracking objects detected through LiDAR points according to the related art.

Referring toFIGS.1and4, objects “A,” “B,” and “C” in a previous frame5are moved and classified as one object X in a current frame7. By using conventional well-known methods (for example, model fitting, boundary-based, graph-based, region-based, and attributes-based methods), LiDAR points in the current frame7are classified as one segment, that is, the object X. Since the objects “A,” “B,” and “C” are clustered close to each other, the objects “A,” “B,” and “C” are classified as one object rather than three objects in the current frame7. In this case, the size of a bounding box also changes. Since three objects are gathered together, the size of the bounding box increases. The bounding box in the current frame7has not yet been annotated, but is arbitrarily denoted as “X” for convenience of description.

In the current frame7, the current frame7overlaps the previous frame5(S2).

A segment tracking algorithm may be applied to annotate the object X in the current frame7(S3).

The objects “A,” “B,” and “C” in the previous frame5are compared with the object X in the current frame7to calculate similarity scores. It is assumed that the similarity score between the object “A” in the previous frame (5) and the object X in the current frame7is the highest, in a current frame9, the object X may be annotated with the letter “A.”

In this case, according to the related art, history information about the object “B” and the object “C” in the previous frame5is deleted in the current frame7. In paragraph number of Korean Patent Publication No. 10-2022-0041485 (Apr. 1, 2022), it is described that “when an associated segment box does not exist, history information about an mthchannel for which the associated segment box does not exist may be deleted.” That is, according to Korean Patent Publication No. 10-2022-0041485 (Apr. 1, 2022), since the similarity score between the object “B” in the previous frame5and the object X in the current frame7and the similarity score between the object “C” in the previous frame5and the object X in the current frame7are not the highest, history information about object “B” and the object “C” is deleted. The history information includes position information and speed information about the object “B” and the object “C” in the previous frame5.

It is assumed that objects “D” and “E in a next frame8correspond to the objects “B” and “C” in the previous frame5. However, according to the related art, the history information about the object “B” and the object “C” in the previous frame5is deleted in the current frame7, and thus, in the next frame8, the objects are not annotated with the letter “B or “C”, but with another letter “D” or “E.” That is, the related art has a problem in that tracking of the objects “B” and “C” in the previous frame5is lost in the next frame8. The present invention is intended to solve this problem.

RELATED ART DOCUMENTS

Patent Documents

SUMMARY OF THE INVENTION

The present invention is directed to providing a method and device for tracking objects detected through light detection and ranging (LiDAR) points that are robust to combination and separation for continuous accurate tracking of objects.

According to an aspect of the present invention, there is provided a method of tracking objects detected through LiDAR points, the method including, when two or more objects are moved in a previous frame and classified as one object in a current frame, clustering LiDAR points in the current frame into a plurality of clusters equal to the number of objects counted in the previous frame, finding center points of the plurality of clusters in the current frame, matching center points of the two or more objects in the previous frame with the center points of the plurality of clusters in the current frame, and updating positions of the center points of the plurality of clusters according to the matching in the current frame.

The method may further include classifying the LiDAR points into the two or more objects in the previous frame, and classifying the two or more objects as one object in the current frame.

The method may further include calculating similarity scores between the one object classified in the current frame and each of the two or more objects in the previous frame, storing a position of a center point of an object in the previous frame corresponding to a highest similarity score among the similarity scores in the current frame, and storing a position of a center point of an object in the previous frame corresponding to a remaining similarity score excluding the highest similarity score among the similarity scores in the current frame.

The method may further include assigning an ID of the object in the previous frame corresponding to the highest similarity score as an ID of the one object classified in the current frame.

The method may further include assigning a first sub-ID to the object in the previous frame corresponding to the highest similarity score among the similarity scores in the current frame, and assigning a second sub-ID to the object in the previous frame corresponding to the remaining similarity score excluding the highest similarity score among the similarity scores in the current frame.

The first sub-ID may include an ID of the object in the previous frame corresponding to the highest similarity score among the similarity scores.

The second sub-ID may include an ID of the object in the previous frame corresponding to the remaining similarity score excluding the highest similarity score among the similarity scores.

The method may further include, when the one object is classified into the two or more objects in a next frame, assigning IDs to the two or more objects in the next frame according to the updated positions of the center points of the two or more objects in the current frame.

The IDs of the two or more objects in the next frame may correspond to IDs of the two or more objects in the previous frame.

The clustering of the LiDAR points in the current frame into the plurality of clusters may include counting the number of objects in the previous frame, and clustering the LiDAR points in the current frame into the plurality of clusters equal to the number of objects counted in the previous frame.

The matching of the center points of the two or more objects in the previous frame with the center points of the plurality of clusters in the current frame may include calculating a distance between each of the center points of the two or more objects in the previous frame and each of the center points of the plurality of clusters in the current frame, and matching points having shortest distances among the calculated distance.

According to another aspect of the present invention, there is provided a device including a processor configured to execute instructions, and a memory configured to store the instructions.

The instructions may be implemented to, when two or more objects are moved in a previous frame and classified as one object in a current frame, cluster LDAR points in the current frame into a plurality of clusters equal to the number of objects counted in the previous frame, find center points of the plurality of clusters in the current frame, match center points of the two or more objects in the previous frame with the center points of the plurality of clusters in the current frame, and update positions of the center points of the two or more objects according to the matching in the current frame.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG.5is a block diagram of a system for tracking objects through light detection and ranging (LiDAR) points according to an embodiment of the present invention.

Referring toFIG.5, a system100for tracking objects through LiDAR points23is a system for tracking objects detected through the LiDAR points23.

A vehicle103, a pedestrian105, or the like may be an object.

The system100for tracking the objects detected through the LiDAR points23may include a vehicle20.

The system100for tracking the objects detected through the LiDAR points23includes a computing device10.

The vehicle20includes a LiDAR sensor21. In addition, the vehicle20may include the computing device10. The LiDAR sensor21generates LiDAR point data25including the LiDAR points23. The LiDAR points23are a plurality of three-dimensional (3D) points.

As the vehicle20moves, the LiDAR sensor21installed in the vehicle20generates the LiDAR point data25about various surrounding environments of the vehicle20. The LiDAR point data25includes the LiDAR points23. That is, the LiDAR point data25refers to the LiDAR points23. The LiDAR points23are 3D point clouds, LiDAR points, or point clouds. According to embodiments, the LiDAR sensor21may be installed in various objects such as fixed objects or robots for sensing.

The computing device10may be implemented as a hardware module combined with other hardware inside the vehicle20or as an independent hardware device. For example, the computing device10may be implemented in an electronic control unit (ECU) of the vehicle20. In addition, the computing device10may be implemented as an external electronic device such as a computer, a laptop, personnel computer (PC), a server, or a tablet PC.

The computing device10includes a processor11and memory13. The processor11executes instructions for tracking objects detected through the LiDAR points23. The memory13stores the instructions.

The processor11receives the LiDAR point data25including the LiDAR points23from the LiDAR sensor21.

FIG.6illustrates frames for describing a method of tracking objects detected through LiDAR points according to an embodiment of the present invention.

Referring toFIGS.5and6, the processor11receives the LiDAR point data25from the LiDAR sensor21and recognizes the LiDAR points23included in the LiDAR point data25.

The processor11segments the LiDAR points23to recognize objects (for example, the vehicle103or the pedestrian105) in an (N−2)thframe (N is a natural number). For a segmentation operation, well-known conventional methods described inFIG.1are used. Objects may be recognized by segmenting the LiDAR points23. In this case, the LIDAR points23are 3D point clouds generated in an (N−2)thframe30.

A frame refers to a 3D map of a scene generated from the LiDAR points23. The system100for tracking the objects detected through the LiDAR points23may have a frame rate between 10 frames per second and 30 frames per second.

When the LiDAR points23are segmented, the processor11marks 3D bounding boxes in the (N−2)thframe30. One segment may be recognized as one object. An object may be a vehicle, a pedestrian, or an obstacle.

The processor11stores coordinates of each vertex of the bounding boxes and widths, lengths, and heights of the bounding boxes in the (N−2)thframe30in a storage space (for example, the memory13). In addition, the processor11stores x, y, and z coordinates of the LiDAR points23included in the bounding boxes in the (N−2)thframe30in the storage space (for example, the memory13). InFIG.6, the bounding box is expressed as a two-dimensional (2D) box, but is actually expressed as a 3D bounding box.

The processor11may annotate the recognized objects. For example, the processor11may annotate the objects in the (N−2)thframe30with letters “A,” “B,” and “C.” According to embodiments, the annotation may be made in various ways using numbers or a combination of letters and numbers.

After the annotation is made in the (N−2)thframe30, the processor11may determine types of the objects (for example, vehicles pedestrians or obstacles).

The processor11stores IDs, ages, speeds, trajectories, and types of the objects in the (N−2)thframe30in the storage space (for example, the memory13).

The ID refers to letters for objects (for example, “A,” “B,” and “C”). Objects may be identified by the ID.

The age refers to the number of frames that continuously inherit the ID after the object is annotated.

The speed refers to the speed of each object. The trajectory refers to a trajectory along which the object has moved during any previous frames. Arrows refer to the trajectories in the (N−2)thframe30.

In the storage space (for example, the memory13), the age, speed, trajectory, type, and position of the bounding box of the object may be stored for each frame.

The processor11segments the LiDAR points23to recognize objects in the (N−1)thframe40. In this case, the LIDAR points23are 3D point clouds generated in the (N−1)thframe40.

In order to determine to which object of the (N−1)thframe40an object (for example, an object A) of the (N−2)thframe30corresponds, the processor11causes the (N−2)thframe30, in which the LiDAR points23are segmented, to overlap the (N−1)thframe40.

In order to determine to which object of the (N−1)thframe40the object (for example, the object A) of the (N−2)thframe30corresponds, a segment tracking algorithm may be applied to the (N−1)thframe40overlapping the (N−2)thframe30.

A similarity score may be used as the segment tracking algorithm. The similarity score means that bounding boxes of objects in the (N−2)thframe30are compared with bounding boxes of objects in the (N−1)thframe40, and a degree of similarity therebetween is expressed as a score. According to the similarity score, the objects in the (N−1)thframe40may be annotated with letters “A,” “B,” and “C.”

The processor11stores coordinates of each vertex of the bounding boxes and widths, lengths, and heights of the bounding boxes in the (N−1)thframe40in the storage space (for example, the memory13).

The processor11stores IDs, ages, speeds, trajectories, and types of the objects in the (N−1)thframe40in the storage space (for example, the memory13).

When two or more objects “A,” “B,” and “C” in a previous frame40move, the processor11may classify the two or more objects “A,” “B,” and “C” as one object in a current frame45. The previous frame40refers to the (N−1)thframe40. The current frame45refers to an Nthframe45.

The classification refers to segmentation. That is, the processor11may segment LiDAR points52,54, and56in the current frame45into one object. When the objects “A,” “B,” and “C” in the previous frame40are clustered close together, the processor11segments the LiDAR points52,54, and56in the current frame45into one object. When the LiDAR points52,54, and56in the current frame45are segmented into one object, a bounding box50corresponding to one object becomes larger than a bounding box in the previous frame40.

The processor11calculates similarity scores between one object classified in the current frame45and each of the objects “A,” “B,” and “C” in the previous frame40.

The processor11assigns an ID (for example, “A”) of an object in the previous frame40corresponding to the highest similarity score as an ID (for example, “A”) of one object classified in the current frame45.

The processor11stores coordinates of each vertex of the bounding box50and a width, a length, and a height of the bounding box50in the current frame45in the storage space (for example, the memory13). The processor11stores the ID (for example, “A”) of the object corresponding to the highest similarity score in the current frame45, an age of the object (for example, “A”), a speed of the object (for example, “A”), a trajectory of the object (for example, “A”), and a type of the object (for example, “A”) in the storage space (for example, the memory13).

Even in the related art, the ID, age, speed, trajectory, and type of the object “A” in the current frame45are stored in the storage space (for example, the memory13). However, in the related art, IDs, ages, speeds, trajectories, and types of the objects “B” and “C” in the current frame45are not stored in the storage space (for example, the memory13), but are deleted. The objects “B” and “C” are not objects corresponding to the highest similarity score. This is because there is no bounding box corresponding to the objects “B” and “C” in the current frame45.

The processor11stores a position of a center point51of the object (for example, “A”) in the previous frame40corresponding to the highest similarity score among similarity scores in the current frame45in the storage space (for example, the memory13). When the LiDAR points23are segmented in the previous frame40, the position of the center point51may be calculated as an average value of the LiDAR points23included in a segment.

The processor11assigns the ID (for example, “A”) of the object in the previous frame40corresponding to the highest similarity score as a first sub-ID (for example, “A”). The first sub-ID is an ID different from the ID of the object. According to embodiments, a sub-ID may be made in various ways using numbers or a combination of letters and numbers.

In the current frame45, the processor11stores the first sub-ID, an age, a speed, a trajectory, and a type of the object (for example, “A”) in the previous frame40in the storage space (for example, the memory13). In addition, in the current frame45, the processor11may store coordinates of each vertex, a width, a length, and a height of a bounding box of the object (for example, “A”) in the previous frame40in the storage space (for example, the memory13).

The processor11stores positions of center points53and55of objects (for example, “B” and “C”) in the previous frame40, which correspond to the remaining similarity scores excluding the highest similarity score among the similarity scores in the current frame45, in the storage space (for example, the memory13).

The processor11assigns IDs (for example, “B” and “C” of the objects in the previous frame40corresponding to the remaining similarity scores as second sub-IDs (for example, “B” and “C”).

In the current frame45, the processor11stores the second sub-IDs, ages, speeds, trajectories, and types of the objects (for example, “B” and “C”) in the previous frame40in the storage space (for example, the memory13). In the related art, in the current frame45, the second sub-IDs of the objects (for example, “B” and “C”) in the previous frame40are not stored in the storage space (for example, memory13).

In addition, in the current frame45, the processor11may store coordinates of each vertex, widths, lengths, and heights of bounding boxes of the objects (for example, “B” and “C”) in the previous frame40in the storage space (for example, the memory13).

The processor11stores the first sub-ID and the second sub-IDs of the objects “A,” “B,” and “C” in the current frame45. In the current frame45, the processor11stores history information about the objects “A,” “B,” and “C” in the previous frame40in the storage space (for example, the memory13). The history information includes an age, a speed, a trajectory, a type, coordinates of a bounding box, a width of the bounding box, a length of the bounding box, a height of the bounding box, or the like in the previous frame40. The first sub-ID may be called a parent, and the second sub-IDs may be called children.

The processor11clusters the LiDAR points52,54, and56in the current frame45into a plurality of clusters C1, C2, and C3. Clustering refers to a segmentation operation. That is, the processor11resegments the LiDAR points52,54, and56in the current frame45into a plurality of objects C1, C2, and C3. The LiDAR points52,54, and56in the current frame45are segmented into one object “A.” A first scale when the LiDAR points52,54, and56are segmented into one object “A” is different from a second scale when the LiDAR points52,54, and56are clustered into the plurality of clusters C1, C2, and C3. The second scale when the LiDAR points52,54, and56are clustered into the plurality of clusters C1, C2, and C3is finer than the first scale when the LiDAR points52,54, and56are segmented into one object “A.” For example, in the case of the first scale, when a Euclidean distance between two LiDAR points is a first random distance (50 cm) or more, the two LiDAR points may be classified into different segments. On the other hand, in the case of the second scale, which is a fine scale, even when a Euclidean distance between two LiDAR points is a second random distance (20 cm) or more and the first random distance (50 cm) or less, the two LiDAR points may be classified into different segments. Accordingly, the LiDAR points52,54, and56may be clustered into the plurality of clusters C1, C2, and C3.

When the first scale when the LiDAR points52,54, and56are segmented into one object “A” is the same as the fine second scale when the LiDAR points52,54, and56are clustered into the plurality of clusters C1, C2, and, an amount of calculation increases, and thus the burden on the processor11increases. However, in the present invention, only when the LiDAR points52,54, and56are segmented into one object “A,” the LiDAR points52,54, and56are clustered into the plurality of clusters C1, C2, and C3in a fine scale, and thus the burden on the processor11may be reduced.

In the related art, an operation of clustering the LiDAR points52,54, and56into the plurality of clusters C1, C2, and C3is not performed.

In order to cluster the LiDAR points52,54, and56into the plurality of clusters C1, C2, and C3, the processor11counts the number (for example, 3) of objects in the previous frame40.

The processor11may cluster the LiDAR points52,54, and56in the current frame45into a plurality of clusters (for example, C1, C2, and C3) equal to the number (for example, 3) of objects counted in the previous frame40. AK-means clustering algorithm may be used to cluster the LiDAR points52,54, and56into the plurality of clusters (for example, C1, C2, and C3). Reference numeral50denotes a bounding box for the object “A” generated in the current frame45. Reference numeral60denotes a virtual bounding box. The virtual bounding box refers to a box illustrated for convenience of description. Actually, the virtual bounding box may not be present.

An algorithm (for example, a region-based algorithm) when the LiDAR points52,54, and56in the current frame45are classified as one object “A” may be the same as an algorithm (for example, a region-based algorithm) when the LiDAR points52,54, and56are reclassified into a plurality of clusters (for example, C1, C2, and C3).

According to embodiments, an algorithm (for example, a region-based algorithm) when the LiDAR points52,54, and56in the current frame45are classified as one object “A” may be different from an algorithm (for example, a K-means clustering algorithm) when the LiDAR points52,54, and56are reclassified into a plurality of clusters (for example, C1, C2, and C3).

The processor11finds center points61,63, and65of the plurality of clusters (for example, C1, C2, and C3) in the current frame45. That is, the processor11finds positions of the center points61,63, and65of the plurality of clusters (for example,1, C2, and C3) in the current frame45. The positions of the center points61,63, and65may be calculated as an average value of the LIDAR points52,54, and56included in the clusters (for example, C1, C2, and C3).

The processor11matches the center points51,53, and55of the two or more objects in the previous frame40with the center points61,63, and65of the plurality of clusters C1, C2, and C3in the current frame45. The matching refers to a process of finding the center points61,63, and65of the clusters C1, C2, and C3that correspond to the center points51,53, and55of the objects. In order to find the center points61,63, and65of the clusters C1, C2, and C3corresponding to the center points51,53, and55of the objects, Euclidean distances between the center points51,53, and55of the objects and the center points61,63, and65of the clusters C1, C2, and C3may be calculated. Points, which have the shortest Euclidean distances among the Euclidean distances between the center points51,53, and55of the objects and the center points61,63, and65of the clusters C1, C2, and C3, are recognized as corresponding points (for example,51and61,53and63, and55and65).

The center point51of the object in the previous frame40corresponds to the center point61of the cluster C1in the current frame45corresponding thereto. The center point53of the object in the previous frame40corresponds to the center point63of the cluster C2in the current frame45corresponding thereto. The center point55of the object in the previous frame40corresponds to the center point65of the cluster C3in the current frame45corresponding thereto.

Reference numeral70denotes a virtual bounding box. The virtual bounding box refers to a box illustrated for convenience of description. Actually, the virtual bounding box may not be present. The matching results of the center points51,53, and55of the objects in the virtual bounding box70and the center points61,63, and65of the plurality of clusters C1, C2, and C3are shown. The LiDAR points52,54, and56are enlarged and shown in the virtual bounding box70.

The positions of the center points51,53, and55of the two or more objects in the previous frame40may be different from the positions of the center points61,63, and65of the plurality of clusters C1, C2, and C3in the current frame45. This is because the LiDAR points52,54, and56have moved in the current frame45. Arrows in the virtual bounding box70indicate movement of the center points.

The center point51in the previous frame40has moved to the center point61in the current frame45. The center point53in the previous frame40has moved to the center point63in the current frame45. The center point55in the previous frame40has moved to the center point65in the current frame45.

The processor11updates the positions of the center points51,53, and55of the two or more objects according to the matching in a current frame90.

The updating means that the positions of the center points51,53, and55of the two or more objects in the current frame90are set to the positions of the center points61,63, and65of the plurality of clusters C1, C2, and C3.

Reference numeral80denotes a virtual bounding box. The virtual bounding box refers to a box illustrated for convenience of description. Actually, the virtual bounding box may not be present. The LiDAR points52,54, and56are enlarged and shown in the virtual bounding box80.

The positions of the LiDAR points52,54, and56and the updated center points61,63, and65in the current frame90are shown.

FIG.7is a flowchart for describing a method of tracking objects detected through LiDAR points according to an embodiment of the present invention.

Referring toFIGS.5to7, the processor11classifies LiDAR points23into two or more objects “A,” “B,” and “C” in a previous frame40, which is an (N−1)thframe (S10). That is, the processor11segments the LiDAR points23to recognize objects in the previous frame40, which is the (N−1)thframe.

When two or more objects “A,” “B,” and “C” in the previous frame40move, the processor11may classify the two or more objects “A,” “B,” and “C” as one object in a current frame45(S20).

The processor11clusters the LiDAR points52,54, and56in the current frame45into a plurality of clusters C1, C2, and C3(S30). Specifically, the processor11counts the number (for example, 3) of objects in the previous frame40. The processor11clusters the LiDAR points52,54, and56in the current frame45into a plurality of clusters C1, C2, and C3equal to the number (for example, 3) of objects counted in the previous frame40.

The processor11finds center points61,63, and65of the plurality of clusters C1, C2, and C3in the current frame45(S40).

FIG.8is a flowchart for describing a method of tracking objects detected through LiDAR points according to an embodiment of the present invention.

Referring toFIGS.5to8, the processor11calculates similarity scores between one object classified in a current frame45and each of objects “A,” “B,” and “C” in a previous frame40(S41). Specifically, the processor11calculates similarity scores between a bounding box50corresponding to one object classified in the current frame45and bounding boxes corresponding to the objects “A,” “B,” and “C” in the previous frame40.

The processor11stores a position of a center point of the object “A” in the previous frame40corresponding to the highest similarity score among the similarity scores in the current frame45(S43). That is, in the current frame45, the processor11stores a position of a center point51of the object “A” in the previous frame40in the storage space (for example, the memory13).

The processor11stores positions of center points53and55of objects “B” and “C” in the previous frame40, which correspond to the remaining similarity scores excluding the highest similarity score among the similarity scores, in the current frame45(S45). That is, in the current frame45, the processor11stores the positions of the center points53and55of the objects “B” and “C” in the previous frame40in the storage space (for example, the memory13).

The processor11assigns an ID (for example, “A”) of an object in the previous frame40corresponding to the highest similarity score as an ID (for example, “A”) of one object classified in the current frame45(S47).

In the current frame45, the processor11assigns the ID of the object in the previous frame40corresponding to the highest similarity score among the similarity scores as a first sub-ID (for example, “A”) (S48).

The first sub-ID (for example, “A”) includes the ID (for example, “A”) of the object in the previous frame40corresponding to the highest similarity score among the similarity scores.

According to embodiments, the first sub-ID may be different from the ID of the object. For example, the first sub-ID may be a lowercase letter “a,” and the ID of the object may be an uppercase letter “A.”

The processor11assigns IDs of objects in the previous frame40corresponding to the remaining similarity scores excluding the highest similarity score among the similarity scores in the current frame45as second sub-IDs (for example, “B” and “C”) (S49).

The second sub-IDs (for example, “B” and “C”) includes the IDs (for example, “B and “C”) of the objects in the previous frame40corresponding to the remaining similarity scores excluding the highest similarity score among the similarity scores.

According to embodiments, the second sub-IDs may be different from the IDs (for example, “B” and “C”) of the objects in the previous frame40. For example, the second sub-IDs may be lowercase letters “b” and “c,” and the IDs of the objects in the previous frame40may be uppercase letters “B” and “C.”

Referring toFIG.7, the processor11matches the center points51,53, and55of the two or more objects in the previous frame40with the center points61,63, and65of the plurality of clusters C1, C2, and C3in the current frame45(S50).

The processor11updates the positions of the center points51,53, and55of the two or more objects according to the matching in a current frame90(S60).

When one object is classified into two or more objects in a next frame (not shown), the processor11assigns IDs to the two or more objects in the next frame according to the updated positions of the center points of the two or more objects in the current frame90(S70).

FIG.9illustrates frames for describing the application of a method of tracking objects detected through LiDAR points according to an embodiment of the present invention.

Referring toFIGS.5and7to9, the processor11updates positions of center points A′, B′; and C′ of two or more objects according to the matching in an Nthframe.

The processor11performs operations S30to S60ofFIG.7in an (N+1)thframe to update the positions of the center points A′, B,′ and C′ of the two or more objects.

It is assumed that LiDAR points from (N+2)thto (N+4)thframes are segmented into one object. The processor11performs operations S30to S60ofFIG.7in each of the (N+2)thto (N+4)thframes to update the positions of the center points A′, B′, and C′ of the two or more objects.

When the LiDAR points in an (N+5)thframe move, the processor11may segment the LiDAR points into three objects.

When one object is classified into three objects in the (N+5)thframe, the processor11assigns IDs to three objects in the (N+5)thframe according to updated positions of center points of the three objects in the (N+4)thframe.

The processor11classifies the LiDAR points into three objects in the (N+5)thframe. The processor11calculates the center points of the three objects in the (N+5)thframe.

The processor11compares the positions of the center points of the objects in the (N+4)thframe with the positions of the center points in the (N+5)thframe. Specifically, the processor11calculates distances between the positions of the center points of the objects in the (N+4)thframe and the positions of the center points in the (N+5)thframe.

The processor11matches the points, which have the shortest distances among the distances between the positions of the center points of the objects in the (N+4)thframe and the positions of the center points in the (N+5)thframe.

The processor11may annotate the objects in the (N+5)thframe according to the corresponding points. For example, the processor11may annotate the objects in the (N+5)thframe with letters “A,” “B,” and “C.”

FIG.10illustrates frames for describing a method of tracking objects detected through LiDAR points according to an embodiment of the present invention.FIGS.10A,10B,10C, and10Dillustrate frames in chronological order.FIG.10Aillustrates a previous frame, andFIG.10Billustrates a more recent frame than that ofFIG.10A.FIG.10Cillustrates a more recent frame than that ofFIG.10B.FIG.10Cillustrates the most recent frame. The frames ofFIGS.10A,10B,10C, and10Dare not consecutive frames.

Referring toFIGS.5and10A, the processor11segments LiDAR points into two objects. InFIG.10A, bounding boxes corresponding to two objects, trajectories, IDs, and the number of persisting frames are shown. For example, among “150” and “138” inFIG.10A,150denotes an ID, and 138 denotes the number of persisting frames. An object with the ID150has persisted from 138 previous frames. That is, the object with the ID150has been tracked from 138 previous frames. InFIG.10A, a line connected to the bounding box represents a trajectory.

Referring toFIGS.5and10B, the processor11segments LiDAR points into one object. The processor11calculates similarity scores between a bounding box in the frame shown inFIG.10Band the bounding boxes in the frame shown inFIG.10A. Assuming that the similarity score between the bounding box with the ID150inFIG.10Aand the bounding box inFIG.10Bis the highest, the processor11assigns150as the ID of the bounding box inFIG.10B.

The processor11may cluster the LiDAR points in the frame ofFIG.10Binto two clusters equal to the number (for example, 2) of objects counted in the frame ofFIG.10A.

The processor11assigns an ID of an object in the frame ofFIG.10Acorresponding to the highest similarity score among the similarity scores in the frame ofFIG.10Bas a first sub-ID (for example, “150”).

The processor11assigns an ID of an object in the frame ofFIG.10Acorresponding to the remaining similarity score excluding the highest similarity score among the similarity scores in the frameFIG.10Bas a second sub-ID (for example, “144”).

The processor11finds center points of the two clusters in the frame ofFIG.10B.

The processor11matches the center points of two objects in the frame ofFIG.10Awith center points of two clusters in the frame ofFIG.10B.

The processor11updates the positions of the center points of the two objects according to the matching in the frame ofFIG.10B.

Referring toFIGS.5and10C, the LiDAR points have moved, but are still segmented into one object. Similar toFIG.10B, the processor11updates the positions of the center points of the two objects according to the matching in the frame ofFIG.10C.

Referring toFIGS.5and10D, the LiDAR points have moved. Therefore, the processor11segments the LiDAR points into two objects.

The processor11may assign IDs to the two objects segmented in the frame ofFIG.10Dusing the positions of the center points of the clusters in the frame ofFIG.10C. The IDs of the objects assigned in the frame ofFIG.10Dare “50” and “144,” which correspond to the IDs of the objects assigned in the frame ofFIG.10A.

With a method and device for tracking objects detected through LiDAR points according to an embodiment of the present invention, when two or more objects in the previous frame are moved in the current frame and classified as one object, by storing the information on objects in the previous frame in the current without deleting it, it is possible to accurately track the objects even if one object is separated into two or more objects.

The present invention has been described with reference to embodiments shown in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached claims.