Patent ID: 12189055

DETAILED DESCRIPTION

Hereafter, example embodiments will be described in detail with reference to the accompanying drawings so that the present disclosure may be readily implemented by those skilled in the art. However, it is to be noted that the present disclosure is not limited to the example embodiments but can be embodied in various other ways. In the drawings, parts irrelevant to the description are omitted for the simplicity of explanation, and like reference numerals denote like parts through the whole document.

Throughout this document, the term “connected to” may be used to designate a connection or coupling of one element to another element and includes both an element being “directly connected” another element and an element being “electronically connected” to another element via another element. Further, it is to be understood that the term “comprises or includes” and/or “comprising or including” used in the document means that one or more other components, steps, operation and/or the existence or addition of elements are not excluded from the described components, steps, operation and/or elements unless context dictates otherwise; and is not intended to preclude the possibility that one or more other features, numbers, steps, operations, components, parts, or combinations thereof may exist or may be added.

Throughout this document, the term “unit” includes a unit implemented by hardware and/or a unit implemented by software. As examples only, one unit may be implemented by two or more pieces of hardware or two or more units may be implemented by one piece of hardware. However, the “unit” is not limited to the software or the hardware and may be stored in an addressable storage medium or may be configured to implement one or more processors. Accordingly, the “unit” may include, for example, software, object-oriented software, classes, tasks, processes, functions, attributes, procedures, sub-routines, segments of program codes, drivers, firmware, micro codes, circuits, data, database, data structures, tables, arrays, variables and the like. The components and functions provided by the “units” can be combined with each other or can be divided up into additional components. Further, the components and the “units” may be configured to implement one or more CPUs in a device or a secure multimedia card.

In the present specification, some of operations or functions described as being performed by a device may be performed by a server connected to the device. Likewise, some of operations or functions described as being performed by a server may be performed by a device connected to the server.

Hereinafter, embodiments of the present disclosure will be explained in detail with reference to the accompanying drawings.

FIG.1is a block diagram of a radar apparatus10according to an embodiment of the present disclosure.

Referring toFIG.1, the radar apparatus10may include a transceiver100, a signal processing unit110, a fusion data generation unit120, a classification unit130, a training unit140and an autonomous driving unit150. However, the radar apparatus10illustrated inFIG.1is just one of embodiments of the present disclosure and can be modified in various ways based on the components illustrated inFIG.1.

The transceiver100may transmit a radar signal to the outside of a vehicle and receive a radar signal reflected from an object. For example, the object may include an obstacle, a moving body, a pedestrian, etc. located in front of, beside and behind the vehicle.

The signal processing unit110may process the reflected radar signal to detect the object. Here, the reflected radar signal is received while the vehicle equipped with the radar apparatus10is driving.

Specifically, the signal processing unit110may perform a signal processing to the reflected radar signal to detect the object. For example, referring toFIG.2A, the signal processing unit110may extract a plurality of point cloud data201(small boxes) constituting at least one object based on the reflected radar signal, and derive a detection result value of the object by using the extracted plurality of point cloud data201. Here, the detection result value of the object may include location information and speed information of the object and angle information between the object and the vehicle (vehicle equipped with the radar apparatus10) derived based on the plurality of point cloud data201.

The signal processing unit110may recognize the object based on the detection result value of the object. The signal processing unit110may determine whether or not the detected object is a real object based on the detection result value of the object, and derive a recognition result value of the object based on the determination. For example, referring toFIG.2A, the signal processing unit110may cluster the plurality of point cloud data201, estimate a clustered point cloud set203(large square box) as a real object and regard the clustered point cloud set203as a recognition result value of the object. Further, the signal processing unit110may track the clustered point cloud set and derive the tracking result as a recognition result value of the object. Furthermore, the signal processing unit110may correct a moving speed of the real object corresponding to the clustered point cloud set depending on whether the vehicle is moving and derive the corrected result as a recognition result value of the object.

The signal processing unit110may generate radar data based on a fast Fourier transform value of the reflected radar signal, the detection result value of the object and the recognition result value of the object.

An image processing unit (not shown) may detect and classify the object based on image data received from a camera installed in the vehicle. For example, referring toFIG.2B, the image processing unit (not shown) may input the image data into an artificial intelligence camera module20and then derive camera data including a detection result value (for example, location information of the object, etc.) and a classification result value (for example, vehicle type information of the object) derived by the artificial intelligence camera module20. Here, the image processing unit (not shown) may recognize the object from the image data through the artificial intelligence camera module20and set a bounding box for the recognized object to search for the location of the recognized object. Further, the image processing unit (not shown) may determine which of a plurality of categories (for example, truck, bus, sedan, etc.) the object (for example, vehicle) recognized through the artificial intelligence camera module20belongs to and then classify the recognized object into the corresponding category.

The fusion data generation unit120may generate fusion data based on the radar data and the camera data.

Herein, the radar data may include data derived from a radar signal, such as a fast Fourier transform value of the radar signal reflected from the object, a detection result value of the object, a recognition result value of the object and the like. Herein, the camera data may be derived through the artificial intelligence camera module and may include a detection result value of an object included in image data generated by the camera installed in the vehicle and a classification result value of the object.

The fusion data generation unit120may project the camera data to a radar coordinate system of the radar apparatus10and match the camera data and the radar data projected to the radar coordinate system for each target to generate fusion data. For example, referring toFIG.2C, the fusion data generation unit120may transform the location information of the object included in the camera data into a radar coordinate system of the radar apparatus10installed in a vehicle205(for example, an XYZ coordinate system around the vehicle205) and check whether the coordinates of the object included in the camera data transformed into the radar coordinate system are similar to the coordinates of the object included in the radar data. In this case, the transformation into the radar coordinate system may be performed to compensate for a location error if there is a location error between the camera data and the radar data. Then, if the coordinates of the object included in the camera data transformed into the radar coordinate system are similar to the coordinates of the object included in the radar data, the fusion data generation unit120may recognize the object included in the camera data and the object included in the radar data as the same one and match each other.

Meanwhile, if the location error between the camera data and the radar data is less than a threshold value, the fusion data generation unit120does not project the camera data to the radar coordinate system of the radar apparatus10, but may match the camera data and the radar data for each target to generate fusion data.

The fusion data may be used to obtain a classification result with high reliability through mutual matching between the camera data and the radar data and cumulative statistics over time.

The advantage of the radar apparatus10is that it has high range and velocity accuracy and can quickly derive a detection result. The advantage of the camera is that it has high horizontal resolution and can distinguish the types of vehicles (for example, truck, bus, sedan, motorcycle, etc.). According to the present disclosure, it is possible to generate fusion data with high accuracy for an object detected by the camera and the radar apparatus10by using the advantages of the camera and the radar apparatus10and complementing the disadvantages thereof.

FIG.3is a diagram for explaining a process for generating fusion data. Referring toFIG.3, the signal processing unit110may generate radar data based on a radar signal reflected from an object. The image processing unit (not shown) may generate camera data based on image data through the artificial intelligence camera module20. Here, the camera data may be coordinate-transformed into a radar coordinate system so as to be matched with the radar data for each target. The fusion data generation unit120may generate fusion data using the radar data and the camera data transformed into the radar coordinate system.

Meanwhile, the fusion data generation unit120may generate fusion data by analyzing driving conditions of a vehicle while the vehicle is driving and giving weightings to the camera data and the radar data based on the analyzed driving conditions. Here, the driving conditions refers to external environmental factors that affect the vehicle while the vehicle is driving and may include various types of driving condition information (for example, rainy weather, foggy weather, night driving, etc.). For example, referring toFIG.5, if it is rainy (or foggy) when the vehicle is driving, the accuracy in detecting an object (for example, vehicle) using camera data50decreases. Therefore, in this case, the fusion data generation unit120may generate fusion data by giving a higher weighting to radar data52than to the camera data50. Here, the fusion data generated by giving a higher weighting to the radar data52may be used for additional training by the training unit140, and location information of the object may be estimated from the camera data50based on location information of the object included in the radar data52.

Meanwhile, the training unit140may input radar data into an artificial intelligence radar module of the radar apparatus10to train the artificial intelligence radar module, and input image data into an artificial intelligence camera module of the camera to train the artificial intelligence camera module. Herein, the radar data may be generated while the vehicle is driving based on a reflected radar signal received while the vehicle is driving, and the image data may be generated while the vehicle equipped with the camera is driving.

The training unit140may input the radar data into the artificial intelligence radar module and train the artificial intelligence radar module so that the artificial intelligence radar module can perform a detection process and a classification process with respect to an object.

The training unit140may input the image data into the artificial intelligence camera module and train the artificial intelligence camera module so that the artificial intelligence camera module can derive camera data including a detection result value of the object and a classification result value of the object.

The radar apparatus and the camera installed in the vehicle may receive a radar signal while the vehicle is driving and generate image data in real time. Therefore, the camera data and the radar data used in the training unit140may be generated in real time based on the radar signal and the image data generated in real time. Further, the camera data and the radar data generated in real time are used in real time for basic training of the artificial intelligence modules and may be used in real time for additional training of the artificial intelligence modules.

When the fusion data are generated by the fusion data generation unit120, the training unit140may additionally input the fusion data into the artificial intelligence radar module and the artificial intelligence camera module to additionally train the artificial intelligence radar module and the artificial intelligence camera module.

An object detection result obtained by the radar apparatus10is more stable than an object detection result obtained by the camera in bad weather or at night time. Also, the radar apparatus10is capable of limiting and classifying an object, which is detected as moving on a road at a predetermined speed or more based on ground speed information, into a specific class. Therefore, it is possible to use the radar data for training of the artificial intelligence camera module in bad weather or at night time.

The fusion data used for training may include, for example, vertical and horizontal location information of the detected object and object type information (for example, vehicle, truck, motorcycle, bicycle, pedestrian, etc.) classified by the previously trained artificial intelligence camera module.

For example, referring toFIG.4, the training unit140may input the fusion data and radar data (radar data generated based on a reflected radar signal received in real time) into the artificial intelligence radar module22and additionally train the artificial intelligence radar module22so that the artificial intelligence radar module22can derive a classification result value of the object. Further, the training unit140may input the fusion data and real-time image data into the artificial intelligence camera module20and additionally train the artificial intelligence camera module so that the artificial intelligence camera module20can derive camera data including a detection result value of the object and a classification result value of the object.

Then, when the real-time image data are input into the additionally trained artificial intelligence camera module, real-time camera data may be derived by the artificial intelligence camera module.

That is, according to the present disclosure, it is possible to primarily train the artificial intelligence modules based on radar data and camera data serving as basic data. This is basic training for the artificial intelligence modules to have minimum performance. Further, according to the present disclosure, it is possible to secondarily train the artificial intelligence modules based on fusion data generated to complement the disadvantages of the radar data and the camera data, and, thus, it is possible to improve performance of the artificial intelligence modules. Particularly, according to the present disclosure, it is possible to effectively train the artificial intelligence radar module by suggesting a fusion data-based classification training method for radar data which it is difficult to annotate.

The classification unit130may classify the object detected by the radar apparatus10through the artificial intelligence modules trained based on the generated fusion data. Specifically, the classification unit130may classify the object detected by the radar apparatus10through the additionally trained artificial intelligence radar module.

Meanwhile, a performance measurement unit (not shown) may measure performance of the additionally trained artificial intelligence radar module based on a classification result value of the object derived through the additionally trained artificial intelligence radar module.

When a performance value of the artificial intelligence radar module exceeds a predetermined threshold value, the fusion data generation unit120may generate fusion data further based on the classification result value of the object derived through the artificial intelligence radar module.

For example, referring toFIG.6, when the performance value of the artificial intelligence radar module exceeds the predetermined threshold value, the fusion data generation unit120may generate fusion data based on the classification result value of the object derived through the artificial intelligence radar module22and the camera data derived through the artificial intelligence camera module20.

When the fusion data are generated further based on the classification result value of the object derived through the artificial intelligence radar module, the training unit140may additionally input the fusion data into the artificial intelligence radar module and the artificial intelligence camera module to additionally train the artificial intelligence radar module and the artificial intelligence camera module.

The autonomous driving unit150may perform autonomous driving of the vehicle based on the classification result value of the object derived through the artificial intelligence radar module. For example, if a vehicle equipped with the radar apparatus10drives without a camera, the vehicle may transmit and receive a radar signal through the radar apparatus10, input a radar signal reflected from an object into the artificial intelligence radar module and perform autonomous driving based on a classification result value of the object derived through the artificial intelligence radar module.

Meanwhile, it would be understood by a person with ordinary skill in the art that each of the transceiver100, the signal processing unit110, the fusion data generation unit120, the classification unit130, the training unit140and the autonomous driving unit150can be implemented separately or in combination with one another.

FIG.7is a flowchart showing a method for classifying an object by the radar apparatus10installed in a vehicle according to an embodiment of the present disclosure.

Referring toFIG.7, in a process5701, the radar apparatus10may transmit a radar signal to the outside of a vehicle.

In a process5703, the radar apparatus10may receive a radar signal reflected from an object.

In a process5705, the radar apparatus10may process the reflected radar signal to detect the object.

In a process5707, the radar apparatus10may generate fusion data based on radar data and camera data.

In a process5709, the radar apparatus10may classify the detected object through artificial intelligence modules trained based on the generated fusion data.

In the descriptions above, the processes5701to5709may be divided into additional processes or combined into fewer processes depending on an embodiment. In addition, some of the processes may be omitted and the sequence of the processes may be changed if necessary.

A computer-readable medium can be any usable medium which can be accessed by the computer and includes all volatile/non-volatile and removable/non-removable media. Further, the computer-readable medium may include all computer storage and communication media. The computer storage medium includes all volatile/non-volatile and removable/non-removable media embodied by a certain method or technology for storing information such as computer-readable instruction code, a data structure, a program module or other data. The communication medium typically includes the computer-readable instruction code, the data structure, the program module, or other data of a modulated data signal such as a carrier wave, or other transmission mechanism, and includes a certain information transmission medium.

The above description of the present disclosure is provided for the purpose of illustration, and it would be understood by those skilled in the art that various changes and modifications may be made without changing technical conception and essential features of the present disclosure. Thus, it is clear that the above-described embodiments are illustrative in all aspects and do not limit the present disclosure. For example, each component described to be of a single type can be implemented in a distributed manner. Likewise, components described to be distributed can be implemented in a combined manner.

The scope of the present disclosure is defined by the following claims rather than by the detailed description of the embodiment. It shall be understood that all modifications and embodiments conceived from the meaning and scope of the claims and their equivalents are included in the scope of the present disclosure.