Patent Publication Number: US-2023161044-A1

Title: Apparatus for generating real-time lidar data in a virtual environment and method for controlling the same

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of International Patent Application No. PCT/KR2022/005161 filed on Apr. 8, 2022, which is based upon and claims the benefit of priority to Korean Patent Application No. 10-2021-0162520, filed on Nov. 23, 2021. The disclosures of the above-listed applications are hereby incorporated by reference herein in their entirety. 
    
    
     TECHNICAL FIELD 
     The present disclosure relates to an apparatus for generating virtual LiDAR data and a method for controlling the same, and more specifically, to an apparatus for generating virtual LiDAR data according to movements of a virtual moving means driving in a virtual environment, and a method for controlling the same. 
     BACKGROUND 
     In general, while a vehicle is driving, an autonomous driving system can automatically control the driving of the vehicle from a starting point to an ending point on a road using road map information, GPS location information, and signals acquired from various sensors. 
     In order to respond to various unexpected situations, the autonomous driving system measures in real time the surrounding objects while driving, and if a variable occurs, adjusts the vehicle control according to the variable. That is, a separate change is made to the control to avoid the risk factor. 
     To this end, the autonomous driving system is configured to perform autonomous driving of the vehicle according to the flow of operations including recognition, determination, and control, and for the recognition operation, recognize the driving environment such as the presence of other vehicles, pedestrians, obstacles, and the like on the road using sensors such as RADAR, LiDAR, camera, and the like mounted on the vehicle. For the determination operation, the autonomous driving system is configured to infer the driving condition based on the data and map information measured in the recognition operation, and for the control operation, cause the actual manipulation to be performed by generating control signals for the components of the vehicle based on the values calculated and inferred in the determination operation. 
     For these operations, the autonomous vehicles are developed in such a way that achieves more appropriate control by recognizing and determining more information in detail. To this end, in the related art, the algorithms of each stage have been developed so as to derive accurate determinations from many variables. 
     Meanwhile, various driving apparatuses and systems loaded with the algorithms developed as described above may be mounted to the vehicle for testing of the developed algorithms, but most of the apparatuses and systems mounted to the vehicle are expensive, and there is a problem of space limitation for testing on actual road. 
     Accordingly, it is necessary to develop a system capable of precisely testing an autonomous vehicle in a virtual environment. 
     SUMMARY 
     An object of the present disclosure is to provide a virtual autonomous vehicle driving in a virtual environment with a test environment that is close to the actual driving environment. 
     Another object of the present disclosure is to provide a test environment in which it is possible to process LiDAR data acquired from driving in a real environment to generate virtual LiDAR data in a virtual environment. 
     The technical problems to be addressed in the present disclosure are not limited to those described above, and other technical problems that are not mentioned herein will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description. 
     An apparatus for generating virtual LiDAR data is provided, which may include a LiDAR data collection unit installed in a moving means driving in a real environment to collect LiDAR data, a database configured to store the collected LiDAR data, a virtual environment generation engine configured to generate a virtual environment, and a LiDAR data generation unit configured to generate virtual LiDAR data corresponding to a movement of a virtual moving means in the generated virtual environment. 
     The LiDAR data collection unit may include a laser light emitting unit and a laser light receiving unit, and the collected LiDAR data may include intensity information of the laser that is emitted through the laser light emitting unit and reflected off from an object. 
     The intensity information may be stored in association with corresponding coordinate information of a point where the laser is reflected. 
     The apparatus for generating virtual LiDAR data is provided, which may further include a point cloud map production unit configured to produce a point cloud map based on the collected LiDAR data. 
     The point cloud map may be data in which the collected LiDAR data is stored in association with corresponding coordinates on the generated virtual environment. 
     The apparatus for generating virtual LiDAR data may further include a mesh conversion unit configured to convert the point cloud map into a 3D mesh map. 
     The LiDAR data generation unit may be configured to emit a virtual laser to a predetermined mesh of the converted 3D mesh, and calculate a virtual intensity of a laser reflected by the emitted virtual laser and generate the virtual LiDAR data. 
     The virtual intensity may be calculated based on intensity information of a plurality of coordinates forming the predetermined mesh. 
     The virtual intensity may be calculated as an average of the intensity information for the plurality of coordinates. 
     According to another aspect of the present disclosure, there is provided a method for controlling an apparatus for generating virtual LiDAR data in a virtual environment, in which the method may include by a LiDAR data collection unit installed in a moving means driving in a real environment, collecting LiDAR data, by a database, storing the collected LiDAR data, by a virtual environment generation engine, generating a virtual environment, and by a LiDAR data generation unit, generating virtual LiDAR data corresponding to a movement of a virtual moving means on the generated virtual environment. 
     The LiDAR data collection unit may include a laser light emitting unit and a laser light receiving unit, and the collected LiDAR data may include intensity information of the laser that is emitted through the laser light emitting unit and reflected off from an object. 
     The intensity information may be stored in association with corresponding coordinate information of a point where the laser is reflected. 
     The method may further include, by a point cloud map production unit, producing the point cloud map based on the collected LiDAR data. 
     The point cloud map may be data in which the collected LiDAR data is stored in association with corresponding coordinates on the generated virtual environment. 
     The method may further include, by a mesh conversion unit, converting the point cloud map into a 3D mesh map. 
     The generating the virtual LiDAR data may include by the LiDAR data generation unit, emitting a virtual laser to a predetermined mesh of the converted 3D mesh, and calculating a virtual intensity of a laser reflected by the emitted virtual laser and generating the virtual LiDAR data. 
     The virtual intensity may be calculated based on intensity information of a plurality of coordinates forming the predetermined mesh. 
     The virtual intensity may be calculated as an average of the intensity information for the plurality of coordinates. 
     The effects of the apparatus for generating virtual LiDAR data and a method for controlling the same according to the present disclosure will be described below. 
     According to at least one of the examples of the present disclosure, there is an advantage in that a test environment close to actual driving environment can be provided in a virtual environment. 
     In addition, according to at least one of the examples of the present disclosure, there is an advantage in that virtual LiDAR data for testing an autonomous vehicle in a virtual environment can be generated. 
     Further scope of applicability of the present disclosure will be apparent from the following detailed description. However, it should be understood that the detailed description and specific examples such as preferred examples of the present disclosure are given by way of illustration only, since various changes and modifications within the spirit and scope of the present disclosure may be clearly understood by those skilled in the art. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a diagram illustrating a conceptual view of an apparatus  100  for generating LiDAR data. 
         FIG.  2    is a diagram illustrating dimensions of a point. 
         FIG.  3    is a flowchart illustrating a method for controlling the apparatus  100  for generating virtual LiDAR data. 
         FIG.  4    is a diagram illustrating a conceptual view of a 3D mesh map conversion. 
         FIG.  5    illustrates an example of converting a point cloud map into a 3D mesh map. 
         FIG.  6    illustrates a conceptual view of determining an intensity of a mesh. 
         FIG.  7    illustrates a screen applying the intensity obtained based on LiDAR data generated according to an example of the present disclosure. 
         FIG.  8    is a diagram schematically illustrating LiDAR data generated according to an example of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     Hereinafter, certain examples will be described herein in detail with reference to the accompanying drawings, but the same or similar components will be assigned the same reference numbers irrespective of the reference numerals in the drawings, and redundant description thereof will be omitted. The term “module” and “unit” used in the names of the components as used in the description are given only in consideration of the ease of writing the description, and not intended to have distinct meanings or roles by themselves. In addition, in describing certain examples, if it is determined that detailed description of related known technologies may obscure the gist of the examples disclosed herein, the detailed description thereof will be omitted. In addition, it should be understood that the accompanying drawings are only for the purpose of facilitating understanding of the examples disclosed in description, and the technical idea disclosed in the description is not limited by the accompanying drawings and includes all modifications, equivalents and substitutes included in the spirit and scope of the present disclosure. 
     Further, the term “module” or “unit” used herein refers to a software or hardware component, and “module” or “unit” performs certain roles. However, the meaning of the “module” or “unit” is not limited to software or hardware. The “module” or “unit” may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Accordingly, as an example, the “module” or “unit” may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, program code segments of program code, drivers, firmware, micro-codes, circuits, data, database, data structures, tables, arrays, and variables. Furthermore, functions provided in the components and the “modules” or “units” may be combined into a smaller number of components and “modules” or “units”, or further divided into additional components and “modules” or “units.” 
     The “module” or “unit” may be implemented as a processor and a memory. The “processor” should be interpreted broadly to encompass a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and so forth. Under some circumstances, the “processor” may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and so on. The “processor” may refer to a combination of processing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other combination of such configurations. In addition, the “memory” should be interpreted broadly to encompass any electronic component that is capable of storing electronic information. The “memory” may refer to various types of processor-readable media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and so on. The memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated with the processor is in electronic communication with the processor. 
     In the present disclosure, a “system” may refer to at least one of a server device and a cloud device, but not limited thereto. For example, the system may include one or more server devices. In another example, the system may include one or more cloud devices. In still another example, the system may include both the server device and the cloud device operated in conjunction with each other. 
     The expressions including ordinal numbers such as “first,” “second,” and so on may be used for describing a variety of elements, although these elements are not limited by those expressions. The expressions are used only for the purpose of distinguishing one element from another. 
     It should be understood that when an element is referred to as being “connected” or “coupled” to another element, it may be directly connected or coupled to another element, with or without yet another element being present in between. On the other hand, it should be understood that when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no other element present in between. 
     Unless otherwise specified, a singular expression includes a plural expression. 
     The term “comprise” or “have” as used herein is intended to designate an existence of features, numbers, steps, operations, elements, components or a combination of these, and accordingly, this should not be understood as precluding an existence or a possibility of adding one or more of other features, numbers, steps, operations, elements, components or a combination of these. 
     The LiDAR data mainly includes a distance and an intensity. In this case, the distance refers to a distance to the surface of an object that the laser emitted from the center of the LiDAR first hits, and the intensity (reflectance) refers to an intensity of the laser that is reflected off from the surface of the object and returns to a light receiving unit of the LiDAR. 
     At this time, the intensity may be basically determined by the color and material of the surface of the reflection point, and the intensity of the laser emitted from the light emitting unit of the LiDAR also depends on the manufacturer. 
     In general, the brighter the color of the reflection point, the higher the intensity, and the darker the color, the lower the intensity. In addition, the smoother the surface of the reflection point, the better the reflection and the higher the intensity, and the rougher the surface of the reflection point, the lower the intensity. 
     For example, the intensity is relatively high for the lane portion of the road that is relatively bright and has a relatively smooth surface because of the paint applied, while for the asphalt part with a rather rough surface and dark color, the intensity is low and it tends to be dark. 
     Accordingly, it is not easy to virtually generate LiDAR data that is affected by various environment variables. That is, a large amount of computation is required in order to reflect all the various environmental variables, and it is not suitable to generate virtual LiDAR data in real time. 
     Accordingly, the present disclosure proposes a method for effectively processing LiDAR data obtained through a moving means in a real environment and providing the processed result as virtual LiDAR data in a virtual environment for autonomous driving simulation. 
       FIG.  1    is a diagram illustrating a conceptual view of an apparatus  100  for generating LiDAR data. 
     The apparatus  100  for generating LiDAR data is configured to include at least one of a LiDAR data collection unit  101  (e.g., including a LiDAR data receiver), a virtual environment generation engine  102 , a database  103 , a LiDAR data generation unit  104 , a sensor data generation unit  105 , a point cloud map production unit  106 , and a mesh map conversion unit  107 . The apparatus  100  may include at least one processor and memory storing instructions that, when executed by the at least one processor, to cause the apparatus  100  to perform one or more operations described herein (e.g., the one or more operations performed by the LiDAR data collection unit  101 , the virtual environment generation engine  102 , the database  103 , the LiDAR data generation unit  104 , the sensor data generation unit  105 , the point cloud map production unit  106 , and/or the mesh map conversion unit  107 ). 
     The components illustrated in  FIG.  1    are not essentially required for implementing the apparatus  100  for generating LiDAR data, and accordingly, the apparatus  100  for generating LiDAR data described herein may have more or fewer components than those listed above. 
     The LiDAR data collection unit  101  is configured to be installed in the moving means  190  driving in the real environment to collect LiDAR data. The LiDAR data collection unit  101  may include a separate storage and store the collected LiDAR data. Alternatively, a wired or wireless communication unit may be provided and the collected data may be provided to the database  103  through wired or wireless communication. 
     The LiDAR data collection unit  101  may be the same as the LiDAR device installed in the actual autonomous vehicle. That is, an aspect of the present disclosure is to utilize the LiDAR data collected through a general LiDAR device and use the same in a virtual environment. 
     The database  103  is a configuration for storing the collected LiDAR data. As described below, the LiDAR data may be processed for use in a virtual environment, and the database  130  may store both the LiDAR data before and after processing. 
     The database  103  may further store at least a piece of environment information for generating a virtual environment. The environment information may include at least one of simulated road information, surrounding vehicle information, surrounding pedestrian information, surrounding obstacle information, surrounding traffic light information, surrounding road signage information, event information, road information, surrounding vehicle information, surrounding pedestrian information, surrounding obstacle information, surrounding traffic light information, surrounding road signage information, and event information. 
     The virtual environment generation engine  102  generates a virtual environment for autonomous driving simulation. 
     The virtual environment generation engine  102  is equipped with a physical engine-based simulator to enable verification of algorithms for the autonomous vehicle. In addition, data such as topographic features, road types, pedestrians, autonomous vehicles, weather, and the like may be implemented in the generated virtual environment, and event information data such as rockfalls, traffic accidents, and the like may also be provided such that it can occur. 
     In addition, a plurality of different vehicle data may be implemented on the virtual environment, and driving routes of some other vehicles may be changed according to the surrounding environment according to an input driving algorithm. In addition, some other vehicles may be configured to verify the driving algorithm according to a predetermined driving route. 
     The virtual environment generation engine  102  may further include an output device connected to the simulator, and the data described above may be visualized on the output device such that the user can check the data as an image. 
     The LiDAR data generation unit  104  generates virtual LiDAR data corresponding to the movement of the virtual moving means in the generated virtual environment. In other words, the LiDAR data generation unit  104  generates virtual LiDAR data that is close to the data of the autonomous vehicle moving in a real environment. 
     The sensor data generation unit  105  generates virtual sensor data by a virtual sensor provided in the virtual moving means on the generated virtual environment. The sensor unit may include at least one of a LiDAR, a RADAR, a GPS, and a camera of the autonomous vehicle simulated in the virtual environment. 
     The point cloud map production unit  106  generates a point cloud map based on the collected LiDAR data. The point cloud map refers to data in which the collected LiDAR data is stored in association with corresponding coordinates on the generated virtual environment. 
     The mesh map conversion unit  107  converts the point cloud map into a 3D mesh map. By connecting a plurality of points forming the point cloud map, a 3D mesh map formed of a plurality of meshes is formed. 
     The point cloud map production unit  106  records, based on the collected LiDAR data, the time for a laser to be sent to the object and return, and calculates the distance to the reflection point and generates a point for the reflection point. The generated point is formed of coordinate (position) information formed of 3D coordinates in a 3D space. The arrangement of a set of points generated as described above on the 3D space of the virtual environment is referred to as a point cloud map. 
     It is proposed such that each point forming the point cloud map does not simply include only coordinate information, but further includes an intensity at the corresponding coordinate. 
       FIG.  2    is a diagram illustrating dimensions of a point. 
       FIG.  2 A  illustrates the coordinate information of the existing point, and  FIG.  2 B  illustrates the information of the point added with the intensity information according to an example of the present disclosure. 
     While the existing point includes only 3D coordinate information, an example of the present disclosure proposes to add information on intensity such that each point has 4D coordinate information. 
     Hereinafter, a method for controlling the apparatus  100  for generating virtual LiDAR data will be described with reference to  FIG.  3   . 
       FIG.  3    is a flowchart illustrating a method for controlling the apparatus  100  for generating virtual LiDAR data. 
     At S 301 , the LiDAR data collection unit  101  collects LiDAR data while driving in a real environment. When collecting LiDAR data, it is apparent that GPS location information can be collected together. 
     The LiDAR data collection unit  101  may include a laser light emitting unit and a laser light receiving unit, and the collected LiDAR data may include intensity information of the laser that is emitted through the laser light emitting unit and reflected off from an object. 
     At S 302 , the point cloud map production unit  106  may produce a point cloud map based on the collected LiDAR data. 
     At S 303 , the mesh map conversion unit  107  converts the point cloud map into a 3D mesh map. 
       FIG.  4    is a diagram illustrating a conceptual view of a 3D mesh map conversion. 
     Referring to  FIG.  4 A , a plurality of points in the point cloud map are illustrated. In addition, as described above, each point stores 4D information that includes the 3D coordinate information together with the intensity. As illustrated in  FIG.  4 B , the mesh map conversion unit  107  may generate a 3D mesh map with a method of repeatedly generating triangular meshes  401 - 2  to  402 - 5  having three adjacent points as one plane. 
       FIG.  5    illustrates an example of converting a point cloud map into a 3D mesh map. 
     Referring to the drawing, the points that formed the point cloud map are now forming a 3D mesh map through a combination of triangular meshes shared with each other. 
     Returning to  FIG.  3   , at S 304 , the virtual environment generation engine  102  generates a virtual environment for the autonomous driving simulation. At S 305 , the virtual environment generation engine  102  performs virtual driving of the autonomous vehicle on the generated virtual environment. 
     The virtual environment generated at S 304  may be mapped with the 3D mesh map illustrated in  FIG.  5   . 
     At S 306 , the LiDAR data generation unit  104  generates virtual LiDAR data that is generated as the autonomous vehicle is driving in the virtual environment. The virtual LiDAR data generated in this way may be used to verify or test the algorithm for the vehicle, at S 307 . 
     To this end, the LiDAR data generation unit  104  emits a virtual laser onto the 3D mesh map in the virtual environment. In addition, if the emitted virtual laser is reflected off from a predetermined mesh, the intensity of the corresponding mesh is read and assumed to be the intensity of the reflected laser, and the LiDAR data is generated accordingly. 
     That is, the LiDAR data is generated by directly setting the intensity of the mesh to the intensity of the reflected laser, without considering the surface properties or color of the reflection point in the real virtual environment. 
     Hereinafter, a method for determining the intensity of the mesh will be described with reference to  FIG.  6   . 
       FIG.  6    illustrates a conceptual view of determining an intensity of a mesh. 
     As described above, a mesh  402  refers to a triangle having three points  401 - 1  to  401 - 3  as one plane. The three points are positioned at each vertex of the triangle. 
     Each of the first to third points  401 - 1  to  401 - 3  includes 4D information. As described above, the 4D information is information including the 3D position information (x, y, z) together with the intensity (i). 
     It is proposed to calculate the intensity of the mesh  402  based on the intensities i 1  to i 3  of the first to third points  401 - 1  to  401 - 3  in the mesh. 
     In order to convert the point cloud map into a 3D mesh map, the mesh map conversion unit  107  may set the average of the intensities i 1  to i 3  of the first to third points  401 - 1  to  401 - 3  as an intensity mesh (i mesh ) of the mesh  402 . When represented by a mathematical expression, the intensity mesh (i mesh ) may be represented as Equation 1 below. 
     
       
         
           
             
               
                 
                   
                     i 
                     mesh 
                   
                   = 
                   
                     
                       
                         i 
                         1 
                       
                       + 
                       
                         i 
                         2 
                       
                       + 
                       
                         i 
                         3 
                       
                     
                     3 
                   
                 
               
               
                 
                   〈 
                   
                     Equation 
                     ⁢ 
                         
                     1 
                   
                   〉 
                 
               
             
           
         
       
     
     Meanwhile, the present disclosure is not limited to the method of Equation 1 above, and may include all methods of using at least one of the first to third points  401 - 1  to  401 - 3 . For example, a maximum or minimum value may be used for the intensities i 1  to i 3  of the first to third points  401 - 1  to  401 - 3 , or a geometric mean rather than an arithmetic mean may be used. 
     That is, each mesh converted by the mesh map conversion unit  107  will have its own preset intensity. The LiDAR data generation unit  104  may emit a virtual laser onto a 3D mesh map in a virtual environment, and generate LiDAR data with the intensity that is set for the mesh at a point where the intensity of the reflected virtual laser is reflected. 
       FIG.  7    illustrates a screen applying the intensity obtained based on LiDAR data generated according to an example of the present disclosure.  FIG.  8    is a diagram schematically illustrating LiDAR data generated according to an example of the present disclosure. 
     Referring to the screens illustrated in  FIGS.  7  and  8   , it can be seen that both the intensity of the reflected laser by the paint applied on the crosswalk and the intensity of the laser reflected by the road signage are effectively reflected. 
     The aspects of the virtual LiDAR data generating apparatus and the method for controlling the same according to the present disclosure have been described above, but these are described as at least one example, and the technical spirit of the present disclosure and configuration and operation thereof are not limited by this example, and the scope of the technical spirit of the present disclosure is not restricted/limited by the drawings or the description with reference to the drawings. In addition, the concepts and examples of the present disclosure presented in the present disclosure can be used by those of ordinary skill in the art as a basis for modifying or designing other structures in order to perform the same purpose of the present disclosure, and the modified or changed equivalent structures by those of ordinary skill in the art to which the present disclosure pertains belong to the technical scope of the present disclosure described in the claims, such that various changes, substitutions and changes are possible without departing from the spirit or scope of the disclosure described in the claims.