Patent Description:
With the fourth industrial revolution, interest in technical fields such as autonomous vehicles, drones, robots, etc. is increasing. In order for autonomous vehicles, drones, robots, etc. to stably and correctly operate, it is important to collect data required for controlling operations. In regard of this, studies into methods for utilizing various kinds of sensors have been conducted.

<CIT> discloses methods for detecting and classifying objects proximate to an autonomous vehicle including a sensor system and a vehicle computing system. The sensor system includes at least one LIDAR system configured to transmit ranging signals relative to the autonomous vehicle and to generate LIDAR data.

<CIT> discloses performing segmentation on voxels representing three-dimensional data to identify static and dynamic objects.

There are disclosed a vehicle, and a sensing device utilizing spatial information acquired using a sensor.

According to a first aspect, a vehicle includes: a sensor unit configured to successively sense a three-dimensional (3D) space by using at least one sensor; a memory storing a computer executable instruction; and a processor configured to execute the computer executable instruction to acquire spatial information over time for the sensed 3D space, identify at least one object in the sensed 3D space by applying a neural network based object classification model to the acquired spatial information over time, track the sensed 3D space including the identified at least one object, and control driving of the vehicle based on information related to the tracked 3D space and information related to a movement and position of the vehicle. The processor is further configured to distinguish an object area from a ground area based on the acquired spatial information over time, cluster the object area to distinguish individual object areas, and track the sensed 3D space based on object information of the at least one object identified by applying the neural network based object classification model to the acquired spatial information over time and the distinguished individual object areas.

According to a second aspect, a sensing device includes: a sensor unit configured to sense a 3Dimensional (3D) space successively by using at least one sensor; a communication interface; a memory storing an computer executable instruction; a processor configured to execute the computer executable instruction to acquire spatial information over time for the sensed 3D space, identify at least one object in the sensed 3D space by applying a neural network based object classification model to the acquired spatial information over time, track the sensed 3D space including the identified at least one object, and transmit information related to the tracked 3D space to outside through the communication interface. The processor is further configured to distinguish an object area from a ground area based on the acquired spatial information over time, cluster the object area to distinguish individual object areas, and track the sensed 3D space based on object information of the at least one object identified by applying the neural network based object classification model based to the acquired spatial information over time and the distinguished individual object areas.

According to one or more embodiments of the disclosure, a vehicle includes the features of independent claim <NUM>.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. For more clearly describing the characteristics of the embodiments, detailed descriptions about contents well-known to one of ordinary skill in a technical field to which the following embodiments belong will be omitted.

The embodiments relate to a vehicle, a sensing device utilizing spatial information acquired using a sensor, and a server, and detailed descriptions about contents well-known to one of ordinary skill in a technical field to which the following embodiments belong will be omitted.

<FIG> shows an arbitrary driving environment wherein a vehicle <NUM> and a sensing device <NUM> according to an embodiment are located.

Referring to <FIG>, the vehicle <NUM> stops at an intersection for waiting for the light, and a plurality of sensing devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> (also referred to as the sensing device <NUM>) are located around corners of the intersection.

The vehicle <NUM> may be driving means, such as a car or a train, traveling on a road or tracks. However, when the vehicle <NUM> flies in the air not on a road or tracks, the meaning of the vehicle <NUM> may extend to flying means, such as a drone, a plane, etc., or when the vehicle <NUM> sails the seas, the meaning of the vehicle <NUM> may extend to vessel means, such as a boat, a ship, etc. Hereinafter, for convenience of description, the vehicle <NUM> is assumed to be an autonomous vehicle. The vehicle <NUM> senses, for autonomous driving, a surrounding space using a sensor to acquire spatial information.

The sensing device <NUM>, which is an apparatus capable of sensing a surrounding space to acquire spatial information, may include at least one sensor. The sensing device <NUM> may be installed on the ground or at a predetermined height from the ground. The sensing device <NUM> may be attached on or fixed at an existing facility.

The vehicle <NUM> and the sensing device <NUM> may each include at least one of various kinds of sensors, such as a Light Detection And Ranging (LiDAR) sensor, a radar sensor, a camera sensor, an infrared imaging sensor, an ultrasonic sensor, etc. The vehicle <NUM> and the sensing device <NUM> may each use a plurality of sensors of the same kind or combine and use a plurality of sensors of different kinds, in consideration of sensing ranges or obtainable data types of the individual kinds of sensors, to acquire spatial information for a 3D space.

A sensing range that may be sensed by the vehicle <NUM> may be identical to or different from a sensing range that may be sensed by the sensing device <NUM>, depending on kinds of sensors included in the vehicle <NUM> and the sensing device <NUM>. In <FIG>, sensing ranges that may be respectively sensed by the vehicle <NUM> and the sensing device <NUM> are shown. Referring to <FIG>, a first sensing range that may be sensed by the vehicle <NUM> is shown to be smaller than a second sensing range that may be sensed by the sensing device <NUM>, although not limited thereto. Although the same kind of sensors are included in the vehicle <NUM> and the sensing device <NUM>, a sensing range of the vehicle <NUM> may be different from that of the sensing device <NUM> depending on installation locations of the sensors in the vehicle <NUM> and the sensing device <NUM> or surrounding environments. For example, because the sensing device <NUM> is fixed at a higher location than the vehicle <NUM> and the moving vehicle <NUM> may meet various objects obstructing sensing of a 3D space at closer locations than the fixed sensing device <NUM>, the second sensing range that the sensing device <NUM> may sense may be larger than the first sensing range that the vehicle <NUM> may sense.

The vehicle <NUM> itself acquires spatial information for a surrounding 3D space by using a sensor for autonomous driving. To acquire spatial information for a wider space corresponding to a traveling direction in advance, the vehicle <NUM> may receive spatial information that itself may not acquire, from outside. For example, the vehicle <NUM> may receive spatial information from other vehicles or the sensing device <NUM> around the vehicle <NUM>.

Hereinafter, a method of using spatial information acquired by the vehicle <NUM> in autonomous driving, a method of transmitting spatial information acquired by the sensing device <NUM> to another surrounding device, and a method of combining spatial information acquired by the vehicle <NUM> with spatial information acquired by the sensing device <NUM> to acquire and use spatial information for a wider 3D space will be described in detail.

<FIG> is a block diagram showing a configuration of a vehicle according to an embodiment.

Referring to <FIG>, the vehicle <NUM> according to an embodiment may include a memory <NUM>, a processor <NUM>, a communication interface <NUM>, a sensor unit <NUM>, and a user interface <NUM>. It will be appreciated by one of ordinary skill in a technical field related to the embodiment that other general-purpose components in addition to those shown in <FIG> may be further included in the vehicle <NUM>.

The memory <NUM> may store software and/or a program. For example, the memory <NUM> may store a program and various kinds of data, such as an application, an application programming interface (API), etc. The memory <NUM> may store instructions that are executable by the processor <NUM>.

The processor <NUM> may access data stored in the memory <NUM> to use the data, or may store new data in the memory <NUM>. The processor <NUM> executes the instructions stored in the memory <NUM>. The processor <NUM> may execute a computer program installed in the vehicle <NUM>. Also, the processor <NUM> may install a computer program or an application received from outside in the memory <NUM>. The processor <NUM> may include at least one processing module. The processing module may be a dedicated processing module for executing a predetermined program. For example, the processor <NUM> may include various kinds of processing modules for executing a vehicle control program for autonomous driving, such as advanced driver assistance system (ADAS), or a processing module for executing a 3D space tracking program, in the form of separated, dedicated chips. The processor <NUM> may control other components included in the vehicle <NUM> to perform an operation corresponding to an instruction or an execution result of a computer program, etc..

The communication interface <NUM> may perform wireless communications with another device or a network. For this, the communication interface <NUM> may include a communication module that supports at least one of various wireless communication methods. For example, the communication interface <NUM> may include a communication module that performs short-range communication such as wireless fidelity (Wi-Fi), various kinds of mobile communication, such as <NUM>, <NUM>, and <NUM>, or ultra wideband communication. The communication interface <NUM> may be connected to a device installed outside the vehicle <NUM> to transmit and receive signals or data. The vehicle <NUM> may perform communication with the sensing device <NUM> or another vehicle through the communication interface <NUM>, or may be connected to a local server managing a region where the vehicle <NUM> is located through the communication interface <NUM>.

The sensor unit <NUM> may include at least one sensor for sensing a 3D space. The sensor unit <NUM> may sense an object located within a sensing range, and acquire data capable of generating coordinates of the sensed object on a 3D space. The sensor unit <NUM> may acquire shape data or distance data for the object located within the sensing range. The sensor unit <NUM> may include at least one among various kinds of sensors, such as a LiDAR sensor, a radar sensor, a camera sensor, an infrared imaging sensor, an ultrasonic sensor, etc. For example, the sensor unit <NUM> may include at least one 3D LiDAR sensor to acquire data for a space of a <NUM>-degree range, and further include at least one selected from the group consisting of a radar sensor and an ultrasonic sensor to acquire data for a blind spot that the 3D LiDAR sensor may not sense or data for a near space within a predetermined distance from the vehicle <NUM>.

The user interface <NUM> may receive a user input, etc. from a user. The user interface <NUM> may display information, such as an execution result of a computer program executed in the vehicle <NUM>, a processing result corresponding to a user input, a state of the vehicle <NUM>, etc. For example, a user may select a computer program that he/she wants to execute from among various kinds of computer programs installed in the vehicle <NUM>, through the user interface <NUM>, and execute the selected computer program. The user interface <NUM> may include hardware units for receiving inputs or providing outputs, and include a dedicated software module for driving the hardware units. For example, the user interface <NUM> may be a touch screen, although not limited thereto.

Although not shown in <FIG>, the vehicle <NUM> may further include components required for autonomous driving, such as global positioning system (GPS), inertial measurement units (IMU), etc. The GPS is global positioning system that calculates a current location of the vehicle <NUM> by receiving signals transmitted from a GPS satellite. The IMU is an apparatus that measures the speed, direction, gravity, and acceleration of the vehicle <NUM>. The processor <NUM> may acquire information related to a movement and position of the vehicle <NUM> by using the GPS and IMU. The processor <NUM> may acquire other information related to the control of the vehicle <NUM> from another sensor or memory included in the vehicle <NUM>.

The processor <NUM> executes a computer-executable instruction to acquire spatial information over time for a 3D space sensed successively by at least one sensor. The processor <NUM> identifies at least one object in the sensed 3D space by applying a neural network based object classification model to the acquired spatial information over time, and tracks the sensed 3D space including the identified at least object. The processor <NUM> controls driving of the vehicle <NUM>, based on information related to the tracked 3D space and the information related to the movement and position of the vehicle <NUM>. The information for the tracked 3D space may include spatial information about the space in which the identified at least one object is located and dynamic information about a movement of the identified at least one object.

The processor <NUM> may receive a time stamp in which time information is recorded and data for the sensed 3D space from the sensor unit <NUM>, and create a 3D image corresponding to the spatial information over time for the sensed 3D space. Because the spatial information over time for the sensed 3D space may have movement coordinate values corresponding to a movement location of the vehicle <NUM> through the GPS, the spatial information over time for the sensed 3D space may be mapped to a corresponding part of coordinate system relative to predetermined coordinates, for example, absolute coordinate system relative to the origin point.

The sensor unit <NUM> may sense a 3D space successively and within different sensing concentric ranges of a spherical shape through a plurality of 3D LiDAR sensors. The processor <NUM> may acquire spatial information over time for the 3D space sensed within the different sensing ranges, and assign an accuracy-related weight to an object commonly identified from the acquired spatial information over time to thereby track the 3D space.

The processor <NUM> may determine attribute information of at least one selected from the group consisting of a kind, 3D shape, location, position, size, trajectory, and speed of the at least one object identified in the sensed 3D space to track the 3D space, predict information related to the tracked 3D space, and control driving of the vehicle <NUM> further based on the predicted information.

The processor <NUM> may control driving of the vehicle <NUM> based on information related to the tracked 3D space and information related to a movement and position of the vehicle <NUM>, changing according to a movement of the vehicle <NUM>, at a processing speed of <NUM> to <NUM> and in real time.

The processor <NUM> may receive, from at least one sensing device <NUM> installed on a path along which the vehicle <NUM> travels, information related to a 3D space tracked by the sensing device <NUM> and corresponding to a fixed location of the sensing device <NUM>, through the communication interface <NUM>. The processor <NUM> may control driving of the vehicle <NUM> further based on the information related to the 3D space corresponding to the fixed location of the sensing device <NUM>.

The processor <NUM> may use the neural network based object classification model to classify the identified at least one object into any one of an object of a first type corresponding to a vehicle of a predetermined level or higher, an object of a second type corresponding to a two-wheeled vehicle or a small vehicle that is lower than the predetermined level, an object of a third type corresponding to a pedestrian, an object of a fourth type corresponding to a traveling path of the vehicle <NUM>, and an object of a fifth type corresponding to another sensed object except for the objects of the first to fourth types. The processor <NUM> may distinctively display the classified at least one object in the tracked 3D space through the user interface <NUM>.

<FIG> is a block diagram showing a configuration of the sensing device <NUM> according to an embodiment.

Referring to <FIG>, the sensing device <NUM> according to an embodiment may include a memory <NUM>, a processor <NUM>, a communication interface <NUM>, and a sensor unit <NUM>. It will be appreciated by one of ordinary skill in a technical field related to the embodiment that other general-purpose components in addition to those shown in <FIG> may be further included in the sensing device <NUM>.

The memory <NUM> may store software and/or a program. The memory <NUM> may store instructions that are executable by the processor <NUM>.

The processor <NUM> may access data stored in the memory <NUM> to use the data, or may store new data in the memory <NUM>. The processor <NUM> executes the instructions stored in the memory <NUM>. The processor <NUM> may execute a computer program installed in the sensing device <NUM>. Also, the processor <NUM> may install a computer program or an application received from outside in the memory <NUM>. The processor <NUM> may include at least one processing module. For example, the processor <NUM> may include a processing module for executing a 3D space tracking program in the form of a dedicated processing module. The processor <NUM> may control other components included in the sensing device <NUM> to perform an operation corresponding to an instruction or an execution result of a computer program, etc..

The communication interface <NUM> may perform wired or wireless communication with another device or a network. For this, the communication interface <NUM> may include a communication module that supports at least one of various wired or wireless communication methods. For example, the communication interface <NUM> may include a communication module that performs short-range communication such as Wi-Fi, wireless communication such as various kinds of mobile communication, or wired communication using a coaxial cable, an optical fiber cable, etc. The communication interface <NUM> may be connected to a device installed outside the sensing device <NUM> to transmit and receive signals or data. The sensing device <NUM> may perform communication with the vehicle <NUM> or another sensing device through the communication interface <NUM>, or may be connected to a local server managing a region where the sensing device <NUM> is located through the communication interface <NUM>.

The sensor unit <NUM> may include at least one sensor for sensing a 3D space. The sensor unit <NUM> may sense an object located within a sensing range, and acquire data capable of generating coordinates of the sensed object on a 3D space. The sensor unit <NUM> may acquire shape data or distance data for the object located within the sensing range. The sensor unit <NUM> may include at least one among various kinds of sensors, such as a LiDAR sensor, a radar sensor, a camera sensor, an infrared imaging sensor, an ultrasonic sensor, etc. For example, the sensor unit <NUM> may include at least one 3D LiDAR sensor to acquire data for a space of a <NUM>-degree range, and further include at least one selected from the group consisting of a radar sensor and an ultrasonic sensor to acquire data for a blind spot that the 3D LiDAR sensor may not sense or data for a near space within a predetermined distance from the sensing device <NUM>.

The processor <NUM> executes a computer-executable instruction to acquire spatial information over time for a 3D space sensed successively by at least one sensor. The processor <NUM> identifies at least one object in the sensed 3D space by applying a neural network based object classification model to the acquired spatial information over time, and tracks the sensed 3D space including the identified at least object. The processor <NUM> may transmit information related to the tracked 3D space to outside through the communication interface <NUM>.

The processor <NUM> may receive a time stamp in which time information is recorded and data for the sensed 3D space from the sensor unit <NUM>, and create a 3D space corresponding to the spatial information over time for the sensed 3D space. Because the spatial information over time for the sensed 3D space may have fixed coordinate values corresponding to a fixed location of the sensing device <NUM>, the spatial information over time for the sensed 3D space may be mapped to a corresponding part of coordinate system relative to predetermined coordinates, for example, absolute coordinate system relative to the origin point.

The processor <NUM> may determine attribute information of at least one selected from the group consisting of a kind, 3D shape, location, position, size, trajectory, and speed of the at least one object identified in the sensed 3D space to track the 3D space. The processor <NUM> may transmit information related to the tracked 3D space to at least one of selected from the group consisting a vehicle <NUM>, another sensing device <NUM>, and a server <NUM>, which is within a predetermined distance from the sensing device <NUM>, through the communication interface <NUM>.

<FIG> is a block diagram showing a configuration of the server <NUM> according to an unclaimed embodiment of the disclosure.

Referring to <FIG>, the server <NUM> according to an embodiment may include a memory <NUM>, a processor <NUM>, and a communication interface <NUM>. It will be appreciated by one of ordinary skill in a technical field related to the embodiment that other general-purpose components in addition to those shown in <FIG> may be further included in the server <NUM>.

The processor <NUM> may use data stored in the memory <NUM>, or may store new data in the memory <NUM>. The processor <NUM> may execute the instructions stored in the memory <NUM>. The processor <NUM> may execute a computer program installed in the server <NUM>. The processor <NUM> may include at least one processing module. The processor <NUM> may control other components included in the server <NUM> to perform an operation corresponding to an instruction or an execution result of a computer program, etc..

The communication interface <NUM> may perform wired or wireless communication with another device or a network. The communication interface <NUM> may be connected to a device located outside the server <NUM> to transmit and receive signals or data. The server <NUM> may perform communication with the vehicle <NUM> or the sensing device <NUM> through the communication interface <NUM>, or may be connected to another server connected through a network.

The processor <NUM> may execute a computer executable instruction to receive information related to a 3D space tracked by at least one vehicle <NUM> and corresponding to a movement location of the vehicle <NUM>, through the communication interface <NUM>, and to receive information related to a 3D space tracked by at least one sensing device <NUM> installed on a path along which the vehicle <NUM> travels and corresponding to a fixed location of the sensing device <NUM>, through the communication interface <NUM>. The processor <NUM> may reconstruct information related to a 3D space corresponding to a predetermined region to which both the movement location of the vehicle <NUM> and the fixed location of the sensing device <NUM> belong, based on the information related to the 3D space corresponding to the movement location of the vehicle <NUM> and the information related to the 3D space corresponding to the fixed location of the sensing device <NUM>.

The processor <NUM> may transmit the reconstructed information related to the 3D space corresponding to the predetermined region to an integrated server of an upper layer through the communication interface <NUM>.

<FIG> is a block diagram showing a hierarchical structure of servers (<NUM>-<NUM> to <NUM>-N and <NUM>) according to an unclaimed embodiment.

Referring to <FIG>, all sensing ranges of the vehicle <NUM> and the sensing device <NUM> belong to a predetermined region. The predetermined region may be managed by any one server <NUM> among servers (<NUM>-<NUM>, <NUM>-<NUM>,. , <NUM>-N, hereinafter, representatively indicated by <NUM>) which are local servers.

The server <NUM> may collect information related to a 3D space tracked by the vehicle <NUM> or the sensing device <NUM> within the region which the server <NUM> manages, and map the information to a corresponding space in the region to thereby reconstruct information related to a 3D space corresponding to the entire region. In other words, the server <NUM> corresponding to a local server may acquire information related to a 3D space for a region which itself manages. When a predetermined criterion is satisfied or a request is received, the server <NUM> corresponding to a local server may transmit the entire or a part of the information related to the 3D space for the region which itself manages to the vehicle <NUM> or the sensing device <NUM> located in the region. Also, the server <NUM> may transmit the information related to the 3D space for the region which itself manages to a server <NUM> corresponding to a global server managing the local servers. The server <NUM> may be an integrated server of an upper layer of the server <NUM>.

The server <NUM> may collect the information related to the 3D space corresponding to the predetermined region reconstructed by the local servers, and map the information to a corresponding space in a global region to thereby acquire information related to a 3D space corresponding to the global region which the server <NUM> manages. In other words, the server <NUM> corresponding to the global server may acquire the information related to the 3D space for the global region which itself manages. When a predetermined criterion is satisfied or a request is received, the server <NUM> corresponding to a global server may transmit the entire or a part of the information related to the 3D space for the global region which itself manages to the server <NUM> corresponding to a local server.

As shown in <FIG>, a hierarchical structure is formed between the vehicle <NUM> including the sensor unit <NUM>, the sensing device <NUM> including the sensor unit <NUM>, the server <NUM> corresponding to a local server, and the server <NUM> corresponding to a global server. The information related to the 3D space tracked by the vehicle <NUM> and the sensing device <NUM> may be transferred to the upper layer and integrated to become information related to a 3D space corresponding to an entire space.

<FIG> is a view for describing a state in which the vehicle <NUM> according to an embodiment travels based on information related to a tracked 3D space.

Referring to <FIG>, the vehicle <NUM> travels based on information related to a 3D space corresponding to a movement location of the vehicle <NUM> and information related to a movement and position of the vehicle <NUM>. The information related to the 3D space corresponding to the movement location of the vehicle <NUM> may include at least one object. As shown in <FIG>, a vehicle of a predetermined level or higher may be classified into an object of a first type, a two-wheeled vehicle or a small vehicle that is lower than the predetermined level may be classified into an object of a second type, a pedestrian may be classified into an object of a third type, a traveling path of the vehicle <NUM> may be classified into an object of a fourth type, and another sensed object except for the objects of the first to fourth types may be classified into an object of a fifth type. The objects of the respective types may have different 3D shapes including heights, different sizes, different colors, etc. Locations, positions, speeds, etc. of the individual objects may be determined based on information related to a 3D space tracked according to a movement location of the vehicle <NUM>, and by continuing to track the objects, displacements, change amounts, trajectories, courses, etc. may be checked or predicted. When the vehicle <NUM> is an autonomous vehicle, surrounding objects changing according to a movement of the vehicle <NUM> may be tracked to be used as data for controlling driving of the vehicle <NUM>.

As shown in the right, lower area of <FIG>, a speed and direction of the vehicle <NUM> and an angle of steering system may be checked, and changes of the speed and direction of the vehicle <NUM> and the angle of the steering system according to a movement of the vehicle <NUM> may be tracked.

<FIG> is a view for describing a state in which a vehicle <NUM> according to another embodiment travels based on information related to a tracked 3D space.

Referring to <FIG>, the vehicle <NUM> travels based on information related to a 3D space corresponding to a movement location of the vehicle <NUM> and information related to a movement and position of the vehicle <NUM>. Comparing FIG. <NUM> to FIG. <NUM>, differences in number and interval of concentric circles surrounding the vehicle <NUM> are found. In the case of <FIG>, the vehicle <NUM> may sense a 3D space successively and respectively within different sensing concentric ranges of a spherical shape through a plurality of 3D LiDAR sensors, and acquire spatial information over time for the 3D space sensed within the different sensing ranges to track a 3D space corresponding to a movement location of the vehicle <NUM>.

<FIG> is a view for describing a state in which the sensing device <NUM> according to an embodiment tracks information related to a 3D space corresponding to its fixed location.

Referring to <FIG>, two sensing devices <NUM> may be located with a predetermined distance, and each of the sensing devices <NUM> may sense a 3D space successively by at least one sensor to track information related to a 3D space corresponding to a fixed location of the sensing device <NUM>. It is seen that a vehicle corresponding to ID <NUM> moves at a speed of about <NUM>/h away from the sensing devices <NUM>, and a pedestrian corresponding to ID <NUM> moves at a speed of about <NUM>/h from the sensing device <NUM> located to the left to the sensing device <NUM> located to the right. In this case, to inform movement directions of the objects, arrows corresponding to the movement directions of the objects may be provided. Because the sensing devices <NUM> determine what object passes through or invades the sensing ranges, the sensing devices <NUM> may be used for a security purpose and for a monitoring purpose such as traffic observation for a predetermined region.

<FIG> is a view for describing a state in which the vehicle <NUM> according to an embodiment travels based on information related to a 3D space tracked by the vehicle <NUM> and corresponding to a movement location of the vehicle <NUM>. <FIG> is a view for describing a state in which the vehicle <NUM> according to an embodiment travels based on information related to a 3D space tracked by the vehicle <NUM> and corresponding to a movement location of the vehicle <NUM> and information related to a 3D space tracked by the sensing device <NUM> and corresponding to a fixed location of the sensing device <NUM>.

Referring to <FIG>, because the vehicle <NUM> shown in <FIG> uses a sensor included therein to acquire information related to a 3D space corresponding to a movement location of the vehicle <NUM>, a sensing range of the sensor may be limited when there are objects surrounding the vehicle <NUM>. In contrast, because the vehicle <NUM> shown in <FIG> acquires, by using a sensor included in at least one sensing device <NUM> installed on a path along which the vehicle <NUM> travels, information related to a 3D space corresponding to a fixed location of the sensing device <NUM>, as well as acquiring information related to a 3D space corresponding to a movement location of the vehicle <NUM> by using the sensor included in the vehicle <NUM>, a 3D space that may be sensed by the vehicle <NUM> of <FIG> may be significantly wider than a 3D space that may be sensed by the vehicle <NUM> of <FIG>. The vehicle <NUM> of <FIG> may track a pedestrian corresponding to the object of the third type and all kinds of vehicles corresponding to the object of the first type and check information about the pedestrian and vehicles, whereas the vehicle <NUM> of <FIG> may not track a pedestrian corresponding to the object of the third type and some kinds of vehicles corresponding to the object of the first type.

<FIG> is a flowchart for describing a process of tracking a sensed 3D space based on spatial information over time for the 3D space and predicting information related to the tracked 3D space. <FIG> are views for describing operations of a process of tracking a sensed 3D space based on spatial information over time for the 3D space and predicting information related to the tracked 3D space. Hereinafter, a process of tracking a sensed 3D space based on spatial information over time for the 3D space and predicting information related to the tracked 3D space will be described with reference to <FIG>.

In operation <NUM>, the vehicle <NUM> or the sensing device <NUM> distinguishes an object area from a ground area based on spatial information over time for a 3D space. The spatial information over time for the 3D space sensed by the vehicle <NUM> or the sensing device <NUM> may be in the form of point cloud data.

The vehicle <NUM> or the sensing device <NUM> may distinguish the ground area from the spatial information over time for the 3D space. The vehicle <NUM> or the sensing device <NUM> may distinguish point cloud data corresponding to the ground area among the point cloud data. The vehicle <NUM> or the sensing device <NUM> may first distinguish the ground area from the spatial information over time for the 3D space, and then distinguish the remaining area as the object area including at least one object. The vehicle <NUM> or the sensing device <NUM> may apply fitting based on a stochastic model to find a ground estimation model. The vehicle <NUM> or the sensing device <NUM> may train a ground shape in real time, and distinguish the ground area depending on whether or not each point cloud data is point cloud data corresponding to the ground area.

In <FIG>, a process of distinguishing an object area from a ground area based on spatial information at a predetermined time for a 3D space sensed by the vehicle <NUM> or the sensing device <NUM> is shown. For convenience of description, as shown in <FIG>, a case in which spatial information includes three objects and a ground will be described as an example.

Spatial information at a predetermined time for a 3D space sensed by the vehicle <NUM> or the sensing device <NUM> may be, as shown in the upper area of <FIG>, point cloud data in which objects are not distinguished from a ground, and may be point cloud data corresponding to all sensed things. The vehicle <NUM> or the sensing device <NUM> may separate point cloud data estimated to correspond to a ground area from the entire point cloud data, as shown in the upper part of <FIG>, thereby distinguishing point cloud data corresponding to an object area from point cloud data corresponding to the ground area, as shown in the lower area of <FIG>. Herein, the object area may include one or more objects. The object area may be point cloud data corresponding to the entire objects that are not distinguished from each other.

Referring again to <FIG>, in operation <NUM>, the vehicle <NUM> or the sensing device <NUM> clusters the object area to distinguish individual object areas. The vehicle <NUM> or the sensing device <NUM> distinguishes the individual object areas in order to distinguish the object area distinguished from the ground area according to objects. Because the object area distinguished from the ground area is point cloud data corresponding to the entire objects, the vehicle <NUM> or the sensing device <NUM> may cluster the point cloud data corresponding to the entire objects to thereby distinguish point cloud data for each object.

In <FIG>, a process in which the vehicle <NUM> or the sensing device <NUM> distinguishes the individual object areas corresponding to the respective objects from the object area distinguished from the ground area is shown.

As shown in the upper area of <FIG>, the point cloud data corresponding to the object area including the entire objects except for the ground may be distinguished from the point cloud data corresponding to the ground area. The vehicle <NUM> or the sensing device <NUM> may cluster the point cloud data corresponding to the object area based on at least one selected from the group consisting of distance information, shape information, and distribution information to thereby distinguish point cloud data corresponding to individual object areas of 'object <NUM>', 'object <NUM>', and 'object <NUM>' from the point cloud data corresponding to the object area including the entire objects, as shown in the lower area of <FIG>. As a result, the vehicle <NUM> or the sensing device <NUM> may acquire information about locations, shapes, and numbers of the objects.

Referring again to <FIG>, in operation <NUM>, the vehicle <NUM> or the sensing device <NUM> acquires object information of at least one object identified by applying a neural network based object classification model to spatial information over time for a 3D space. The vehicle <NUM> or the sensing device <NUM> may input the spatial information over time for the 3D space to the neural network based object classification model to identify at least one object, and acquire object information of the identified object. The neural network based object classification model may have been trained by using a training image databased for each object. The neural network based object classification model may estimate object information of each object identified based on at least one of distance information, shape information, and distribution information, for point cloud data corresponding to all sensed things. The vehicle <NUM> or the sensing device <NUM> may identify a movable object of interest, such as a vehicle, a small vehicle, a two-wheeled vehicle, a pedestrian, etc., through the neural network based object classification model, to estimate object information for the object of interest. Operation <NUM> may be performed in parallel to operations <NUM> and <NUM>.

In <FIG>, a process of acquiring object information of each object from spatial information at a predetermined time for a 3D space sensed by the vehicle <NUM> or the sensing device <NUM> is shown.

The spatial information at the predetermined time for the 3D space sensed by the vehicle <NUM> or the sensing device <NUM> may be point cloud data corresponding to all sensed things, as shown in the upper area of <FIG>. The vehicle <NUM> or the sensing device <NUM> may identify and classify 'object <NUM>', 'object <NUM>', and 'object <NUM>' and acquire object information of 'object <NUM>', 'object <NUM>', and 'object <NUM>' by applying the neural network based object classification model to the entire point cloud data, as shown in the lower area of <FIG>. By setting an object of a type corresponding to 'object <NUM>' to an object of interest in advance, the vehicle <NUM> or the sensing device <NUM> may identify and classify 'object <NUM>' set to an object of interest to acquire object information for 'object <NUM>'. As shown in the lower area of <FIG>, the vehicle <NUM> or the sensing device <NUM> may estimate a kind, location, and size value of each object, and determine various shapes of boundary lines or bounding boxes for the objects.

Referring again to <FIG>, in operation <NUM>, the vehicle <NUM> or the sensing device <NUM> tracks the sensed 3D space, based on the object information of the at least one object identified by applying the neural network based object classification model to the spatial information over time for the 3D space, the object information acquired in operation <NUM>, and the individual object areas acquired in operation <NUM>. The neural network based object classification model applied in operation <NUM> may have difficulties in identifying an object that has not sufficiently been trained and estimating object information of the object. Therefore, the vehicle <NUM> or the sensing device <NUM> may acquire information related to a location, shape, number, etc. of an object that may not be identified, from the individual object areas acquired in operation <NUM>. Also, the information of the object estimated by the neural network based object classification model applied in operation <NUM> may be different from actual information of the object on the sensed 3D space. Therefore, the vehicle <NUM> or the sensing device <NUM> may correct the information of the object by using the information related to the location, shape, number, etc. of the object, which may be acquired from the individual object areas acquired in operation <NUM>. As a result, the information of the object estimated by the neural network based object classification model may be integrated with the information related to each object acquired from the individual object areas distinguished through clustering to identify the object and to acquire correct information related to the location, shape, etc. of the object. Also, an object that may be not identified or distinguished through the neural network based object classification model may be identified by using the information related to the object acquired from the individual object areas distinguished through clustering, so that all objects on the sensed 3D space may be tracked without being missed.

In <FIG> and <FIG>, a process of integrating information related to an object acquired from the individual object areas distinguished through clustering by the vehicle <NUM> or the sensing device <NUM> with information of the object estimated by the neural network based object classification model to track the object on the sensed 3D space is shown.

Referring to <FIG>, the vehicle <NUM> or the sensing device <NUM> may integrate information related to individual objects of 'object <NUM>', 'object <NUM>', and 'object <NUM>' acquired based on point cloud data corresponding to the individual object areas with object information of the individual objects identified as and classified into 'object <NUM>', 'object <NUM>', and 'object <NUM>' to acquire correct information for all objects on the sensed 3D space. The vehicle <NUM> or the sensing device <NUM> may correct various shapes of boundary lines or bounding boxes for the individual objects according to information about the individual objects acquired based on point cloud data corresponding to the individual object areas to acquire correct information about the objects.

Referring to <FIG>, the vehicle <NUM> or the sensing device <NUM> may acquire continuous information for all objects on the sensed 3D space, over time, from spatial information over time for the sensed 3D space, to track all the objects on the sensed 3D space. For example, the vehicle <NUM> or the sensing device <NUM> may track each object over time according to an object tracking method using a Kalman filter. As shown in <FIG>, the vehicle <NUM> or the sensing device <NUM> may track a speed and movement direction of each object based on a change amount in location of the object, from continuous frame information over time, and record the tracked result.

Referring again to <FIG>, in operation <NUM>, the vehicle <NUM> or the sensing device <NUM> may predict information related to the tracked 3D space. The vehicle <NUM> or the sensing device <NUM> may accumulate the tracked information of the object, and analyse a movement pattern of the object from the accumulated tracked information to predict a movement of the object. The vehicle <NUM> or the sensing device <NUM> may predict a movement of an object of interest among at least one object identified on the tracked 3D space to reduce a computation amount of related processing, thereby planning efficient driving or monitoring.

Each of the above-described embodiments may be provided in the form of a computer program or an application stored in a computer readable storage medium to perform a method, in an electronic device, including predetermined operations using spatial information acquired by using a sensor. In other words, each of the above-described embodiments may be provided in the form of a computer program or an application stored in a computer readable storage medium to enable an electronic device to execute a method including predetermined operations using spatial information acquired by using a sensor.

Claim 1:
A vehicle (<NUM>) comprising:
a sensor unit (<NUM>) configured to successively sense a three-dimensional (3D) space by using at least one sensor;
a memory (<NUM>) storing a computer executable instruction; and
a processor (<NUM>) configured to execute the computer executable instruction to acquire spatial information over time for the sensed 3D space, identify at least one object in the sensed 3D space by applying a neural network based object classification model to the acquired spatial information over time (<NUM>), track the sensed 3D space including the identified at least one object, and control driving of the vehicle (<NUM>) based on information related to the tracked 3D space and information related to a movement and position of the vehicle (<NUM>);
characterized in that the processor (<NUM>) is further configured to distinguish an object area from a ground area based on the acquired spatial information over time (<NUM>), cluster the object area to distinguish individual object areas (<NUM>), and track the sensed 3D space based on object information of the at least one object identified by applying the neural network based object classification model to the acquired spatial information over time and the distinguished individual object areas (<NUM>).