VEHICLE CONTROL SYSTEM AND OPERATING METHOD THEREOF

A vehicle control system comprises an image signal processor (ISP), a first neural processing unit (NPU), a second NPU, a data processing circuit, and sensors mounted on a vehicle. The ISP receives a first image and processes it to generate a second image. The first NPU and the second NPU both independently segment the second image to identify a type of the object and produce data related to the object. The data processing circuit receives data from both NPUs and sensor data from the sensors and determines whether either of the first NPU or the second NPU is abnormal by comparing correlations between the NPU data and the sensor data during a first frame interval.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application No. 10-2023-0096624 filed on Jul. 25, 2023 in the Korean Intellectual Property Office under 35 U.S.C. 119, the entire contents of which are herein incorporated by reference.

BACKGROUND

The present disclosure relates to a vehicle control system, an automotive system including the vehicle control system, and a method for operating the same.

In recent years, automobiles are rapidly becoming smarter due to the fusion of information and communication technology and the automobile industry. Accordingly, automobiles are evolving from simple mechanical devices into smart cars, and in particular, an Advanced Driver Assistance System (ADAS) and an Automatic Driving (AD) system are attracting attention as core technologies for the smart cars.

For the ADAS and the AD system, various technologies are used, such as a technology for recognizing the vehicle's driving environment including other nearby vehicles and pedestrians, technology for determining the vehicle's driving situation, and control technology such as vehicle's driving, startup, and steering. These technologies may rely on accurately and efficiently recognizing and detecting objects around the vehicle, using, e.g., artificial intelligence.

SUMMARY

In general, in some aspects, the present disclosure relates to vehicle control systems for improving the stability of an autonomous vehicle.

In general, in some aspects, the present disclosure relates to automotive systems for improving the stability of the autonomous vehicle.

In general, in some aspects, the present disclosure relates to methods for operating the vehicle control system for improving the stability of the autonomous vehicle.

In general, in some aspects, the present disclosure relates to a vehicle control system that comprises an image signal processor (ISP), which is configured to receive a first image obtained by capturing an object around a vehicle during a predetermined frame interval and to process the first image to generate a second image. The vehicle control system also comprises a first neural processing unit (NPU), which is configured to: receive the second image from the ISP; perform a first image segmentation on the second image to identify a type of the object; and generate first data about a numerical value of a region occupied by the object within the second image. The vehicle control system also comprises a second NPU, which is configured to: receive the second image from the ISP; perform a second image segmentation on the second image to identify the type of the object; and generate second data about the numerical value of the region occupied by the object within the second image. The vehicle control system also comprises a data processing circuit, which is configured to: receive each of the first and second data from the first and second NPUs; receive sensing data about a driving state of the vehicle during the predetermined frame interval from a sensing system mounted on the vehicle; and then process the first and second data and the sensing data to determine whether each of the first and second NPUs is abnormal. The data processing circuit is configured to learn a correlation between an amount of change in the first data during the first frame interval and an amount of change in the sensing data during the first frame interval, in response to the fact that the amount of change in the first data during the first frame interval is equal to the amount of change in the second data during the first frame interval.

In general, in some aspects, the present disclosure relates to an automotive system comprising a camera, which is configured to capture an object around a vehicle during a predetermined frame interval to generate a first image, a sensing system, which is configured to sense data related to a driving state of the vehicle during the predetermined frame interval to generate sensing data, and a vehicle control system, which is configured to control the vehicle based on the first image. The vehicle control system includes an image signal processor (ISP), which is configured to process the first image to generate a second image. The vehicle control system includes a first neural processing unit (NPU), which is configured to receive the second image from the ISP; performs a first image segmentation on the second image to identify a type of the object; and generate a first data about a numerical value of a region occupied by the object within the second image. The vehicle control system includes a second NPU, which is configured to receive the second image from the ISP; perform a second image segmentation on the second image to identify the type of the object; and generate a second data about the numerical value of the region occupied by the object within the second image. The vehicle control system includes a data processing circuit, which is configured to receive each of the first and second data from the first and second NPUs; receive the sensing data from the sensing system; and then process the first and second data and the sensing data to determine whether each of the first and second NPUs is abnormal. Based on an amount of change in the first data during a first frame interval being equal to an amount of change in the second data during the first frame interval, the data processing circuit is configured to learn a correlation between the amount of change in the first data during the first frame interval and the amount of change in the sensing data during the first frame interval.

In general, in some aspects, the present disclosure is related to a method for operating a vehicle control system, in which the method includes receiving a first image obtained by capturing an object around a vehicle during a predetermined frame interval, and processing the first image to generate a second image by an image signal processor (ISP). The method also comprises receiving, by a first neural processing unit (NPU), the second image from the ISP, performing a first image segmentation on the second image to identify a type of the object, and generating a first data about a numerical value of a region occupied by the object within the second image. The method also comprises receiving, by a second NPU, the second image from the ISP, performing a second image segmentation on the second image to identify the type of the object, and generating a second data about the numerical value of the region occupied by the object within the second image. The method also comprises receiving, by a data processing circuit, each of the first and second data from the first and second NPUs, and a plurality of pieces of sensing data about a driving state of the vehicle during the predetermined frame interval from a sensing system mounted on the vehicle and, based on an amount of change in the first data during a first frame interval being equal to an amount of change in the second data during the first frame interval, calculating, by the data processing circuit, each correlation between the amount of change in the first data during the first frame interval and the amount of change in each of the plurality of the sensing data during the first frame interval. The method also comprise storing, by the data processing circuit, an item of first sensing data that is determined to have a correlation with the amount of change in the first data during the first frame interval, among the plurality of pieces of sensing data.

It should be noted that the effects of the present disclosure are not limited to those described above, and other effects of the present disclosure will be apparent from the following description.

DETAILED DESCRIPTION

A vehicle control system, an automotive system including the vehicle control system, and a method for operating the same are described below with reference to the accompanying drawings.

FIG.1is an example diagram showing a vehicle control system.FIG.2is an exemplary diagram showing an automotive system including the vehicle control system ofFIG.1. Hereinafter, the vehicle control system and the automotive system will be described with reference toFIGS.1and2.

An automotive system400may include a camera200, a vehicle control system100, and a sensing system300. The camera200may be mounted on a vehicle, and may include an image sensor that senses objects around the vehicle by the use of light to generate image signals. The camera200may generate the image10by capturing images of a plurality of objects around the vehicle, including objects in front of the vehicle.

The vehicle control system100may be an ADAS (Advanced Driver Assistance System) or an automatic driving system installed in the vehicle. The vehicle control system100may be implemented as an integrated circuit (IC), a system on chip (SOC) or an application processor (AP). The vehicle control system100may include an image signal processor (ISP), a first NPU (neural processing unit)120, a second NPU130, a data processing circuit140, a controller150, memories121,131and141, and a bus160. However, the vehicle control system100may further include other configurations (for example, IP (Intellectual Property)), in addition to the configurations shown inFIGS.1and2.

The ISP110may receive the image10from the camera200and process the image10to generate an image20. For example, the ISP110may receive the image10output from the image sensor of the camera200, and fabricate or process the received image10to facilitate recognition and processing by the NPUs120and130. In some implementations, the ISP110may perform digital binning on the image10that is output from the image sensor. At this time, the image10that is output from the image sensor may be a raw image signal that is output from the pixel array of the image sensor without analog binning, or may be an image signal subjected to analog binning.

The first NPU120may receive the image20from the ISP110, identify information of the objects included in the received image20, and generate selected data30about the identified information of the objects. For example, the first NPU120may perform an image segmentation on the received image20to generate bounding boxes for each object within the image20. The operations of the NPUs120and130will be described in detail later with reference toFIG.4or the like.

In this way, the first NPU120may receive the image20about objects around the vehicle from the ISP110, perform a neural network operation based on the image20, and generate an information signal (i.e., selected data30) about the results obtained by recognizing the image20. At this time, the selected data30generated by the first NPU120may be information that serves as a reference when the controller150controls the autonomous vehicle. For example, the selected data30generated by the first NPU120based on the image20received from the ISP110may be transferred to the controller150, and the controller150may control the autonomous vehicle based on the selected data30received from the first NPU120.

The first NPU120may also transfer first data40generated based on the image received from the ISP110to the data processing circuit140. In some implementations, the first data40may correspond to some of the selected data30. That is, the first NPU120may transmit all (that is, the selected data30) of the information signals about the result obtained by recognizing the image20received from the ISP110to the controller150, and may transmit the first data40corresponding to some of the information signals to the data processing circuit140. For example, the first NPU120may transmit only the first data40corresponding to highly reliable data among the generated selected data30to the data processing circuit140. The criteria of the reliability described above will be described later with reference toFIG.4or the like.

The second NPU130may include the same configuration as that of the first NPU120, and may perform the same functions as the first NPU120. For example, like the first NPU120, the second NPU130may receive the image20from the ISP110, and perform the image segmentation or the like on the received image20. In some implementations, the first NPU120and the second NPU130may each receive images20from the ISP110and perform the image segmentation in parallel. That is, the operations of the first NPU120and the second NPU130for performing the image segmentation on the image20may occur independently of each other.

In addition, in parallel with the operation of transmitting the selected data30generated by the first NPU120to the controller150and the operation of transmitting the first data40corresponding to some of the selected data30generated by the first NPU120to the data processing circuit140, the second NPU130may independently transfer the second data50generated by performing the image segmentation to the controller150. In some implementations, the second data50may correspond to some of highly reliable data generated by performing the image segmentation by the second NPU130. The criteria of reliability described above will be described later with reference toFIG.4or the like.

In some implementations, the controller150may also control the autonomous vehicle based on data received from the second NPU130in addition to data received from the first NPU120. For example, if there is an abnormality in the operation of the first NPU120, the controller150may control the autonomous vehicle based on data received from the second NPU130. In this way, the vehicle control system100may include two or more NPUs that perform the same function. Alternatively, the vehicle control system100may further include at least one NPU that performs a different function from those of the NPUs120and130. Hereinafter, a case in which the vehicle control system100includes two NPUs120and130that perform the same function will be described as an example.

In some implementations, the first NPU120and the second NPU130may be divided into a main NPU and a sub NPU inside the vehicle control system100. For example, the main NPU may perform a neural network operation based on the image20received from the ISP110, and transmit the executed result to the controller150. On the other hand, like the main NPU, the sub NPU may perform the neural network operations based on the image20received from the ISP110, but may not transmit the executed results to the controller150. Therefore, the controller150may control the autonomous vehicle based on only the information received from the main NPU.

However, if the main NPU is out of order or not operating normally, the sub NPU may transmit the results obtained by performing the neural network operation to the controller. At this time, the main NPU may not transmit the data to the controller150, and thus, the controller150may control the autonomous vehicle based on only the data received from the sub NPU. Hereinafter, an example will be described in which the first NPU120is the main NPU and the second NPU130is the sub NPU.

The data processing circuit140may receive each of first data40and second data50from the first NPU120and the second NPU130. Further, the data processing circuit140may receive sensing data60about the driving state of the vehicle from the sensing system300. The data processing circuit140may process the received first data40and50and the sensing data60to determine whether the first NPU120and the second NPU130operate normally.

For example, the data processing circuit140may determine whether both the first NPU120and the second NPU130operate normally based on the first data40and the second data50. When it is determined that both the first NPU120and the second NPU130operate normally, the data processing circuit140may learn a correlation between an amount of change in first data40during a predetermined frame interval and an amount of change in sensing data60during a predetermined frame interval, based on the first data40generated by the first NPU120corresponding to the main NPU and the sensing data60. In some implementations, when a plurality of pieces of sensing data60are received from the sensing system300, the data processing circuit140may learn multiple correlations between the amount of change in first data40during a predetermined frame interval and the amounts of change in each of the plurality of pieces of sensing data60during a predetermined frame interval.

Further, the data processing circuit140may determine that either the first NPU120or the second NPU130operates abnormally based on the first data40and the second data50. When it is determined that either the first NPU120or the second NPU130operates abnormally, the data processing circuit140may discern the NPU that operates abnormally among the first NPU120and the second NPU130based on the result of the correlation learning. The operation of performing the correlation learning and the operation of discerning the abnormally operating NPU by the data processing circuit140will be described later with reference toFIG.8.

In some implementations, the data processing circuit140may be implemented as a digital signal processor (DSP), a NPU, core, and the like. For example, the data processing circuit140may be an extra core inside a big core for AP application.

Memories121,131, and141may be used as main memory or system memory of the vehicle control system100. The memory121may be connected to the first NPU120, the memory131may be connected to the second NPU130, and the memory141may be connected to the data processing circuit140. For example, each of the memories121,131, and141may temporarily store data or signals to be transferred to the first NPU120, the second NPU130, and the data processing circuit140, and may temporarily store data or signals generated by the first NPU120, the second NPU130, and the data processing circuit140.

In some implementations, each of the memories121,131, and141may include, but not be limited to, a volatile memory such as a dynamic random access memory (DRAM), a static random access memory (SRAM), and a synchronous dynamic random access memory (SDRAM). For example, the memories121,131, and141may include a non-volatile memory such as a flash memory, a flash change RAM (PRAM), a resistive RAM (RRAM), and a magnetic RAM (MRAM).

When the first NPU120corresponding to the main NPU operates normally, the controller150may control driving, maneuvering, steering, and the like of the vehicle based on the selected data30received from the first NPU120. For example, the controller150may be ADAS function software. As mentioned above, the controller150may control the vehicle based on the selected data30received from the first NPU120when the first NPU120corresponding to the main NPU operates normally. However, the controller150may control the vehicle based on data received from the second NPU130corresponding to the sub NPU when it is determined that the main NPU operates abnormally by the data processing circuit140.

The components110,120,130,140, and150included in the vehicle control system100may communicate with each other through an interconnection such as a bus160.

The sensing system300may include a plurality of sensors310,320,330,340, and350. The plurality of sensors310,320,330,340, and350are mounted on the vehicle, and may sense the state of the vehicle and generate sensing data60including information about the driving state of the vehicle. However, the types of sensors included in the sensing system300are not limited to those shown inFIG.2. The plurality of sensors310,320,330,340, and350include an accelerator sensor310, a gyroscope sensor320, a torque sensor330, a wheel speed sensor340, a steering angle sensor350, and the like. In addition, the sensing system300may include a collision sensor, a speed sensor, a tilt sensor, a weight perception sensor, a heading sensor, a position module, a vehicle forward/reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a vehicle internal temperature sensor, a vehicle internal humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like. The sensing system300may transmit at least one of the sensed data sensed by the aforementioned sensors to the data processing circuit140.

FIG.3is a diagram showing an example in which a vehicle control system is installed in a vehicle.FIG.4is an exemplary diagram in which the NPU included in the vehicle control system ofFIG.3executes the image segmentation. Hereinafter, an example in which the NPU included in the vehicle control system performs the image segmentation will be explained with reference toFIGS.3and4.

Referring first toFIG.3, the vehicle control system100(shown inFIG.1) may be an ADAS, an automatic driving system installed in the vehicle1000, and a camera200and a sensing system300may be mounted on the vehicle1000. The vehicle control system100may receive images from the camera200installed in front of the vehicle1000, but implementations are not limited thereto. For example, the vehicle control system100may receive images from a camera that may capture images of the surroundings of the vehicle1000. The surroundings of the vehicle1000may include, for example, a front, a side, and a rear of the vehicle.

In some implementations, the camera200may image a plurality of objects located in front of the traveling vehicle1000during a predetermined frame interval. InFIGS.3and4, among a plurality of objects located in front of the vehicle1000, a tree1001, another vehicle1002, and a pedestrian1003will be explained examples. The plurality of objects may have a static speed (e.g., the tree1001) or may have a dynamic speed (e.g., another vehicle1002).

Next, referring toFIG.4, the image20may be an image obtained by fabricating and processing by the ISP110such that the image (10, shown inFIG.2) captured by the camera200may be recognized by the first NPU120and the second NPU130. As mentioned above, the first NPU120and the second NPU130perform the same function, and the explanation will be given by an example in which the first NPU120, corresponding to the main NPU, performs the image segmentation. However, it goes without saying that the following explanation may be similarly applied to a case where the second NPU130corresponding to the sub NPU performs the image segmentation.

After the first NPU120receives the image20from the ISP110, it may perform image segmentation on the image20. The image segmentation may be a process of assigning label information to each pixel of a two-dimensional image. For example, the first NPU120may generate bounding boxes B1, B2, and B3for recognizing and separating the objects on the plurality of objects1001,1002, and1003included in the image20. The bounding box may be a box of the smallest size which may include the shapes of all objects in the image. The label information may include information about the type of object within the bounding box.

For example, the first NPU120may generate a bounding box BI for the tree1001, and may identify that the object within the bounding box B1is a tree. Similarly, the first NPU120may generate each of bounding boxes B2and B3for another vehicle1002and the pedestrian1003, and identify that the objects in the bounding boxes B2and B3are another vehicle and pedestrian.

In this way, the bounding box may correspond to a region occupied by the object within image20, and the first NPU120may define positions of each object within the image20through the bounding box. However, the embodiment is not limited thereto, and the first NPU120may receive a 3D image from the ISP110, generate a bounding box for each object in the 3D image, and assign label information of each pixel.

In some implementations, the first NPU120may identify the type of each object for each object included in the bounding boxes B1, B2, and B3, and tag a confidence score for the confidence of the identification result to each of the bounding boxes B1, B2, and B3. For example, the first NPU120may identify that the object included in the bounding box B1is a tree1001, and tag a confidence score thereof (score1) to the bounding box B1.

The first NPU120may determine that the reliability score is high when the reliability score is equal to or greater than a predetermined threshold value. For example, if the reliability score (score2) tagged to the bounding box B2is equal to or greater than the predetermined threshold value, the first NPU120may determine that the reliability obtained by identifying the type of object included in the bounding box B2is high. Alternatively, the first NPU120identifies that the object included in the bounding box B3is the pedestrian1003, and tags the corresponding confidence score (score3) to the bounding box B3, but the confidence score may be equal to or less than the predetermined threshold value. In this way, when the confidence score tagged to the bounding box is equal to or less than a predetermined threshold value, the type of object included in the corresponding bounding box identified by the first NPU120may differ from the type of object actually included in that bounding box.

In some implementations, the first NPU120may generate data about the numerical value of the region occupied by the objects1001,1022, and1003within the image20. For example, the first NPU120may identify the reference coordinates of each of the bounding boxes B1, B2, and B3, the lengths of each of the bounding boxes B1, B2, and B3in the first direction X, and the lengths of each of the bounding boxes B1, B2, and B3in the second direction Y, and generate data about the reference coordinates and the lengths. Hereinafter, the reference coordinates A and lengths W1and H1of the bounding box B1will be explained as examples, but it goes without saying that the same explanation is also applicable to the reference coordinates B and lengths W2and H2of the bounding box B2and the reference coordinates C and length W3and H3of the bounding box B3.

For example, the reference coordinate A of the bounding box B1may be any of the four vertices of the bounding box B1. In some implementations, as shown inFIG.4, the reference coordinates of the bounding box B1may be, but not limited to, coordinates (X1, Y1) of a vertex A corresponding to a top left of the bounding box B1. Further, the first NPU120may generate data about a length W1in the first direction X, which corresponds to the width of the bounding box B1, and a length H1in a second direction Y perpendicular to the first direction X, which corresponds to the height of the bounding box B1.

In this way, the first NPU120may perform the image segmentation on the plurality of objects1001,1002, and1003within the image20received from the ISP110, form a bounding box for each of the plurality of objects, after identifying the type of object within the bounding box, tag a confidence score to the bounding boxes, and generate data about the reference coordinates of each bounding box and the widths and heights of each bounding box.

FIG.5is an exemplary flowchart for explaining the operation of the automotive system ofFIG.2.FIGS.6and7are exemplary diagrams for explaining a step in which the NPU performs the image segmentation to generate data among the steps ofFIG.5. Hereinafter, with reference toFIGS.5to7, an operation will be explained in which the data processing circuit140receives first data generated by performing the image segmentation during the first frame interval by the first NPU120, and second data generated by performing the image segmentation during the first frame interval by the second NPU130. In addition, below, in order to distinguish the image segmentations performed by the first NPU120and the second NPU130, the image segmentations performed by the first NPU120and the second NPU130will be defined as a first image segmentation and a second image segmentation, respectively.

First, the camera200mounted on the vehicle1000(shown inFIG.3) may capture objects around the vehicle1000during a first frame interval to generate an image10(S100). In some implementations, the first frame interval may be, but not be limited to, 100 to 150 milliseconds. The image10may include a plurality of images obtained by capturing the objects around the vehicle1000during the first frame interval. Subsequently, the camera200may transmit the generated image10to the ISP110(S101).

The ISP110may receive the image10from the camera200(S102), and process the image10to generate the image20(S103). The image20may include a plurality of images obtained by processing a plurality of images10obtained by capturing the objects around the vehicle1000during the first frame interval. After that, the ISP110may transmit the generated image20to each of the first NPU120and the second NPU130(S104).

The first NPU120may receive the image20from the ISP110(S105), and perform the first image segmentation on the image20to generate first data40(S106). The first data40generated by the first NPU120may include at least one of a bounding box generated for the object included in the image20by performing the first image segmentation on the image20, information about the type of object included in the bounding box, the confidence score about the type of object tagged to the bounding box, reference coordinates of the bounding box, and information about the width and height of the bounding box. Subsequently, the first NPU120may transmit the first data40to the data processing circuit140(S107).

In some implementations, when the image20includes the plurality of objects, the first NPU120may generate a plurality of pieces of selected data30corresponding to each of the plurality of objects. At this time, the first NPU120may transmit only the first data40corresponding to the object with a high reliability score among the plurality of generated selected data30to the data processing circuit140. However, the embodiment is not limited thereto, and the first NPU120may transmit all of the plurality of pieces selected data30corresponding to each of the plurality of objects generated by performing the first image segmentation to the data processing circuit140.

The second NPU130may receive image20from the ISP110(S108), and perform the second image segmentation on the image20to generate second data50(S109). The second data50generated by the second NPU130may include at least one of a bounding box generated for the object included in the image20by performing the second image segmentation on the image20, information about the type of object included in the bounding box, a confidence score about the type of object tagged to the bounding box, reference coordinates of the bounding box, and information about the width and height of the bounding box.

Further, the second data50may correspond to some of the data generated by performing a second image segmentation by the second NPU130. For example, when the image20includes a plurality of objects, the second NPU130may generate a plurality of pieces of second data50corresponding to each of the plurality of objects. At this time, the second NPU130may transmit only the second data50corresponding to the object with a high reliability score among the plurality of generated second data50to the data processing circuit140. However, the embodiment is not limited thereto, and the second NPU130may transmit all of the plurality of pieces of second data50corresponding to each of the plurality of objects generated by performing the second image segmentation to the data processing circuit140.

Subsequently, the second NPU130may transmit the second data50to the data processing circuit140(S110). Therefore, the data processing circuit140may receive the first data40and the second data50(S111).

In some implementations, the operations in which the first NPU120and the second NPU130each receive the image20from the ISP110and perform the first and second image segmentations to generate the first data40and the second data50may be performed in parallel.

In parallel with the operation of generating the first data40and the second data50by the first NPU120and the second NPU130, the sensing system300may sense data about the driving state of vehicle1000during the same frame interval as the first frame interval at which the camera200captures the object to generate the sensing data60(S112). The sensing system300may transmit the generated sensing data to the data processing circuit140(S113). Therefore, the data processing circuit140may receive the sensing data60(S114).

Next, referring toFIG.6,FIG.6is an exemplary diagram showing that the first NPU120performs a first image segmentation when the vehicle1000(shown inFIG.3) makes a left turn during a first frame interval. Hereinafter,FIGS.6and7explain a case in which the first NPU120performs the first image segmentation on the images during the first frame interval as an example, but it goes without saying that the following explanation is also similarly applicable to the case where the second NPU130performs the second image segmentation on the images during the first frame interval.

The images20A,20B, and20C may be images sequentially received from the ISP110by the first NPU120. In some implementations, the first frame interval may include three consecutive frames, e.g., a frame A, a frame B, and a frame C. The frame A, the frame B, and the frame C may represent the same frame in chronological order. At this time, the image20A may correspond to the frame A, the image20B may correspond to the frame B, and the image20C may correspond to the frame C. Hereinafter, a case in which the object A has a static speed and the first NPU120operates normally will be explained.

When the vehicle1000makes a left turn, the object A, which has a static velocity, moves relatively to the right within the same frame. As a result, an X value of the reference coordinates of the bounding box B4(for example, the coordinates corresponding to the top left of the bounding box B4) may become increasingly large as time passes. For example, an X value X4of the reference coordinate D1of the bounding box B4in the frame A may be smaller than an X value X5of the reference coordinate D2of the bounding box B4in the frame B. The X value X5of the reference coordinate D2of the bounding box B4in the frame B may be smaller than an X value X6of the reference coordinate D3of the bounding box B4in the frame C.

The first NPU120may transmit each of the X value X4of the reference coordinate D1of the bounding box B4in the frame A, the X value X5of the reference coordinate D2of the bounding box B4in the frame B, and the X value X6of the reference coordinate D3of the bounding box B4in the frame C to the data processing circuit140. Accordingly, the data processing circuit140may calculate the amount of change in the reference coordinates of the bounding box B4during the first frame interval.

At this time, the sensing system300may transmit sensing data60generated by sensing data related to the driving state of the vehicle1000during the first frame interval to the data processing circuit140. For example, when the vehicle1000makes a left turn, a torque of the vehicle1000that is input from the torque sensor330may become increasingly large, and a steering angle sensor input of the vehicle1000that is input from the steering angle sensor350may become increasingly large. Furthermore, the wheel speed sensor input of the vehicle1000that is input from the wheel speed sensor340may be relatively larger on the right wheel of the vehicle than on the left wheel. Further, a lateral acceleration value of the acceleration sensor310and a yaw value of the gyroscope sensor320that are input from the acceleration sensor310and the gyroscope sensor320may become increasingly large.

In this way, the data processing circuit140may calculate the amount of change in each sensing data of the first frame interval, based on the sensing data during the first frame interval received from the sensing system300.

Next, referring toFIG.7,FIG.7is an exemplary diagram showing that the first NPU120performs the first image segmentation when the vehicle1000accelerates during the first frame interval. Images20A′,20B′, and20C′ may be images that are sequentially received from the ISP110by the first NPU120. An image20A′ may correspond to the frame A, an image20B′ may correspond to the frame B, and an image20C′ may correspond to the frame C. Hereinafter, a case in which the object B has a static speed and the first NPU120operates normally will be explained as an example.

When the vehicle1000accelerates, the object B having a static speed becomes closer to the vehicle1000and therefore becomes relatively larger within the same frame. Accordingly, the width and length of the bounding box B4′ may become increasingly large. For example, a length W4in the first direction X of the bounding box B4′ in the frame A and a length H4in the second direction Y perpendicular to the first direction X may be smaller than a length W5in the first direction X and a length H5in the second direction Y of the bounding box B4′ in the frame B. The length W5in the first direction and the length H5in the second direction Y of the bounding box B4′ in the frame B may be smaller than the length W6in the first direction X and the length H6in the second direction Y of the bounding box B4′ in the frame C.

The first NPU120may transmit the length W4in the first direction X and the length H4in the second direction Y of the bounding box B4′ in the frame A, the length W5in the first direction X and the length H5in the second direction Y of the bounding box B4′ in the frame B, and the length W6in the first direction X and the length H6in the second direction Y of the bounding box B4′ in the frame C to the data processing circuit140. Therefore, the data processing circuit140may calculate the amount of change in the length in the first direction X of the bounding box B4′ during the first frame interval and the amount of change in the length in the second direction Y of the bounding box B4′ during the first frame interval.

At this time, the sensing system300may sense data related to the driving state of the vehicle1000during the first frame interval, and transmit the generated sensing data to the data processing circuit140. For example, when the vehicle1000accelerates, the wheel speed sensor input value of the vehicle1000that is input from the wheel speed sensor340may increase on both the left wheel and right wheel of the vehicle1000. Furthermore, the longitudinal acceleration value of the acceleration sensor310and the pitch value of the gyroscope sensor320that are input from the acceleration sensor310and the gyroscope sensor320may become increasingly large or small, depending on determine whether the driving wheels of the vehicle1000are the front wheels or rear wheels.

In this way, the data processing circuit140may calculate the amount of change in each sensing data of the first frame interval based on the sensing data during the first frame interval received from the sensing system300.

FIG.8is an exemplary flowchart for explaining the operation of the vehicle control system.FIG.9is an exemplary diagram for explaining the operation of the vehicle control system ofFIG.8. The operation of the vehicle control system is described below with reference toFIGS.8and9. Reference numbers in the following description may be the same as reference numbers in the drawings referred to above.

First, referring toFIG.8, the data processing circuit140may store the first data40and the second data50received from the first NPU120and the second NPU130in the memory141(S200). For example, the data processing circuit140may store the first data40during the first frame interval and the second data50during the first frame interval in the memory141. Further, the data processing circuit140may store the sensing data60received from the sensing system300in the memory141(S201). For example, the data processing circuit140may store the sensing data60during the first frame interval in the memory141. At this time, the sensing data60may include a plurality of pieces of sensing data sensed by the plurality of sensors310,320,330,340, and350during the first frame interval.

Next, the data processing circuit140may compare whether the amount of change in the first data40and the amount of change in the second data50are equal to each other (S202). For example, when the data processing circuit140receives the first data40, the second data50, and the sensing data60during the first frame interval, the data processing circuit140may calculate each of the amount of change in the first data40during the first frame interval, the amount of change in the second data50during the first frame interval, and the amount of change in the sensing data60during the first frame interval. The data processing circuit140may compare the amount of change in the first data40during the first frame interval with the amount of change in the second data50during the first frame interval, and compare whether both the amounts of change shows a similar tendency.

If the amount of change in the first data40during the first frame interval and the amount of change in the second data50during the first frame interval show the same or similar tendency (S202—Y), the data processing circuit140may calculate a correlation coefficient R between the amount of change in the first data40during the first frame interval and the amount of change in the sensing data60during the first frame interval (S203). When the sensing data60includes a plurality of pieces of sensing data, the data processing circuit140may calculate the correlation coefficients of the multiple correlations for each item of the sensing data60.

For example, the data processing circuit140may calculate the correlation coefficient R between the amount of change in the first data40during the first frame interval and the amount of change in the data sensed from the torque sensor330during the first frame interval, and may calculate the correlation coefficient R between the amount of change in the first data40during the first frame interval and the amount of change in the data sensed from the wheel speed sensor340during the first frame interval.

Subsequently, the data processing circuit140may store items of sensing data that are determined to have a correlation with the amount of change in the first data40during the first frame interval (i.e., correlated sensing data) (S204). In this way, when the amount of change in the first data40generated by the first NPU120and the amount of change in the second data50generated by the second NPU130during a predetermined frame interval show the same or similar tendency, the data processing circuit140may determine that both the first NPU120and the second NPU130operate normally, and learn the correlation (or multiple correlations) between the amount of change in the first data40generated by the first NPU120corresponding to the main NPU during the predetermined frame interval and the amount of change in the sensing data60.

For example, when the sensing data includes a plurality of pieces of sensing data, the data processing circuit140determines that, among the plurality of pieces of sensing data, items of sensing data with a correlation coefficient R of, for example, 0.3 or more have a significant correlation with the amount of change in first data40generated by the first NPU120. The data processing circuit learns the multiple correlations between the amount of change in data generated by the first NPU120and the amount of change in sensing data and stores those items of the sensing data.

For example, when the sensing data includes data sensed from the torque sensor310and data sensed from the wheel speed sensor340, the data processing circuit140may calculate each of the correlation coefficient R between the amount of change in the first data40during the first frame interval and the amount of change in the torque sensor input during the first frame interval, and the correlation coefficient R between the amount of change in the first data40during the first frame interval and the amount of change in the input of the wheel speed sensor during the first frame interval, and store items with a correlation coefficient R of, for example, 0.3 or more among the torque sensor input and the wheel speed sensor input in the memory141.

Meanwhile, in step S200, the data processing circuit140may store the first data40and the second data50in the memory141during a second frame interval different from the previous first frame interval. Also, in step S201, the data processing circuit140may store the sensing data60during the second frame interval in the memory141. Subsequently, in step S202, it is possible to compare whether the amount of change in the first data40during the second frame interval and the amount of change in the second data50during the second frame interval are equal to each other. At this time, if the amount of change in the first data40during the second frame interval and the amount of change in the second data50during the second frame interval are significantly different from each other (S202—N), it may be determined which of the first NPU120and the second NPU130operates abnormally, based on the result of learning in steps S203and S204, and the sensing data60in the second frame interval received and stored from the sensing system300in step S201.

The data processing circuit140may calculate the first correlation coefficient R1between the amount of change in the first data40during the second frame interval and the amount of change in the correlated sensing data during the second frame interval with respect to the item of correlated sensing data stored in step S204(S205). In some implementations, if there are a plurality of items of the correlated sensing data that are determined to have a correlation with the amount of change in the first data40during the first frame interval in steps S203and S204, the data processing circuit140may calculate correlation coefficients of the multiple correlations between the amount of change in the first data40during the second frame interval and the amount of change in each of the plurality of pieces of correlated sensing data during the second frame interval.

Subsequently, the data processing circuit140may calculate the second correlation coefficient R2between the amount of change in the second data50during the second frame interval and the amount of change in the correlated sensing data during the second frame interval with respect to the item of correlated sensing data stored in step S204(S206). In some implementations, if there are a plurality of items of correlated sensing data that are determined to have a correlation with the amount of change in the second data50during the first frame interval in steps S203and S204, the data processing circuit140may calculate correlation coefficients of the multiple correlations between the amount of change in the second data50during the second frame interval and the amount of change in each of the plurality of pieces of correlated sensing data during the second frame interval.

Next, the data processing circuit140may compare the first correlation coefficient R1and the second correlation coefficient R2calculated in each of the steps S205and S206, and determine whether the first correlation coefficient R1is smaller than the second correlation coefficient R2(S207). If the first correlation coefficient R1is larger than the second correlation coefficient R2(S207—N), the data processing circuit140may determine that the first NPU120corresponding to the main NPU operates normally.

However, if the first correlation coefficient R1is smaller than the second correlation coefficient R2(S207—Y), the data processing circuit140may determine that the first NPU120corresponding to the main NPU operates abnormally, and notify the driver of the vehicle1000of this fact (S208).

Referring toFIG.9, in some implementations, if the first NPU120is determined to operate abnormally, the data processing circuit140may control the second NPU130to replace the operation of the first NPU120. At this time, unlikeFIG.2, the first NPU120may not transmit the selected data30to the controller150, and the second NPU130may transmit the second selected data70to the controller150instead of the first NPU120. At this time, the second selected data70may be all of the plurality of pieces of second data50corresponding to each of the plurality of objects generated by performing the second image segmentation on the image20by the second NPU130in step S109ofFIG.5. The controller150may control driving, startup, steering, and the like of the vehicle1000based on the second selected data70received from the second NPU130that is determined to operate normally.

FIG.10is an exemplary flowchart for explaining the operation of the vehicle control system. Reference numbers in the following description may be the same as reference numbers in the drawings referred to above.

Referring toFIG.10, the data processing circuit140may store the first data40and the second data50received from the first NPU120and the second NPU130in the memory141(S300). For example, the data processing circuit140may store the first data40during the third frame interval and the second data50during the third frame interval in the memory141.

Further, the data processing circuit140may store the first sensing data and the second sensing data received from the sensing system300in the memory141(S301). For example, the data processing circuit140may store the first sensing data during the third frame interval and the second sensing data during the third frame interval in the memory141. At this time, the first sensing data and the second sensing data may be data sensed by each of two different sensors among the plurality of sensors included in the sensing system300. In the following description, a case will be explained as an example in which the first sensing data is sensing data sensed by the torque sensor330of the sensing system300, and the second sensing data is sensing data generated by the wheel speed sensor340of the sensing system300.

Next, the data processing circuit140may compare whether the amount of change in the first data40and the amount of change in the second data50are equal to each other (S302). For example, when the data processing circuit140receives the first data40during the third frame interval and the second data50during the third frame interval, the amount of change in the first data40during the third frame interval and the amount of change in the second data50during the third frame interval may be calculated, respectively.

If the amount of change in the first data40during the third frame interval and the amount of change in the second data50during the third frame interval show significantly different tendencies from each other (S302—N), the data processing circuit140may enter a NPU failure detection mode (S303). At this time, the NPU failure detection mode may correspond to steps S205to S208ofFIG.8.

On the other hand, if the amount of change in the first data40during the third frame interval and the amount of change in the second data50during the third frame interval show the same or similar tendency to each other (S302—Y), the data processing circuit140may enter the sensor failure detection mode (S304). That is, the data processing circuit140may determine that both the first NPU120and the second NPU130operate normally, if the amount of change in the first data40during the third frame interval and the amount of change in the second data50during the third frame interval show the same or similar tendency to each other. Therefore, in this case, unlike the NPU failure detection mode of step S303, the data processing circuit140may determine whether the sensors included in the sensing system300mounted on the vehicle1000fail, rather than whether the NPUs120and130of the vehicle control system100fail.

Subsequently, the data processing circuit140may calculate a third correlation coefficient R3between the amount of change in the first data40during the third frame interval and the amount of change in the first sensing data during the third frame interval (S305). Furthermore, the data processing circuit140may calculate a fourth correlation coefficient R4between the amount of change in the first data40during the third frame interval and the amount of change in the second sensing data during the third frame interval (S306). In this way, when both the first NPU120and the second NPU130are determined to operate normally, the data processing circuit140may determine whether the sensors fail, based on the amount of change in the selected data30generated by the first NPU120corresponding to the main NPU.

Next, the data processing circuit140compares the third correlation coefficient R3with the fourth correlation coefficient R4, and may determine whether the amount of change in the first sensing data during the third frame interval and the amount of change in the second sensing data during the third frame interval show the same or similar tendency to each other (S307). At this time, if the amount of change in the first sensing data during the third frame interval and the amount of change in the second sensing data during the third frame interval show the same or similar tendency to each other (S307—Y), all sensors are determined to operate normally, and the sensor failure detection mode may be ended.

On the other hand, as a result of comparing the third correlation coefficient R3with the fourth correlation coefficient R4, when the amount of change in the first sensing data during the third frame interval and the amount of change in the second sensing data during the third frame interval are different from each other or show different tendencies (S307—N), the data processing circuit140may determine whether a difference between the third correlation coefficient R3and the fourth correlation coefficient R4is equal to or greater than a predetermined threshold value (S308). At this time, the data processing circuit140may end the sensor failure detection mode, if the difference between the third correlation coefficient R3and the fourth correlation coefficient R4is equal to or less than a predetermined threshold value.

However, if the difference between the third correlation coefficient R3and the fourth correlation coefficient R4is equal to or greater than a predetermined threshold value, the data processing circuit140determines that there is an abnormality in at least one of the sensors, and may determine which sensor operates abnormally based on the amount of change in the first data40during the third frame interval (S309).

At this time, the data processing circuit140may compare the amount of change in the first data40during the third frame interval, the amount of change in the first sensing data during the third frame interval, and the amount of change in the second sensing data during the third frame interval. For example, a case where the first data40generated by the first NPU120is a reference coordinate of the bounding box will be explained. When the X value of the reference coordinate of the bounding box becomes increasingly large during the third frame interval, the data processing circuit140may determine that vehicle1000is turning left.

At this time, when the steering angle of the vehicle1000that is input from the steering angle sensor350corresponding to the first sensing data becomes increasingly large during the third frame interval, and the wheel speed sensor input of the vehicle1000that is input from the wheel speed sensor340corresponding to the second sensing data during the third frame interval becomes relatively larger on the left wheel of the vehicle1000than the right wheel, the data processing circuit140may determine that the wheel speed sensor340is abnormal.

That is, the data processing circuit140may compare the amount of change in the first data40, the amount of change in the first sensing data, and the amount of change in the second sensing data during the third frame interval, and detect a sensor that shows a different tendency from the remaining outputs. Thereafter, the data processing circuit140may notify the driver of the vehicle1000of the abnormally operating sensor (S310).

FIG.11is a diagram of a vehicle including the vehicle control system.

Referring toFIG.11, a vehicle500may include a plurality of electronic control units (ECU)510. Each electronic control unit of the plurality of electronic control units510is electrically, mechanically, and communicatively connected to at least one of the plurality of devices provided in the vehicle500, and may control the operation of at least one device based on any one function execution command. The vehicle500may correspond to a vehicle1000ofFIG.3, and the vehicle control system100ofFIGS.1and2may be included in the electronic control unit510.

Here, the plurality of devices may include a storage device520, an image sensor530that acquires an image required to perform at least one function, a driving unit540that performs at least one function, and a sensing system570including at least one sensor.

For example, the image sensor530may correspond to an automotive image sensor including a unit pixel, and the image sensor530may be included in the camera200ofFIG.2.

The driving unit540may include a fan and a compressor of an air conditioner, a fan of a ventilation device, an engine and a motor of a power device, a motor of a steering device, a motor and a valve of a brake device, an opening/closing device of a door or a tailgate, and the like.

The sensing system570may include at least one sensor that senses data about the driving states of the vehicle500, and may correspond to the sensing system300ofFIG.2.

The plurality of electronic control units510may communicate with the storage device520, the image sensor530, the driving unit540, the input unit550, the CCU560, and the sensing system570, using at least one of, for example, an Ethernet, a low voltage differential signaling (LVDS) communication, and a LIN (Local Interconnect Network) communication.

The plurality of electronic control units510determine whether there is a need to perform the function based on the information acquired from the sensing system530, and when it is determined that there is a need to perform the function, the plurality of electronic control units510control the operation of the driving unit540that performs that function, and may control an amount of operation based on the acquired information.

The plurality of electronic control units510are able to control the operation of the driving unit540that performs that function based on the function execution command that is input through the input unit550, and are also able to check the setting amount corresponding to the information that is input through the input unit550and control the operation of the driving unit540that performs that function based on the checked setting amount.

Each electronic control unit510may control any one function independently, or may control any one function in cooperation with other electronic control units.

For example, the data processing circuit140(shown inFIG.1) in the electronic control unit510may output a warning sound for an abnormal operation through a speaker, when there is an abnormality in the first NPU120(shown inFIG.1) corresponding to the main NPU, or when there is an abnormality in any one of the sensors included in the sensing system570.

An electronic control unit of an autonomous driving control device may receive navigation information, road image information, and distance information to obstacles in cooperation with the electronic control unit of the vehicle terminal, the electronic control unit of the image acquisition unit, and the electronic control unit of the collision prevention device, and control the power device, the brake device, and the steering device using the received information, thereby performing the autonomous driving.

A connectivity control unit (CCU)560is electrically, mechanically, and communicatively connected to each of the plurality of electronic control units510, and communicates with each of the plurality of electronic control units510.

That is, the connectivity control unit560is able to directly communicate with a plurality of electronic control units510provided inside the vehicle, is able to communicate with an external server, and is also able to communicate with an external terminal through an interface.

Here, the connectivity control unit560may communicate with the plurality of electronic control units510, and may communicate with the server610, using an antenna (not shown) and a RF communication.

Further, the connectivity control unit560may communicate with the server610by a wireless communication. At this time, the wireless communication between the connectivity control unit560and the server610may be performed through various wireless communication methods such as a GSM (Global System for Mobile communication), a CDMA (Code Division Multiple Access), a WCDMA (Wideband Code Division Multiple Access), a UMTS (Universal Mobile Telecommunications System), a TDMA (Time Division Multiple Access), and an LTE (Long Term Evolution), in addition to a Wifi module and a Wireless broadband module.

While the present disclosure has been particularly illustrated and described with reference to exemplary implementations thereof, it will be understood by those of ordinary skill in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims. The exemplary implementations should be considered in a descriptive sense only and not for purposes of limitation.