Patent Description:
In conventional vehicle damage assessment approaches, a vehicle insurance company can send a professional (such as an insurance adjuster) to conduct a manual on-site survey and determine a repair plan for a damaged vehicle. After the professional captures images of the damaged vehicle, these "captured images" can be compared against similar images in a library or database of images. The library can include images of damaged vehicle parts, which were previously captured from other images of other damaged vehicles ("library images"). In a conventional vehicle damage assessment approach, a system can determine the damage assessment result of the damaged vehicle based on the comparison of the captured images with the library images.

However, conducting a manual survey and determining a repair plan based on the comparison of the captured images against library images can result in a long processing time, and can incur a significant cost in manpower and training of professionals. Current approaches can use image-based artificial intelligence and machine learning in vehicle damages assessment approaches that may reduce both processing time and labor costs. For example, these approaches can enable automatic identification of damaged parts and the degree of the damage based on pictures of a vehicle taken by users at the site of an accident or other incident. In addition, the on-site pictures of the vehicle can be used by computer-vision image identification technology with artificial intelligence to provide a repair solution. Therefore, by automating the survey and damage assessment approaches, the labor costs incurred by a vehicle insurance company can be reduced and the vehicle insurance claiming experience of a user can be improved.

While using the automated survey and damage assessment approach can result in some benefits, some challenges still remain in providing accurate identification or recognition of vehicle parts and the degree of damage of those parts.

<CIT> in an abstract states that "A system and method are provided for automatically estimating a repair cost for a vehicle. A method includes: receiving, at a server computing device over an electronic network, one or more images of a damaged vehicle from a client computing device; performing image processing operations on each of the one or more images to detect external damage to a first set of parts of the vehicle; inferring internal damage to a second set of parts of the vehicle based on the detected external damage; and, calculating an estimated repair cost for the vehicle based on the detected external damage and inferred internal damage based on accessing a parts database that includes repair and labor costs for each part in the first and second sets of parts.

One embodiment facilitates a computer system for recognizing parts of a vehicle. During operation, the system generates a convolution feature map of an image of the vehicle. The system determines, based on the convolution feature map, one or more proposed regions, wherein a respective proposed region corresponds to a target of a respective vehicle part. The system determines a class and a bounding box of a first vehicle part corresponding to a first proposed region based on a feature of the first proposed region. The system optimizes classes and bounding boxes of the vehicle parts based on correlated features of the corresponding proposed regions. The system generates a result which indicates a list including an insurance claim item and corresponding damages based on the optimized classes and bounding boxes of the vehicle parts.

In some embodiments, determining the one or more proposed regions is performed by a fully convolutional network. Determining the one or more proposed regions comprises: performing, for a plurality of convolutional mapping positions, a convolutional operation for a respective convolutional mapping position with a sliding window in the convolution feature map; obtaining a feature vector of the respective convolutional mapping position; predicting whether the respective convolutional mapping position comprises a foreground target with respect to a plurality of predetermined anchors based on the feature vector of the respective convolutional mapping position; and predicting a border of a proposed region in the respective convolutional mapping position that corresponds to each anchor.

In some embodiments, the correlated features of the corresponding proposed regions include one or more of: a size of a respective proposed region; a position relationship between the proposed regions; a distance between the proposed regions; and an intersection-over-union ratio of the proposed regions.

In some embodiments, prior to optimizing the classes and the bounding boxes of the vehicle parts based on the correlated features of the corresponding proposed regions: the system obtains the classes and the bounding boxes of the vehicle parts corresponding to the proposed regions; and the system extracts the correlated features of the proposed regions.

In some embodiments, the system determines an energy function and a corresponding probability function of the conditional random field. The system solves the energy function while minimizing the probability function. The energy function comprises: a data term which is based on a probability of each proposed region belonging to each class; and a smoothing term which is based on the correlated features of the proposed regions.

In some embodiments, optimizing the classes and bounding boxes of the vehicle parts is performed by a recurrent neural network. The system solves the energy function while minimizing the probability function by performing multiple iterative operations through the recurrent neural network to obtain an approximation of the probability function. An iterative operation comprises updating the probability of each proposed region belonging to each class based on a pre-trained compatibility matrix. The pre-trained compatibility matrix indicates a probability of compatibility between the classes of the vehicle.

In some embodiments, parameters used to perform the method are jointly trained from end to end based on training samples.

In some embodiments, the parameters which are jointly trained from end to end are trained by the following operations. The system inputs the training samples into a convolution module, wherein the training samples indicate corresponding classes and bounding boxes. The system obtains a prediction result based on the classes and the bounding boxes optimized by the conditional random field module, wherein the prediction result comprises predicted classes and predicted bounding boxes of a plurality of target regions. The system determines a prediction error for the target regions based on the prediction result and the corresponding classes and bounding boxes indicated by the training samples. The system determines a loss function based on the prediction error, wherein the loss function comprises a cross term of the prediction error for the target regions. The system counter-propagates the prediction error based on the loss function by counter-propagating the prediction error of a first target region to a set of target regions correlated with the first target region.

One embodiment facilitates a computer system for recognizing parts of a vehicle. The system comprises a convolution module, a region proposal module, a classification module, a conditional random field module, and a reporting module. The convolution module is configured to generate a convolution feature map of an image of the vehicle. The region proposal module is configured to determine, based on the convolution feature map, one or more proposed regions, wherein a respective proposed region corresponds to a target of a respective vehicle part. The classification module is configured to determine a class and a bounding box of a first vehicle part corresponding to a first proposed region based on a feature of the first proposed region. The conditional random field module is configured to: optimize classes and bounding boxes of the vehicle parts based on correlated features of the corresponding proposed regions; and output on a display screen the image with annotations indicating the optimized classes and bounding boxes of the vehicle parts. The reporting module is configured to generate, based on the image with annotations, a report which indicates a degree of damage to a respective vehicle part.

In the figures, like reference numerals refer to the same figure elements.

The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements.

The embodiments described herein address the challenges of accurately detecting and identifying vehicle parts by training neural network to recognize vehicle parts and by using a target detection algorithm with a conditional random field (CRF). This allows information flow between various proposed target regions of a single vehicle image, which can result in a more accurate classification of a plurality of parts in the single vehicle image.

As described above, sending a professional to capture on-site images of a damaged vehicle and to determine a repair plan may involve subsequently comparing the captured images with "library images. " The damage assessment result can be based on the comparison of the captured images with the library images. However, conducting a manual survey and determining a repair plan based on the comparison of the captured images against library images can result in a long processing time, and can incur a significant cost in manpower and training of professionals. Current approaches can use image-based artificial intelligence and machine learning in vehicle damages assessment approaches that may reduce both processing time and labor costs. For example, these approaches can enable automatic identification of damaged parts and the degree of the damage based on pictures of the vehicle taken by users at the site of an accident or other incident. In addition, the on-site pictures of the vehicle can be used by computer-vision image identification technology with artificial intelligence to provide a repair solution. Therefore, by automating the survey and damage assessment approaches, the labor costs incurred by a vehicle insurance company can be reduced and the vehicle insurance claiming experience of a user can be improved.

The embodiments described herein address these challenges by providing a system which can be trained to recognize vehicle parts. The system can be a neural network which is trained to recognize a plurality of vehicle parts based on a single vehicle image, such that the system can perform simultaneous detection of multiple vehicle parts (i.e., targets). The system can be trained by using a sample training set of labeled or annotated vehicle images, including images of damaged vehicles and images of undamaged vehicles. The labels or annotations can include both a class of a given part in a vehicle image and a bounding box region where the given part is located.

Once the system has been trained with the sample training set, the system may be considered a "vehicle parts recognition model. " Using a target detection algorithm, the trained neural network (or the vehicle parts recognition model) can also use a conditional random field (CRF) to allow information flow between the proposed regions, which results in an improved and more accurate classification of the plurality of vehicle parts. Thus, the system can jointly determine the class of a given vehicle part in a proposed region by combining the correlations between the various proposed regions, which results in optimizing the accuracy of the result of recognizing the vehicle part.

The system can receive as input captured images of vehicles, e.g., images captured onsite by a user on a mobile device with camera function at the scene of an accident, and output a vehicle image with annotations indicating the classes and bounding boxes of multiple vehicle parts in the vehicle image. An exemplary neural network for facilitating recognition of vehicle parts is described below in relation to <FIG> and <FIG>.

Furthermore, based on the annotated image output by the system and based on other annotated images of other vehicle parts, the system can generate a report which indicates a degree of damage to a specific vehicle part, and, generally, a degree of damage to the vehicle. This can result in an improved damage assessment approach which reduces the labor costs incurred by a vehicle insurance company and also increases the efficiency of the user's experience in reporting the damages, e.g., in a claims report or subsequent to an accident. Moreover, the embodiments described herein can provide an improved and more accurate method of damage assessment by clearly labeling (annotating) the captured image with the class and bounding boxes, where the annotations of the class and bounding boxes have been optimized based on the trained neural network model. This can result in a more efficient and accurate understanding of the degree of damages, which can result in an improved vehicle loss assessment report.

The terms "region proposal" and "proposed region" are used interchangeably in this disclosure, and refer to a potential region in which a target part may be detected and located (i.e., recognized).

The terms "module," "component," "layer," and "network layer" are used interchangeably in this disclosure and refer to a structure or unit which is configured to perform the methods described herein.

The terms "neural network," "neural network system," "parts recognition neural network," and "parts recognition neutral network model" are used interchangeably in this disclosure and refer to a model which can be trained using sample images (e.g., supervised data, labeled data, annotated data with class information and bounding boxes, etc).

The terms "parts" and "vehicle parts" are used interchangeably in this disclosure and refer to one or more parts of a vehicle.

The terms "class" or "parts class" are used interchangeably in this disclosure and refer to a category or classification for a part of a vehicle, as determined by the neural network system described herein.

<FIG> illustrates an exemplary environment for facilitating recognition of vehicle parts, in accordance with an embodiment of the present application. Environment <NUM> can include: a vehicle <NUM>; a user <NUM> associated with a computing device <NUM>; a server <NUM>; a server <NUM>; and a server <NUM> associated with a display <NUM> and a user <NUM> (not shown). Computing device <NUM> and servers <NUM>-<NUM> can communicate with each other over a network <NUM>. Server <NUM> can include a parts recognition neural network <NUM> (or a model <NUM>), which can include a parts target detection network <NUM> and includes a conditional random field (CRF) module <NUM>). Parts recognition neural network <NUM> is described below in relation to <FIG>.

During operation, server <NUM> can send training data <NUM> to server <NUM> (as training data <NUM>). Training data <NUM> can include labeled or annotated vehicle images, including images of damaged vehicles and images of undamaged vehicles. The labels or annotations can include both a class of a given part in a vehicle image and a marked bounding box region in which the given part is located. Server <NUM> can receive training data <NUM> and can train parts recognition neural network <NUM> (function <NUM>). Upon training model <NUM>, server <NUM> can receive images captured on site (such as a captured image <NUM>) and automatically detect and recognize vehicle parts, which, when combined with damage information, can be used to efficiently generate a vehicle loss assessment report.

For example, computing device <NUM> can include a camera which can capture images of, e.g., a vehicle <NUM> which is a damaged vehicle on-site after an accident. Using computing device <NUM>, user <NUM> can take a picture of vehicle <NUM>, and user <NUM> (via computing device <NUM>) can send a captured image <NUM> (of vehicle <NUM>) to server <NUM> as a captured image <NUM>. Upon obtaining captured image <NUM>, server <NUM> can convolve and process captured image <NUM>, and can generate a convolution feature map (function <NUM>). Server <NUM> can determine, based on the convolution feature map, one or more proposed regions, where a proposed region corresponds to a target of a vehicle part (function <NUM>). Server <NUM> can determine and optimize classes and bounding boxes for the vehicle parts (function <NUM>). Subsequently, server <NUM> can output on a display screen the image with annotations (e.g., annotated images <NUM> on display <NUM> associated with server <NUM>). The image with annotations can indicate the optimized classes and bounding boxes of the vehicle parts (function <NUM>). Server <NUM> can generate, based on the image with annotations, a report which indicates a degree of damage to a respective vehicle part (e.g., a vehicle loss assessment report) (function <NUM>). The generated report can indicate a degree of damage to the detected and identified vehicle parts, and can also provide a repair plan for the damaged vehicle parts. The generated report can include, e.g., information about the detected and identified parts and a repair plan for fixing those parts.

Server <NUM> can send an annotated image <NUM> and a report <NUM> back to computing device <NUM> (as an annotated image <NUM> and a report <NUM>), which can be used to display various items and images on a display screen of computing device <NUM>, e.g., based on specific application scenarios.

In addition, server <NUM> can send a request <NUM> to server <NUM>. Request <NUM> can be, e.g.: a request for all vehicle loss assessment reports generated within a predetermined period of time; a request for a vehicle loss assessment report or a repair plan specific to a particular user, time, or time interval; a request for information about parts relating to a particular vehicle or user or captured image. Server <NUM> can receive request <NUM> as a request <NUM>, and, subsequent to performing functions <NUM>-<NUM> (as described above), server <NUM> can send annotated image <NUM> and report <NUM> back to server <NUM> (as an annotated image <NUM> and a report <NUM>). Upon receiving annotated image <NUM> and report <NUM>, server <NUM> can display, on its display <NUM>, annotated image <NUM> of previously captured image <NUM>. Annotated image <NUM> can indicate the optimized classes and bounding boxes of a plurality of vehicle parts of the vehicle in captured image <NUM>. As described above, computing device <NUM> can also display, on its respective display screen, annotated image <NUM> and report <NUM>.

Server <NUM> may also receive annotated image <NUM> and report <NUM> automatically, e.g., at a predetermined time or time interval and/or without first sending request <NUM>. A user <NUM> (not shown) associated with display <NUM> can view this information on display <NUM>, and can use the information to conduct further actions, e.g., based on the received information being integrated into a damage assessment application or program (not shown) running on server <NUM> or <NUM>.

<FIG> illustrates an exemplary structure or neural network system <NUM> for facilitating recognition of vehicle parts, in accordance with an embodiment of the present application. System <NUM> can include a convolution module <NUM>, a region proposal module <NUM>, a classification module <NUM>, and includes a conditional random field (CRF) module <NUM>. Each module can be a network layer or a component of the overall neural network system. During operation, convolution module <NUM> can receive a vehicle image <NUM>. Convolution module <NUM> can convolve and process vehicle image <NUM>, and generate a convolution feature map <NUM> of vehicle image <NUM> (via a communication <NUM>).

- Convolution module: Convolution module <NUM> can be a Convolutional Neural Network (CNN), which is a network structure frequently adopted in image processing. A CNN can include a plurality of convolution layers, which are used to convolve and process images. A convolution core may be utilized to perform a series of operations with respect to each pixel in an image. The convolution core can be based on a matrix (e.g., a pixel matrix of the image), and can be a grid structure of a square shape (e.g., a 3x3 matrix or pixel region) where each grid can have a weighted value. When performing a convolution calculation with respect to an image, the convolution core can slide on the pixel matrix of the image. For each step of the slide, the system can perform a multiplication and summation for each element in the convolution core and the corresponding pixel value. This can result in a new feature value matrix, which can be used to form the convolution feature map (e.g., convolution feature map <NUM>), which can be transmitted to a classification module <NUM> via communications <NUM> and <NUM>.

The convolution calculation can also extract the abstract features from the pixel matrix of the original image. Based on the design of the convolution core, these abstract features may reflect the overall features of the image, e.g., the line shape, the distribution of color, any patterns of shading, etc. As described above, convolution module <NUM> can include one or more convolution layers, where each convolution layer can perform a convolution processing once with respect to an image. After the convolution layers have successfully completed performing their respective convolution processing operations, the system can obtain convolution feature map <NUM> which corresponds to vehicle image <NUM>.

In one example, convolution module <NUM> can include a plurality of convolution layers, and can further include at least one Rectified Linear Unit (ReLU) activating layer between the convolution layers or after certain convolution layers, for non-linear mapping of the output result of the convolution layers. The result from the non-linear mapping may be input into the next convolution layer to continue the convolution operation, or may be output as convolution feature map <NUM>.

In another example, convolution module <NUM> can include a plurality of convolution layers, and can further include a pooling layer between the convolution layers, for a pooling operation on the output result of the convolution layers. The result from the pooling operation may be input into the next convolution layer to continue the convolution operation.

- Region proposal module: Region proposal module <NUM> can receive convolution feature map <NUM> (via a communication <NUM>). Region proposal module <NUM> can determine, based on convolution feature map <NUM>, one or more proposed regions, wherein a respective proposed region corresponds to a target of a respective vehicle part. That is, a proposed region is the region where the target in the image could possibly appear. The proposed region may also be referred to as the region of interest (ROI). The determination of the proposed regions can provide a basis for the subsequent classification of the targets and the regression of the bounding boxes. Exemplary proposed regions A, B, and C are indicated on convolution feature map <NUM> (via a communication <NUM>).

Region proposal module <NUM> can be a fully convolutional network, such as a region Convolutional Neural Network (R-CNN) and a Fast R-CNN. In order to extract or determine the proposed regions, the system can adopt a selective search method in both the R-CNN model and the Fast R-CNN model. The advanced Faster R-CNN model can also be used for generating or suggesting the proposed regions. As a fully convolutional network, region proposal module <NUM> can effectively realize the suggestion and generation of the proposed regions based on convolution feature map <NUM> as returned by a basic network (i.e., including the convolution module <NUM>) and through a full convolution. An exemplary region proposal module is described below in relation to <FIG>.

- Classification module: Classification module <NUM> can receive convolution feature map <NUM> and the generated results from region proposal module <NUM> based on the convolution feature map <NUM>, e.g., as input to classification module <NUM> (via a communication <NUM>). Classification module <NUM> can determine the class and the bounding box of a respective vehicle part of the image, for each proposed region and based on a feature of the proposed region itself.

Classification module <NUM> can be a fully connected layer, which carries out the classification of the part class and bounding box regression based on the feature of the region for each region input from the previous layer. Specifically, classification module <NUM> can include a plurality of classifiers, where each classifier is trained to recognize the different classes of targets in the proposed regions. In detecting vehicle parts in vehicle image <NUM>, each classifier can be trained to recognize vehicle parts of different classes, e.g., a bumper, a front door of the vehicle, an engine hood, a headlight, rear lights, etc. Classification module <NUM> can also include a regressor, which can be used for regressing the bounding box corresponding to the recognized target in order to determine the smallest rectangular region surrounding the recognized target as the bounding box.

Convolution module <NUM>, region proposal module <NUM>, and classification module <NUM> form the main network structure of the Faster R-CNN. Through this network structure, the system may carry out a preliminary recognition for the input vehicle images (e.g., vehicle image <NUM>), which results in labeling the class of the part and the bounding box of the region where the part is located.

As described herein, classification module <NUM> can determine the class and bounding box of the vehicle part in a proposed region based on the feature of each proposed region itself. In other words, each proposed region is separately considered and independently processed. For example, given proposed region A as illustrated convolution feature map <NUM> in <FIG>, classification module <NUM> can extract the feature of proposed region A, and can determine whether the target in proposed region A is a certain pre-trained part class based on the extracted feature. If it is, the system can output the class label and bounding box of the target in region A as the predicted result. While performing this determination (e.g., parts detection and recognition) for proposed region A, the system considers only the feature of proposed region A, and does not consider the other proposed regions. Thus, the system can output this preliminary recognition result of the vehicle part through the Faster R-CNN network structure.

Furthermore, a positional relationship between the parts can determine the mutual constraint relations between the classes for the parts. By adding a conditional random field (CRF) module to the neural network of system <NUM>, the system can rectify and optimize the preliminary recognition result obtained by classification module <NUM>, e.g., by capturing and processing the associated features between the proposed regions, thereby further increasing the accuracy of detecting the vehicle parts. The CRF module is described below in further detail.

Conditional random field (CRF) module <NUM> receives both class and bounding box <NUM> from classification module <NUM> and can extract correlated features <NUM> from convolution feature map <NUM>. Correlated features <NUM> include the associated relations of the plurality of proposed regions from convolution feature map <NUM>. This allows conditional random field module <NUM> to establish a random field between the parts, which allows the information between the various proposed regions to mutually flow. The class of a given proposed region may be jointly determined based on the features of the surrounding proposed regions, which can result in rectifying and optimizing the part class result of each proposed region.

Specifically, the conditional random field (CRF) can be a probability model of an undirected graph, with the vertex representing a variable, and the edge between the vertexes representing the dependent relationship between two variables. An overall normalization may be carried out for all the variable features to obtain an overall optimization.

In the image processing field, the CRF may be used for semantic segmentation of the images. An image may be considered as a set of pixels, and a segmentation of the image can be used to determine the class label to which each pixel belongs. The image may be mapped to an undirected graph, with each vertex of the image model corresponding to one pixel. In performing segmentation of the image, a hidden variable Xi may be defined as the classification label of pixel i, and its value range can the semantic label L= {l1, l2, l3. }to be classified. Yi may be defined as an observed value of each random variable Xi, which is the color value of each pixel. An energy function E(x) and a corresponding probability function P of the CRF are defined based on these terms.

The energy function of image segmentation can include a data term and a smoothing term. The data term can be based on the probability of each pixel belonging to each class, and the smoothing term can be based on the energy between the pixel pairs, e.g., the difference of grey scale values and the spatial distance. The purpose of image semantic segmentation of a conditional random field is to minimize the energy function E(x), which corresponds to the maximization of posterior probability function P. The class label obtained for each pixel at this point can correspond to the optimum segmentation result.

In the embodiments described herein, by applying CRF in image segmentation, a system for detecting and recognizing vehicle parts is optimized by using the conditional random field (CRF) module or component. The essential concept is that when performing image segmentation, the CRF component may obtain a more accurate pixel level segmentation (i.e., which pixel belongs to which class) by: capturing the positional relationship and correlation between the pixels (represented by the presence of the smoothing term in the energy function); and utilizing certain information around the pixel.

Part detection and image segmentation differ in that, for part segmentation the system must determine which pixel belongs to which part, while for part detection the system need only determine which region belongs to which part, without the need to know the class for each pixel. Thus, by applying the method of image segmentation, the system can capture the correlation between the detection boxes of the region proposals, and can also utilize the features of the surrounding proposed regions, which can result in more accurately detecting and determining the class of the parts in each of the proposed regions.

Thus, when using the CRF component to optimize detection of vehicle parts, the CRF energy function is defined as described above, e.g., including a data term and smoothing term, where the data term is based on the probability of each proposed region belonging to each part class (e.g., from class and bounding box information <NUM>), and where the smoothing term is based on the correlated features between the various proposed regions (e.g., based on correlated features <NUM>). The system can determine the probability function of the conditional random field based on the energy function, and can obtain the probability of each proposed region of the energy function belonging to each part class, when the probability function is minimized.

As an example, the CRF component can realize an update of the parts recognition result output from the classification regression layer of the Faster R-CNN, for the pre-established model, and by solving the above-noted probability function and energy function.

As another example, the CRF component can be realized through a recurrent neural network (RNN), i.e., a CRF component as a RNN component. Thus, the CRF component can become a learnable, trainable network component, which may be directly embedded into an already known network model.

Specifically, in the CRF as RNN model, the energy function E(x) may be defined as:
<MAT>
wherein
<MAT>
is the data term which represents the probability of each region proposal xi belonging to the corresponding part class, and wherein
<MAT>
is a smoothing term which represents the correlated feature between the region proposals xi and xj. More specifically, the smoothing term may be represented as the sum of a plurality of Gaussian functions.

In one example, the correlated features between the various region proposals can include one or more of: the size of the proposed region; a positional relationship between the various proposed regions; a distance between the various proposed regions; and an intersection-over-union (IoU) ratio between the various proposed regions. The system can determine the above noted smoothing term based on these correlated features.

Based on the above noted energy function of Equation (<NUM>), the probability function of the conditional random field (CRF) may be determined as:
<MAT>
The minimization with respect to the energy function E(x) in Equation (<NUM>) can correspond to the maximization of posterior probability function P(x), so as to obtain an optimum parts detection result.

Because it can be difficult to directly calculate the probability function P(x), the system can obtain an approximation of P(x) through a probability function Q(x), which can be more convenient to calculate. <MAT>
Q(X) may be allowed to be an approximation of P(x) to the largest extent, through the iterative computation method.

Due to the time sequence characteristics and memory characteristics of the recurrent neural network (RNN), an iteration calculation process may be realized through the network layer in the RNN. The iteration calculation can include multiple iteration operations. Each iteration operation can include a message delivery, a filter weight output, a conversion of class compatibility, an addition of a data term, and a normalization of the probability.

Specifically, under the circumstance when the CRF as RNN component is applied in the vehicle parts recognition, in the step of converting the class compatibility, the system can update the probability of each proposed region belonging to each class via a compatibility matrix. The compatibility matrix may indicate the probability of the compatibility between various vehicle part classes. For example, if part A is the door handle, and if the adjacent part is a vehicle door, then the compatibility probability between the two corresponds to a higher value. However, if part A is the door handle, and if the adjacent part is a bumper, then the compatibility probability between the two corresponds instead to a low value. The compatibility probability value in the compatibility matrix may be obtained through pre-training. That is, the positional relationship and compatibility relationship between various parts in the same vehicle may be learned through training with a large number of images with marked vehicle parts, and the learned positional relationship may be represented by the compatibility probability value in the compatibility matrix.

Thus, the system can perform a continuous iteration operation for approximation of the probability function P(x), and can determine the probability of each proposed region belonging to each corresponding part when the probability function P(x) is maximized. As a result, the system can optimize the result of the parts detection and recognition.

In summary, the system: establishes a conditional random field (via the CRF component) on the convolution feature map; extract the correlated features between various proposed regions; allow the flow of energy and feature information between different proposed regions; and jointly determine the part class of a certain proposed region based on a plurality of proposed regions. Thus, this can result in optimizing the result of detecting and recognizing vehicle parts based on a single proposed region independently in the previous network layer, which in turn can improve the accuracy of parts recognition.

<FIG> illustrates an exemplary structure of a region proposal module or network <NUM> for facilitating recognition of vehicle parts, in accordance with an embodiment of the present. application. Region proposal module <NUM> can correspond to region proposal module <NUM> of <FIG>. Region proposal module <NUM> can include a convolution processing module <NUM>, a bounding box classification module <NUM>, and a bounding box prediction module <NUM>. Convolution processing module <NUM> can be configured to perform convolutional operator processing with respect to each convolutional mapping position with a sliding window in the convolution feature map output from the previous convolution layer. Convolution processing module <NUM> can further be configured to obtain a feature vector of each convolution mapping position.

- Convolution processing module: In convolution processing module <NUM>, a small network (similar to the convolution core) can be used to perform slide scanning on the convolution feature map output from the previous convolution layer (e.g., convolution feature map <NUM>). This sliding window <NUM> can be in full contact with a box of a certain size on the feature map each time (similar to the convolution operator), and can then be mapped to a reduced dimension (i.e., as the feature vector of the center position of this box).

Region proposal module <NUM> can use the concept of an "anchor. " As described herein, the purpose of performing scanning by using the sliding window to slide over each position is to determine whether a target is present in the receptive field corresponding to the center of each sliding window. As the sizes and length / width ratios of the targets are different, boxes of many different sizes may be required. The anchor can provide the size of a standard box, and the system can obtain boxes of different sizes based on the multiplied times (sizes) and the length / width ratios. One example can be seen in setting the size of the standard box as <NUM>, and using boxes of three multiplied times (<NUM>, <NUM> and <NUM>) and based on three ratios (<NUM>, <NUM> and <NUM>). This can result in a total of nine anchors of different sizes (such as anchors <NUM>), as described below in relation to <FIG>.

<FIG> illustrates a diagram of exemplary anchors <NUM> in nine sizes, in accordance with an embodiment of the present application. Note that in order to clearly illustrate the anchors of various sizes, the centers of the anchors depicted in <FIG> do not correspond to the same position.

Each anchor (e.g., <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>) can correspond to a box of a different size in the original image, while the sliding window for performing the convolution calculation can act upon the convolution feature map. Each convolution mapping position (depending on the times of the convolution processing and the size of the convolution core in each processing) of the convolution feature map can correspond to a bigger region in the original image. For example, an entire region <NUM> may act upon the pixel region of the original image corresponding to one sliding window on the convolution feature map, and the anchors can be the boxes of various sizes (e.g., <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>) marked with bounding boxes that adopt the position on the original image corresponding to the center point of the sliding window (convolution core) as the center.

The system can perform processing with the sliding window by sliding across each position to obtain the feature vector corresponding to each position. The system can input the feature vector into bounding box classification module <NUM> and bounding box prediction module <NUM> (via, respectively, communications <NUM> and <NUM>).

- Bounding box classification module: Returning to <FIG>, bounding box classification module <NUM> can be configured to predict whether each position constitutes or includes a foreground target with respect to the various predetermined anchor points based on the feature vector of each convolutional mapping position. Bounding box classification module <NUM> can output an objectness score of a certain anchor with respect to each convolutional mapping position, where the objectness score can indicate the probability of the current anchor being the foreground target with respect to a given convolutional mapping position. The objectness score may be used for distinguishing whether the given convolutional mapping position is or is not the target, rather than as a classification of the target itself.

- Bounding box prediction module: Bounding box prediction module <NUM> can be configured to predict the border of the region in each position that corresponds to each anchor. Bounding box prediction module <NUM> can output the regression borders of a plurality of proposed regions of various sizes and length / width ratios at each given convolutional mapping position. Therefore, in <FIG>, given nine anchors of different sizes, for each position, bounding box prediction module <NUM> can output nine regression borders.

By combining the results of both bounding box classification module <NUM> and bounding box prediction module <NUM>, region proposal module <NUM> of <FIG> (i.e., region proposal module <NUM> of <FIG>) can directly generate a plurality of proposed regions with the vehicle parts as the potential targets based on convolution feature map <NUM>.

As described above in relation to <FIG>, the neural network system for parts detection and recognition is jointly formed by convolution module <NUM>, region proposal module <NUM>, and classification module <NUM>, as well as conditional random field (CRF) module <NUM>. These modules can each be a network portion, such that the overall neural network system can include a plurality of network portions. A network portion, e.g., the CRF module, can act as a sub-network of the overall neural network system. Furthermore, a network portion may be trained independently to determine the parameters for the model.

The system can jointly train the multiple network portions in the neural network system, e.g., by an end to end training with respect to the entire neural network system. Specifically, the joint training from end to end may be carried out through the following method. Training samples can be input into the entire neural network system. A training sample may be an image containing the vehicle parts, where the vehicle parts may have labeled classes and bounding boxes as the ground truth. Convolution module <NUM> can perform convolution processing on the training sample to obtain the convolution feature map. Region proposal module <NUM> can determine one or more proposed regions based on the convolution feature map. For all the proposed regions, based on a feature of a respective proposed region, classification module <NUM> can process the respective proposed region to determine a preliminary class and a preliminary bounding box of a vehicle part corresponding to the respective proposed region. Subsequently, conditional random field (CRF) module <NUM> can optimize the preliminary result and output the prediction result of the entire network, where the prediction result includes the predicted part classes and predicted bounding boxes of a plurality of target regions.

The system can determine the prediction error of the various target regions based on the predicted result of the network and the ground truth of the training samples (e.g., the part class label and the bounding box label). The system can determine the loss function based on the prediction error. In one example, the loss function includes an intersection term of the prediction error of a plurality of target regions. This corresponds to the prediction result of a certain proposed region, which prediction result is determined based on the correlated features (after establishing correlations between a plurality of proposed regions in the CRF module).

The system can counter-propagate the prediction error based on the loss function, so as to adjust and determine the parameters of the network in the neural network system. Because the loss function can include the intersection term of the prediction error of a plurality of target regions, when a gradient counter-propagation of the error is carried out, the system can counter-propagate the prediction error of a certain target region to the other target regions related to this target region, thereby optimizing the network parameters relevant to the computing of the correlated features of the proposed regions.

Thus, training the system in this manner can result in a deep learning neural network system. This neural network system may simultaneously detect and recognize a plurality of vehicle parts contained in the image, based on a single vehicle image. By using the conditional random field (CRF) and by performing the detection of the parts based on the correlated features between the various proposed regions, the system can take into account the constraint positional relationship between the unique parts of the vehicle in the detection process, which can result in a more accurate detection result.

<FIG> illustrates an exemplary method <NUM> for facilitating recognition of vehicle parts, in accordance with an embodiment of the present application. During operation, the system obtains an image of a vehicle (operation <NUM>). The image can be captured by a user of a mobile device with a camera, and the image can be of a damage vehicle at the site of an accident. The system generates a convolution feature map of the image of the vehicle (operation <NUM>). The system determines, based on the convolution feature map, one or more proposed regions, wherein a respective proposed region corresponds to a target of a respective vehicle part (operation <NUM>). The system determines a class and a bounding box of a first vehicle part corresponding to a first proposed region based on a feature of the first proposed region (operation <NUM>). The system optimizes classes and bounding boxes of the vehicle parts based on correlated features of the corresponding proposed regions (operation <NUM>). The system generates a result which indicates a list including an insurance claim item and corresponding damages based on the optimized classes and bounding boxes of the vehicle parts (operation <NUM>). The system can store the generated result, and use the generated result to produce, e.g., reports. In some embodiments, the system can output on a display screen the image with annotations indicating the optimized classes and bounding boxes of the vehicle parts. The system can generate, based on the image with annotations, a report which indicates a degree of damage to a respective vehicle part. Note that the annotated image is not always necessary for generating the result.

<FIG> illustrates an exemplary method <NUM> for facilitating recognition of vehicle parts, in accordance with an embodiment of the present application. During operation, the system obtains a plurality of images of a vehicle, wherein a vehicle image comprises a plurality of vehicle parts (operation <NUM>). The system convolves and processes an image of the vehicle (operation <NUM>). The system generates a convolution feature map of the image of the vehicle (operation <NUM>).

The system performs, for a plurality of convolutional mapping positions, a convolutional operation for a respective convolutional mapping position with a sliding window in the convolution feature map (operation <NUM>). The system obtains a feature vector of the respective convolutional mapping position (operation <NUM>). The system predicts whether the respective convolutional mapping position comprises a foreground target with respect to a plurality of predetermined anchors based on the feature vector of the respective convolutional mapping position (operation <NUM>). The system predicts a border of a proposed region in the respective convolutional mapping position that corresponds to each anchor (operation <NUM>). Operations <NUM> to <NUM> correspond to operation <NUM> of <FIG>. The operation continues at Label A of <FIG>.

<FIG> illustrates an exemplary method <NUM> for facilitating recognition of vehicle parts, in accordance with the present application. The system obtains classes and bounding boxes of the vehicle parts corresponding to the proposed regions (operation <NUM>). The system extracts correlated features of the proposed regions (operation <NUM>). The system optimizes the classes and the bounding boxes of the vehicle parts based on the correlated features of the corresponding proposed regions (operation <NUM>). The system generates a result which indicates a list including an insurance claim item and corresponding damages based on the optimized classes and bounding boxes of the vehicle parts (operation <NUM>). In some embodiments, the system can output on a display screen the image with annotations indicating the optimized classes and bounding boxes of the vehicle parts. The system can generate, based on the image with annotations, a report which indicates a degree of damage to a respective vehicle part. As described above, the annotated image is not always necessary for generating the result.

<FIG> illustrates an exemplary computer system <NUM> for facilitating recognition of vehicle parts, in accordance with an embodiment of the present application. Computer system <NUM> includes a processor <NUM>, a volatile memory <NUM>, and a storage device <NUM>. Volatile memory <NUM> can include, e.g., random access memory (RAM), that serves as a managed memory, and can be used to store one or more memory pools. Storage device <NUM> can include persistent storage. Furthermore, computer system <NUM> can be coupled to a display device <NUM>, a keyboard <NUM>, and a pointing device <NUM>. Storage device <NUM> can store an operating system <NUM>, a content-processing system <NUM>, and data <NUM>.

Content-processing system <NUM> can include instructions, which when executed by computer system <NUM>, can cause computer system <NUM> to perform methods and/or processes described in this disclosure. Specifically, content-processing system <NUM> can include instructions for receiving and transmitting data packets, including: data to be processed, annotated, classified, and stored; an image; a class; indicators of a bounding box; and a report (communication module <NUM>).

Content-processing system <NUM> includes instructions for generating a convolution feature map of an image of the vehicle (convolution module <NUM>). Content-processing system <NUM> can include instructions for determining, based on the convolution feature map, one or more proposed regions, wherein a respective proposed region corresponds to a target of a respective vehicle part (region proposal module <NUM>). Content-processing system <NUM> can include instructions for determining a class and a bounding box of a first vehicle part corresponding to a first proposed region based on a feature of the first proposed region (classification module <NUM>). Content-processing system <NUM> can include instructions for optimizing classes and bounding boxes of the vehicle parts based on correlated features of the corresponding proposed regions (conditional random field (CRF) module <NUM>). Content-processing system <NUM> can include instructions for generating a result which indicates a list including an insurance claim item and corresponding damages based on the optimized classes and bounding boxes of the vehicle (reporting module <NUM>). Content-processing system <NUM> can include instructions for outputting on a display screen the image with annotations indicating the optimized classes and bounding boxes of the vehicle parts (via, e.g., an image display module, not shown). Content-processing system <NUM> can include instructions for generating, based on the image with annotations, a report which indicates a degree of damage to a respective vehicle part (reporting module <NUM>).

Data <NUM> can include any data that is required as input or generated as output by the methods and/or processes described in this disclosure. Specifically, data <NUM> can store at least: data; convolved or processed data; an image; a captured image; an image of a part or parts of a vehicle; a predetermined class or classes; an annotated image; a report; a degree of damage to a vehicle part; a convolution feature map; a proposed region; a class; a bounding box; a report; a mapping position; a convolutional operation; a sliding window; a feature vector; a prediction; a foreground target; an anchor; a border; a border of a proposed region; a size; a position relationship; a distance; an intersection-over-union ratio; a correlated feature; an energy function; a probability function; a data term; a smoothing term; a recurrent neural network; an iterative operation; a compatibility matrix; a probability of compatibility between classes; a parameter; a training sample; a prediction result; a prediction area; a loss function; a cross term; a model based on a neural network; a result; and a list including an insurance claim item and corresponding damages;.

<FIG> illustrates an exemplary apparatus <NUM> for facilitating recognition of vehicle parts, in accordance with an embodiment of the present application. Apparatus <NUM> can comprise a plurality of units or apparatuses which may communicate with one another via a wired, wireless, quantum light, or electrical communication channel. Apparatus <NUM> may be realized using one or more integrated circuits, and may include fewer or more units or apparatuses than those shown in <FIG>. Further, apparatus <NUM> may be integrated in a computer system, or realized as a separate device(s) which is/are capable of communicating with other computer systems and/or devices. Specifically, apparatus <NUM> can comprise units <NUM>-<NUM>, which perform functions or operations similar to modules <NUM>-<NUM> of <FIG>, including: a communication unit <NUM>; a convolution processing unit <NUM>; a region generation unit <NUM>; a target detection unit <NUM>; an optimization unit <NUM>; and a report generation unit <NUM>.

The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing computer-readable media now known or later developed.

The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and/or data stored on the computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer-readable storage medium.

Claim 1:
A computer-implemented method for recognizing parts of a vehicle, the method comprising:
generating a convolution feature map of an image of the vehicle;
determining, based on the convolution feature map, one or more proposed regions, wherein a respective proposed region corresponds to a target of a respective vehicle part;
determining a class and a bounding box of a first vehicle part corresponding to a first proposed region based on a feature of the first proposed region;
optimizing classes and bounding boxes of the vehicle parts based on correlated features of the corresponding proposed regions comprising establishing a conditional random field, via a conditional random field module, on the convolution feature map; extracting the correlated features between proposed regions; and jointly determining the vehicle part class of a proposed region based on a plurality of proposed regions;
generating a result which indicates a list including an insurance claim item and corresponding damages based on the optimized classes and bounding boxes of the vehicle parts.