Image processing system to align a target object in a target object image with an object model

An image processing system that may be used to detect changes in objects, such as damage to automobiles, compares a base object model, which depicts the object in an expected condition, to one or more target object images of the object in the changed condition. To do so, the image processing system first processes a target object image to detect one or more predefined landmarks in the target object image using one or more correlation filters. The image processing system then uses the detected landmarks to determine a camera model for the target object image and uses the camera model to correct for camera distortions and to align the target object depicted in a target object image with the object in the base object model to put these objects in a common frame of reference for use in subsequent image processing. The image processing system may then determine contours of the target object within the target object image by overlaying an aligned base object model with the target object image, may remove background pixels or other extraneous information based on this comparison, and may perform a statistical processing routine on the identified target object image to detect changes to the target object as depicted in the target object image as compared to the base object model.

TECHNICAL FIELD

This patent relates to an image processing system and technique, and more particularly, to an image processing system and methodology that detects changes to objects caused by, for example, degradation or damage to the object caused by external or internal forces.

DESCRIPTION OF THE RELATED ART

Image processing systems typically operate on images, such as photos, digital images, etc., to enhance the image in some manner or to detect certain features or characteristics of an image, such as to determine information about objects in the image, to recognize persons or things in the image, etc. For example, there are many image processing system that perform character or facial recognition in images to identify text, people, particular buildings, or other features of objects within images in order to automatically identify people, objects, or other features depicted within the image. In many cases, these image processing systems use statistical processing techniques to detect particular features or characteristics of the set of pixels that make up a feature based on the similarity of the characteristics of the image pixels to other images of the same or similar object, feature, etc. being detected. In other cases, image processing systems look for and detect defects in the image caused by the camera, such as red eye detection, distortion detection, color balance detection, etc., all with the goal of correcting or altering the image to make the image a better or more realistic image. As a further example, some image processing systems operate to perform edge detection to detect objects within an image, to filter images in various manners to reduce or enhance particular features of the image, etc. The goal of most of these image processing systems is to create a better or more useful image, or to detect features within the image for some other purpose.

However, currently, there is no known image processing system that can quickly and effectively detect changes that have occurred to an object as depicted in an image to thereby detect the manner in which an object depicted in the image has changed from a known state or condition. For example, known image processing systems are unable to quickly and effectively detect, isolate, or quantify damage that may have occurred to an automobile in an accident, changes to buildings or other structure that may have occurred due to further construction on the buildings, due to tornado or flood damage, etc. While it is possible to compare images of the same object at different times to detect changes to the image of the object, it is difficult to automatically detect or quantify actual changes to the object based on this comparison of the images for a number of reasons. In particular, such a comparison requires the images of the object to be from the same perspective, angle, distance, etc. which is difficult to achieve in practice. Moreover, a high fidelity comparison would normally require that the images be obtained from the same camera to account for distortions that would typically be introduced by different cameras taking images at different times. Still further, a simple comparison of images of the same object from the same perspective and using the same camera may still result in the detection of changes that are not substantive in nature due to differences in lighting, surrounding objects, etc. Moreover, it would be difficult to quantify the nature or type of changes even if such changes could be detected.

SUMMARY

An image processing system that can be used to detect changes in objects, such as to detect damage to automobiles, buildings, and the like, compares a base object model which depicts the object in an expected or pre-changed condition to one or more target images of the object in the changed condition to thereby detect and/or quantify changes to the object as depicted in the one or more target images. More particularly, the image processing system stores a set of base object models, which may each be, for example, a three dimensional model of an object to which change detection is to be applied. Each base object model may include a plurality of landmarks defined therein, which each landmark defining a particular point or spot on the base object.

Thereafter, the image processing system processes one or more target object images to detect changes to the target object as depicted within the target object images, with each of the target object images being an image of the target object after a change has potentially occurred to the target object. The image processing system may detect one or more of the predefined landmarks in each of the target object images using a difference or correlation filter, for example, and may estimate the position of one or more of the other predefined landmarks based on the position of the detected landmarks and the known position of those landmarks in the base object model. Next, the image processing system corrects for camera and positional distortions by, for example, determining a camera model for each of the target images based on the detected and/or estimated landmarks within the target object images. Using the camera model, the image processing system corrects the target object images for distortions and aligns the base object model with each of the target object images to produce one or more base object images in which the base object is aligned with (e.g., viewed from the same perspective, distance, etc.) the target object as depicted in the distortion corrected target object images.

Thereafter, the image processing system determines the contours (e.g., three dimensional contours) of the target object or of various components of the target object within the target object images by overlaying or superimposing the aligned base object model or images with the target object images. The image processing system may then, using the contours of the target object as defined in the base object model, identify the same components in the aligned target object image and may delete or remove background pixels or other extraneous information based on this comparison. The image processing system may next perform a statistical processing routine on each of the identified target object image components, such as a deep learning technique, like one that uses one or more convolutional neural networks, to detect changes, the likelihood of changes, and/or a quantification of an amount or type of change, to each such body component of the target object as depicted in the target object images. Such change detections may then be used for other purposes to, for example, estimate repair costs, estimate the amount of change that has occurred in the object, estimate the amount of time or effort needed to correct or fix the change, etc. Still further, in one case, the change determination may be displayed to a user in the form of a heat map that illustrates areas of the target object that have undergone change, the amount of such change, the type of such change, etc.

DETAILED DESCRIPTION

FIG. 1illustrates one example of an image processing system100which may be used to determine changes to a target object, automatically or semi-automatically, generally by determining differences between one or more images of the target object as changed and one or more images generated from a base model of the object representing the object prior to the changes occurring to the object. After changes to the target object have been determined and quantified in some manner, further processing systems, which may be incorporated into the image processing system100, may be used to determine secondary characteristics or features associated with the target object, such as the amount of change or damage to the target object (assuming that the changes to the object represent damage), costs associated with repairing or replacing the target object, the time it may take to repair the target object, the progress the target object has undergone in changing states, etc.

As illustrated inFIG. 1, the image processing system100may include various user interface devices102, which may be general purpose computers, handheld devices, workstations, etc., connected through one or more communication networks104, to one or more further computing devices106which may operate as servers and/or databases in a traditional client-server or cloud-based network. For example, the further computing devices or servers106may include one or more databases108,109, and110, and one or more servers112and114having computer processing and storage capabilities.

In this case, the database108may store one or more base object models120(which may be, for example, two dimensional and/or three dimensional object models) defining or depicting one or more base objects upon which change detection image processing may be performed. Moreover, the database108may store changed or example target image files122, which may be files that include images of changed target objects illustrating various manners in which changes to the base objects within the base object models120may occur. Still further, the database108may store information files124relating to or providing specific information about each of the changed target image files122or the changed objects within the changed target image files122. In one example, in which the image processing system100may be used to detect changes to automobiles, such as to detect damage caused to an automobile in an accident, the base object models120may be three dimensional (or two dimensional) models of various automobiles upon which changes may be detected. Generally in this case, a different base object model120will be stored for each make/model/year or make/model/year/trim-type of automobile that is to be analyzed for damage. Still further, in this example, each of the changed image files122may include one or more images of a damaged vehicle (conforming to one of the make/model/year types, for example, of the base object models120stored in the database108). Generally, each such changed image file122may include digital photos taken of a particular automobile that has been damaged in, for example, an accident. Such photos may be collected by, for example, owners of the automobiles depicted in the photos, an automobile insurer against whom an insurance claim was made for repairing or replacing the damaged automobile, etc. Still further, each of the information files124may store information pertaining to the damaged automobiles in one of the changed target image files122, such as the make/model/year and trim-type of the damaged automobile, the country, state, city, zip code, and/or other geographical region in which the automobile was insured or damaged, the mileage of the damaged automobile, the color of the damaged automobile, the type of or location of the damage to the automobile, telematics data obtained from or about the damaged automobile associated with the accident, the parts which needed to be repaired or replaced as a result of the damage, the cost of repair or replacement of each such part, the type of damage to each such part, whether the automobile was considered a total loss as a result of the damage, the cost of repair of the automobile if the automobile was repaired, the insurer of the automobile, if any re-inspection was performed on the automobile during repair, capitation of the automobile, etc. Of course, other information could be stored for any or all of the changed target image files122, and the type of information stored for each of the changed target image files122may vary depending on use, the type of object upon which change detection is to be performed, etc. As will be described in more detail herein, the base object models120, the changed target image files122, and the information files124may be used by the image processing system100, for example, to perform primary and secondary processing on photographs of a newly damaged automobile to determine the type of and/or the extent of damage (change) to the damaged automobile.

The server112, which may include a microprocessor128and a computer readable memory129, may store one or more image processing training routines130. The training routines130may be implemented on the microprocessor128using the base object models120, and the images and data within the files122and124to generate various other components used in a further image processing routine that operates to determine changes to target objects (which are represented by one of the base object models120) on which changes have occurred but for which changes have not been quantified. In one example, one or more of the training routines130may determine a set of correlation filters132(also referred to as difference filters) for each of the target objects, and one or more of the training routines130may determine a set of convolutional neural networks (CNNs)134for the objects represented by each of the base object models120. The correlation filters132and the CNNs134are illustrated inFIG. 1as being stored in the database110, but these components could be stored in other memories associated with the image processing system100if desired.

Thus, generally speaking, and as will be described in further detail herein, the image training routines130use the base object models120, the changed image files122, and the image information files124to produce and/or select the correlation filters132and the CNNs134, in one example, that will be used by the image processing system100to detect changes to target objects (such as to detect damage to automobiles or other vehicles) based on images of the target objects, such as based on photographs of damaged automobiles. In one example, the image training routines130may calculate, for each of the base object models120, a set of correlation filters132and a set of CNNs134based on the training set of data in the changed target image files122, and these tools132and134may be stored and then later used to detect damages on images of target objects for which changes to the target objects (such as damage to the target object) is unknown or not quantified.

Moreover, as illustrated inFIG. 1, the server114, may include a microprocessor138and a memory139that stores an image processing routine140that may perform image processing on a set of target images142, which images depict a target object represented by one of the base object models120, but upon which change has occurred, to thereby detect changes to the target object, such as to detect or quantify damage to the target object. In this case, the images142, which may be stored in the database109for example, represent the target object as changed or altered with respect to the same object as represented by one of the base object models120. Generally speaking, the image processing routine140may use the base object model120corresponding to the target object, the correlation filters132and the CNNs134(as stored in the database or memory110) corresponding to that base object model120, as well as the set of target images142for the target object to detect changes that have occurred to the target object, where the changes are represented or depicted within the set of target images142of the target object.

During operation, a user may log onto or access the system100via one of the user interfaces102, may upload or store a new set of images142of a target object in the database109, and may additionally provide or store information in the database109related to the new set of images142, such as an identification of the target object within the new set of images142(e.g., the year/make/model of a damaged automobile depicted in the images), information about the target object (such as vehicle mileage, location or geographical region of the automobile, etc.), as well as any other desired information about the images142or the target object within the images142, such as telematics data collected by the automobile depicted in the photographs, first notice of loss information as collected by the insurance carrier of the automobile depicted in the photographs, an indication of the angle or perspective at which the object is depicted in the photograph, etc. The user may then initiate the image processing routine140to operate on the new set of target images142to detect changes within the target object depicted in the new set of target images142as compared to the base object model120for that same object. Of course, the new set of target images142, along with information related to the new set of target images142, may be stored in the database109and/or provided to the database109in other manners, such as via a direct or indirect connection to a camera, via another device in the communication network(s)104, e.g., via a handheld device102A connected to the network(s)104via a wireless interface152, etc. Moreover, if desired, a user may use one of the user interfaces102,102A to additionally or alternatively select a sub-set of a series of images that have been previously collected or taken of the target object, such as different views of the target object from different angles, and may provide these images to the database109or may mark these images in the database109for use as and/or inclusion in the new set of target images142to be processed by the image processing routine140.

Generally speaking, once initiated, the image processing routine140will operate to detect changes to the target object depicted in the new set of target images142as compared to the target object as represented by a base object model120for that same object to, for example, detect and quantify changes to the target object, such as damage to the target object. Generally speaking, the image processing routine140will use the correlation filters132and the CNNs134to detect differences between the target object depicted in the new set of target images142and information about the target object as stored in the base object model of the object120to determine an amount of, or a quantification of change, such as damage, to the target object as depicted in the target images142. The image processing routine140may then may provide this change detection information to the user via one of the user interfaces102,102A in any of various different manners. More particularly, the image processing routine140uses the correlation filters132or other difference filters to detect one or more predetermined landmarks (of or on the target object) in each of the target images142and uses these landmarks to detect the positioning, orientation, depth of field, etc. of the target object as captured in the target images142. The image processing routine140then uses these landmarks to perform a multi-dimensional alignment between the base object model120(associated with the target object depicted in the target images142) and the target object as actually depicted in the target images142. Such a multi-dimensional alignment may include matching or aligning the three dimensional orientation and depth of field or other positioning of the target object as depicted in one or more of the images142with the base object model120. This multi-dimensional alignment may also include detecting and correcting for camera distortion introduced into the target images142by the camera that took those images. In one case, the image processing routine140may then generate one or more images of the target object from the base object model120that are aligned with the target object as depicted in the target images142, and uses these aligned images to detect and eliminate background information in the target images142as well as to detect various components or parts of the target object as depicted in the images142. In another embodiment, the image processing routine140may use the landmarks to create a composite image or model of the target object from the set of target images, enabling the target object, as depicted in the target images, to be modeled in, for example, three dimensions and thereby be rotated, zoomed into and out of, etc. This model of the target object may then be aligned with the base object model120to enable comparisons between the two objects.

Thereafter, the image processing routine140uses the CNNs134to detect change parameters for each of the various components or parts of the target object as depicted in the images142or the composite target object model created from the images142. Such change parameters may include, for example, indications of points or areas of change (e.g., damage), the probability of change at each point or area, the severity of change at each point or area, the type of change at each point or area, etc. These change parameters may then be used to display a heat map to the user, wherein the heat map is indicative of, corresponds to, and/or defines the areas of change to the target object depicted in the target object images142as compared to the same object as defined by the base object model120. Likewise, these change parameters may be used to determine secondary factors or features associated with the target object, such as cost to repair the changes, the estimated time it will take to repair the changes, an indication of a degree of metamorphosis of the object from one state or condition to another state or condition, etc. To initially set up the image processing routine140for use in detecting changes to particular objects, such as to detect damage of a particular type of object (e.g., as represented by a particular base object model120), the same or a different user may access the training routines130via one of the user interfaces102,102A and may execute the training routines130to create the correlation filters132and the CNNs134needed for the image processing routine140for the particular object to be analyzed. Thereafter, these tools132and134may be stored in the database110for example, for later use by the image processing routine140. Generally speaking, the training routine130will access the base object model120for a particular object as well as a training set of data associated with the base object model120(e.g., a training set of images stored in the associated changed target image files122and the information about these objects as stored in the associated information files124), and will process the training set of data as compared to the base object model120for the object to determine a set of processing tools needed by the routine140, including for example, the correlation filters132and the CNNs134. The training routine130may then store these processing tools in the database110for later use by the processing routine140.

While it will be understood that the image processing system100ofFIG. 1is illustrated as a hardwired system having various servers and databases106connected to various user interfaces102via a hardwired communication network104, other processing and communication configurations could be used as well or instead. For example, the network104could be a wireless communication network, or could be a combined wired and wireless communication network, and the network104may include any type of communication network, including a public local area network (LAN) or wide area network (WAN), a private LAN or WAN, a set of direct communication connections, etc. Moreover, the network104may include or use the Internet or the World Wide Web, to enable users at the user interfaces102and102A to communicate with the servers and databases106via an Internet connection to upload photos or images of a target object for which change is to be detected to the database109, to initiate the image processing routines130or140, to view the results of the routine140, etc. Still further, while the servers and databases106are illustrated inFIG. 1as including five different devices that have different information stored therein, any number of servers, databases, and/or other computing devices could be used and, in fact, the elements illustrated in the servers and databases106ofFIG. 1could be stored within or distributed among more or less than five devices. For example, these components could be all stored in the same computer processing device, which could be the same computer processing device as one of the user interfaces102or102A. In another example, at least some (or all) of these components could be implemented using a computing cloud architecture. Thus, the architecture of the image processing system100ofFIG. 1could be changed to include more or less computer devices connected in various different manners and still function as described herein.

FIG. 2depicts a block diagram200of a system and/or method that may implemented using the image processing algorithm or routines130and140ofFIG. 1to detect changes in a target object based on images of the target object and a base object model associated with the target object. In particular, a block202collects images of the target object on which or for changes are to be detected and quantified in some manner, such as images of a damaged automobile that are to be analyzed to determine the extent of or nature of the damage thereto. The image collector block202may, as indicated inFIG. 2, receive raw images of the target object (also referred to interchangeably herein as “source images”) on which change detection is to occur, e.g., from a user, a database, or any other source. Such images may include still image or video images, and/or may include two-dimensional and/or three-dimensional images. Moreover, in some cases, the image collector block202may perform some processing on the collected images to select images that are of higher quality or that are better suited for use in processing by the image processing routine140to detect changes to the target object depicted in the image as compared to a base object model of the target object. For example, the block202may collect information about the target object depicted in the target images (such as the make/model/year of an automobile in the images, mileage of the automobile, trim type of the automobile, location or region of the automobile, etc.), and the block202may collect data about the images themselves, e.g., an indication of the perspective of the object captured in each image (such as whether an image depicts a front view, a side view, a corner view, a top view, a bottom view of the object, etc.). Likewise, the block202may receive other information about the nature of changes to the target object or indicating information that indicates potential changes to the target object. For example, when analyzing damaged automobiles, the block202may receive telematics information collected from the automobile during the accident in which the damage was incurred, which information may be used to infer where damage generally was likely to have occurred on the automobile as depicted in the target images. In other cases, the block202may receive indications of where damage occurred to the automobile or other object as inputs from a user, such as the owner of the automobile, from insurance claims information, etc. In one example, the block202may receive first notice of loss (FNOL) information, which is generally information collected by an insurance carrier at the first notice of loss, e.g., when the owner first notifies the carrier of a claim or a potential claim. Standard FNOL data may include, for example, vehicle year, make, model, and trim type, vehicle telematics data such as a primary impact code, a secondary impact code, a drivable code, an airbag code, etc., vehicle odometer reading, vehicle state (zip code), vehicle insurance company, etc.

In any event, the block202may select a subset of the raw images of the target object to use in processing to determine or quantify the change to the target object as compared to the base object model for the target object. If not enough images, or if images of the right type or quality, have not been collected, the block202may provide feedback to a user (via a user interface102ofFIG. 1, for example,) that additional images of the target object are needed for change processing. This operation is indicated by the dotted lines203inFIG. 2. Such an indication may include a request for more images of the target object, a request for images of the target object from a particular angle or perspective, distance, etc.

Next, when a set of new images has been collected/selected for a target object to be analyzed for changes, such as for damage, a block204determines a base object model (i.e., one of the base object models120ofFIG. 1) to use in the change detection processing. As an example, the block204, when analyzing images of an automobile for damage, may determine the make/model/year of the automobile within the collected images and may obtain a base object model120for that make/model/year from the database108ofFIG. 1. As generally indicated above, the base object model120may be a three dimensional model of the object that, for example, defines the three dimensional contours or shape of the object in its pristine or pre-changed condition. In other cases, the base object model120may be a set of two dimensional models (e.g., images) that define different views of a three dimensional object, as well as information defining the exact view (e.g., the angle, the distance, etc. of the object from the focal point of the camera) of the object within the images. In cases in which the object being analyzed is a two dimensional object or may be represented adequately as a two dimensional object, one or more two dimensional base object models may be used as a composite base object model. Generally speaking, the block204may develop or obtain one or more images from the base object model120during further processing, and these images may be used to determine changes to the target object being analyzed (i.e., to the target object depicted in the images selected by the block202).

Next, a block206, aligns, on an image by image basis, or on a composite basis, each of the selected images (the set of images142ofFIG. 1) with one or more images developed from the base object model120to enable a comparison between the target object as depicted in the selected images and the base object model. Generally speaking, the block206performs multi-dimensional alignment so that images of the object generated from the base object model are aligned in three dimensions with the target object as depicted in the corresponding images of the target object as collected by the block202. In some cases, the block206may use the correlation filters132ofFIG. 1to perform this alignment in a manner described in more detail herein. However, generally speaking, for each of the selected images, the block206produces an image from the base object model that matches or aligns with the orientation of the target object as depicted in the selected image. On the other hand, if desired, the block206may process the selected image to reorient the target object within the selected image to align with a particular image generated from the base object model or with a particular view of the base object model. In other cases, the block206may create a composite target object model from the target object images and may align the composite target object model with the base object model. Additionally, the block206may correct for camera distortion by determining one or more camera transfer functions (e.g., distortion transfer functions) and applying the one or more transfer functions to the image developed from the base object model or by applying an inverse of the one or more camera transfer functions to the selected images or to the composite target object model to correct the selected images or model. Thus, in one example, the block206produces, for each of the selected images (from the block202), an image of the pristine or unchanged object from the selected base object model, wherein the image of the pristine or unchanged object is aligned with (e.g., oriented in the same manner as) the target object as depicted in the selected image. This alignment can correct for (or be used to match) the camera position (e.g., camera angle), focal length, distance to the target object from the focal point of the camera, etc., to thereby size, scale, and orient the image from the base object model in a manner that is matched in size, scale, and orientation to the target object as depicted in the selected image. Moreover, this alignment may correct for distortion introduced by the camera that took the target object image.

Still further, the block206may isolate the target object as depicted in each of the one or more selected images (from the block202) by eliminating or reducing background effects and/or content, noise, or other image artifacts, such as glare, reflections, etc. in the selected images, to thereby enable a better comparison between the object as depicted in the base object model and the target object as depicted in the selected images. In some cases, the block206may apply image filters to remove known types of image artifacts, such as glare, reflections, etc. As such image filtering techniques are well-known, they will not be described in further detail herein.

In some cases, the base object model may have information on or may define parts or components of the object being analyzed. For example, if the object being analyzed is an automobile, the base object model may identify the various body panels of the automobile separately and may generate images of the object with the various components of the object identified, outlined, or isolated. Such an image that is developed from the base object model with various components thereof delineated is referred to herein as a segmented image or a componentized image. Moreover, as illustrated inFIG. 2, the block206may operate to detect or determine if more images of the target object are needed to perform change processing thereon. For example, the block206may detect that various angles of or components of the target object are not depicted in or detectable from the selected images of the target object and that further images of the target object showing these components are needed to perform a complete change processing routine. Still further, the block206may determine that various images as selected by the block204are not able to be aligned with the base object model, that the camera distortion is too great, that the image detail of the target object as depicted in a selected image is not high enough (e.g., the object is too far away from the camera or too small within the image) to be processed adequately, etc. In these cases, the block206may provide feedback as depicted by the line203that more or better images of the target object are needed.

Next, a block208may determine the change to the target object as depicted in the selected, aligned, and corrected target object images as compared to the pristine or unchanged object depicted by the base object model. The block208may determine where change has occurred, exists, or is likely to exist, and in some cases may determine the extent of the change or even a quantification of the type of change (e.g., damage) that has occurred to the target object as depicted in the selected, aligned, and corrected images of the target object. Generally speaking, the block208may overlay the aligned, segmented image developed from the base object model with the corrected image of the target object to enable identification of or detection of the boundaries of the various different components of the target object as depicted in the corrected images of the target object. Moreover, this overlay may be used to eliminate pixel information in the target object image that is not associated with the target object, such as background information or pixels, foreground pixel information, etc. In cases in which the system200uses a composite target object model, the block208may perform a three-dimensional overlay or comparison between the composite target object model and the base object model. This technique is different than normal edge detection techniques which process the image pixels looking for edges of the object based on the difference or sudden change of pixel values within the image. In this case, the edges of the object components as defined by the base object model are aligned with and overlaid onto the target image to define the edges of these same components in the target object as depicted in the target image. The pixels outside of these edges are then designated as background pixels, not associated with the target object or with the component of the target object.

Thereafter, the block208performs change detection image processing on each of the various components of the target object as depicted and identified in the target object images (or the composite target object model), to identify and, in some cases, quantify changes that have occurred to the target object depicted in the target object images. In one case, the block208may apply one or more CNNs (such as the CNNs134ofFIG. 1) to the image areas defined by each of the components of the target object to detect the existence of change, the likelihood of the existence of change, a degree of change, or a quantification of change (e.g., damage) to the target object as depicted in the target object images (or composite target object model). A separate CNN may, for example, be applied to each different component of the target object as depicted in the target object images. As will be understood, a CNN may determine, for each pixel or set of pixels in the image corresponding to a particular component or part of a component, a likelihood of change or damage at that point, an estimate of the likelihood of change or damage at that point, and/or a type or severity of change or damage at that point, and may represent this likelihood of change or other change parameter as a number on a scale, such as a number between 0-99, in which 0 represents no likelihood of change or damage and 99 represents a high likelihood of change or damage. The CNNs may, in some cases, determine a similarity of the analyzed portion of a component of the target object to the object as defined by the base object model.

More specifically, the block208may select a CNN that has been developed for a particular component or segment of the base object, and may apply the CNN to each pixel (or every other pixel, or every third pixel, etc.) of the changed or target image that has been identified as being associated with that component or segment, and in doing so may calculate a value for that particular application of the CNN. In some cases, each CNN may define a particular image area (e.g., a four pixel square, a 16 pixel square, a circular section, a rectangular section, etc. of the image) and the pixels of the target image object covered by that area are used as inputs to the CNN. The block208may apply a CNN to a particular pixel of the image by, for example, centering the CNN image area over the pixel being analyzed. The pixels of the image that are then coincident with the CNN image area are used as inputs to the CNN, which may have been trained for example to produce an output based on those inputs indicating the likelihood of change at the center pixel. The block208may scan the CNN over each pixel of the image (or other area of the image, such as every second pixel, every fifth pixel, etc.) associated with a component of the object to perform a CNN calculation at each of the scanned pixels. If desired, the block208may apply one or more CNNs to each of the components of the object as depicted in the target object image and, in some cases, may apply more than one CNN to each of the components of the object.

Thus, to perform damage detection, the block208uses each of the target object images (of the changed or damaged vehicle or other target object), in which the target object depicted in the image is isolated, corrected for distortion, and aligned with a segmented image of the base object model. As part of this process, the block208uses the aligned, corrected, segmented image of the base object model (or such an image produced from the base object model) to identify object segments or components of the damaged or target object as depicted in the target object images. The block208then uses various tools, such as different types of processing routines, to determine various changes, the extent of the changes, and/or potential types of changes, that have occurred to the target object as depicted in the target object images. For example, the determination of changes may include indications of dents, scratches, creases, folds, or other types of damage to a particular component of a vehicle, which may help quantify the extent of the damage to the component of the vehicle.

Still further, as illustrated inFIG. 2by the dotted line203, the block208may determine that the analysis of change of the target object is indeterminate or of low statistical quality, and may inform the user of this problem, to enable the user to obtain better images of the target object to perform better or more accurate change detection. For example, the results of the change detection on a particular component of a target object may indicate a high likelihood of severe change at all points or at no points, or may fluctuate between high change and low change in a manner that seems unlikely or impossible, etc. These or other analyses may be performed by the block208on the change parameters as determined by the block208to determine if further or other images of the target object are needed to provide better or more accurate change detection.

If desired, a block210may then display indications of the detected changes or change parameters to a user via, for example, one of the user interfaces102(or102A) ofFIG. 1. For example, the block210may display a heat map of the target object indicating the spots, locations, or areas (e.g., pixels) at which change is detected or is detected to be likely. Such a heat map may be included in, overlaid on, superimposed on, or otherwise applied to an image of the object (such as one of the target object images) wherein different pixels of the target object within the image are displayed using different colors that indicate different degrees of change or likelihood of change to respective spots, locations, or areas of the target object (such as an amount of damage or a likelihood of damage). Such a heat map may be generated using the outputs of the CNNs so that the respective color or appearance of each of the pixels or surface segments of the image that correspond to various components of the object depicted in the image is based on and/or corresponds to a respective CNN output (e.g., to respective values of one or more change parameters). Still further, such a heat map may include change depictions or indications on any number of the selected images (or from an image generated from a composite target object model) to illustrate detected changes to the target object from various different viewpoints or sides of the target object. Still further, the heat map may be used to determine if other or better images of the target object need to be obtained and processed to perform better or more accurate change detection, as illustrated by the dotted line203. For example, a user may view the heat map and determine that areas of the target object have undergone change or types of change that were not detected by the block208, indicating that the images of the target object used for change processing were not adequate to detect all of the changes. On the other hand, the user may determine that the block208detected changes at points on the target object at which no changed occurred and, once again, may use this knowledge to instruct the system200to obtain other or better images of the target object for processing. Additional discussion of heat maps (e.g., block210) is provided in later sections of the present disclosure

Still further, once an indication of the change to a target object is determined by the block208, a block212may use the change indications or the change determinations for the various components of the target object (or the heat map created by the block210) to determine secondary considerations based on the determined changes. Such secondary considerations might include determining an estimate of the repair costs associated with repairing the target object, the time associated with correcting or fixing the target object based on the changes made to the object, a representation of an amount of metamorphosis (e.g., from one state to another state) that the target object has undergone, etc. The secondary considerations may be used to, for example, order parts for repairing a changed or damaged vehicle, estimate insurance settlements, estimate repair times, etc. Of course, the block212may hand off the results of the change estimation and/or the quantification of the change estimation to one or more other routines to perform the secondary considerations based on the detected extent of the changes, type of changes, etc. As indicated inFIG. 2, various types of information or data may be used to perform or determine these secondary considerations, such as historical data related to damaged vehicles (e.g., that may define costs, times, etc. for similar types of detected changes in the target object), telematics data from a vehicle for example on which change processing is being performed, information associated with FNOL of the target vehicle (e.g., location of accident, location of insurance policy, primary impact code, etc.), cost of components of the object from vendors, predictive models, estimating programs, etc. In some embodiments, the block212utilizes one or more predictive models215that are generated based on the various types of information, and/or utilizes one or more filters218to determine these secondary considerations. Additional discussion of post-processing and secondary considerations (e.g., block212) is provided in later sections of the present disclosure.

Still further, as indicated inFIG. 2by the dotted line203, the output of or the calculations performed by any of the blocks202,206,208,210, and212may be used to determine that new or different images of the target object are needed to perform better or more accurate change detection. For example, as noted above, the block206may determine that it is not possible to align one or more of the target object images with the base object model and, as such, that new images of the target object are needed. Likewise, the block206may determine that it is not possible to correct for distortion in a target object image or to remove background or foreground objects in the image, and that new or better images of the target object are thus needed. Still further, the change detection block208may determine that the change determination for a particular image or images is not conclusive or accurate enough, or that it may have a high likelihood of error, and that new images may be needed to perform better change detection processing on the target object. Still further, a user may use a heat map generated by the block210to determine that the detected changes to the target object are not accurate or complete, and that new images are needed. Moreover, the block212may determine that secondary considerations associated with the detected changes are inaccurate or unlikely to be correct, and that processing on new images of the target object should be performed. In any or all of these cases, the image processing system may optionally ask for or request zoom-in photos of a damaged or changed area of the object to further identify and quantify the damage type/severity, or could ask for or request zoom-out photos if not enough of the object is depicted in a particular photo or target object image or if damage spans to the edge of the image. In some cases, the system could perform 2D/3D registration using the zoom-in (or zoom-out) photos and therefore have better capture of corresponding damages or changes on the target object.

It will be understood that, while the specific discussion of, and the specific example of change processing provided below is directed to detecting damage to an automobile, such as damage caused by an accident in which the automobile was involved, the same or similar image processing techniques may be applied to other types of objects, for detecting other types of changes, including, for example, detecting changes in building structures or buildings, changes to other forms of real property, such as land, crops, etc., as well as to detecting changes to many other types of objects. Moreover, the detected changes do not have to be related to damage to an object but could instead be related to changes based on or that result from a metamorphosis of an object from one state or condition to another state or condition. For example, it may be possible to detect the growth of a set of crops, animals, etc. over time during various stages of development. In one specific example, it may be possible to detect when fruit bearing trees, or crops have ripened to a point that they are ready to be picked or harvested, to detect a state of construction of a building, etc. Still further, it may be possible to detect changes caused to an object in the form of natural aging or erosion of a target object, or changes in the form of damage caused by third parties or even by nature, such as damage caused by tornadoes, wind, rain, floods, etc. Thus, while the following discussion is centered around the detection of damage to an automobile, other types of damage or other type of changes to automobiles or other types of objects may be detected using the same or similar procedures.

FIG. 3illustrates an example routine or method300that may be implemented by the systems ofFIGS. 1 and 2to detect damage to an automobile or other vehicle caused by, for example, an accident. Generally speaking, at a block302, the routine300acquires one or more images of the damaged vehicle, referred to herein as target vehicle images. These images may include what is generally called the four-corner images (that is, images taken of the vehicle from each of the four corners), side view images, front and/or rear view images, top and/or bottom view images, inside view images (such as of the dash board, the trunk compartment, the passenger compartment, the engine compartment, etc.) of the automobile, an any other types of images. Of course, any number of images may be taken and collected of a particular damaged vehicle and any of these images can be provided to and used in the image processing routine300in determining damage caused to the vehicle.

At a block304, the system or routine300may analyze each of the images in some preliminary manner to determine whether each image is usable or necessary for use in determining the damage to the vehicle. In many cases, different images may show the same or overlapping portions of the vehicle, with some of the images being of better quality or usefulness than other images. The block304may select a subset of the provided images, where the subset of images is deemed to be best to be used to determine the damage to the vehicle. In some cases, the block304may determine that not enough images have been taken, or that images of the right quality have not been obtained, or that a complete set of images or images from a necessary set of angles have not been obtained, etc., to enable the image processing routine300to determine a reliable damage estimate. In such cases, the block304may return control to the block302, which may then instruct a user to take or otherwise obtain new images of the damaged automobile. The block304may provide instructions as to the type of images to take, such as the angle from which to take the images, one or more potential problems with the original images (e.g., too much glare, foreground objects are in the way, the images were taken from a point too close or too far away from the vehicle, etc.). Based on these comments, the block302may, for example, instruct the user to take or otherwise acquire particular types of images, such as four corner images, an image of the right or left side of the car, etc. In any event, the blocks302and304may iterate until an acceptable subset of images have been obtained, which will enable the imagine processing routine300to reliably predict the damage to the vehicle based on the image processing techniques described herein. Of course while the block304is illustrated as being immediately downstream of the block302, the block304could be dispersed at any part throughout the flowchart of300ofFIG. 3, in the manner illustrated inFIG. 2, to enable the system300to more accurately determine whether a set of target vehicle images has been obtained depicting the damaged vehicle that are usable in reliably detecting and/or quantifying damage to the vehicle.

Next, a block306operates to select or identify a base object model of the automobile or vehicle depicted in the obtained target object images. The block306may obtain an identification of the automobile depicted in the selected target object images (such as the make/model/year of the automobile or some other identification of the automobile type depicted in the obtained images) from, for example, image information files developed for the target vehicle images, the FNOL information submitted to an insurance carrier, the user directly, etc. The block306may use this identification information to select or find a base object model, as stored in one of the base object model files120ofFIG. 1, that corresponds to the identified target vehicle. As noted above, the base object model for a vehicle or automobile generally represents the vehicle depicted in the target vehicle images (e.g., the target vehicle) in a condition prior to the occurrence of the damage, or in an as expected condition or state. In some cases, the block306may automatically process the target vehicle images to provide a best fit between the vehicle image depicted in the target vehicle images and a vehicle represented in the base object models120, to thereby determine the specific type of vehicle depicted in the damaged vehicle images.

As an example,FIG. 4illustrates a point cloud object model402of a vehicle, which in this case is a Toyota Camry® of a particular year, that may be used as a base object model for damaged or target vehicle images depicting a damaged Toyota Camry of the same year. The point cloud object model402may be stored in, for example, the vehicle image database108ofFIG. 1, and may be obtained from, for example, the automobile manufacturer, or from any other source. Generally speaking, the point cloud object model402is a three dimensional model that stores points in space that define or identify the contour of the body of a vehicle being depicted or associated with the object model. In some cases, the point cloud object model402may include points that define the outer contours of the various body panels of the automobile, but the point cloud model402could additionally or instead define other contours of the automobile, including inner contours (such as contours defining the engine compartment, the passenger compartment, the inside of the trunk, between various body parts or panels, etc.). If desired, the base object model, such as the point cloud model402ofFIG. 4, may be rotated in various manners to correctly illustrate the object (e.g., the automobile) from different angles, positions, view-points, distances, etc. Moreover, while a point cloud object model402is illustrated inFIG. 4as being used as a base object model, other types of object models could be used as well, including any types of mesh models such as three dimensional (3D) triangle mesh models (including triangulated 3D models, point cloud models, polygon mesh models, triangle mesh models), wireframe models, surface patch models, rendered surface models, etc.

Still further, while the base object model in the form of the point cloud model402is depicted and described as a three dimensional model, the base object model could be a two dimensional model (such as a flat image or model of the object in a non-damaged or expected condition) or could be made up of a set of two dimensional models (e.g., from a set of two or more images of the object from different points of view or angles, in a pre-damaged condition). Still further, the point cloud object model402may be manipulated, e.g., rotated and viewed from different angles, perspectives and distances, using any well-known three dimensional viewing routine or software, such as a computer aided drawing (CAD) program or the like.

As a still further example,FIG. 5illustrates a base object model502, which may be developed from, for example, the point cloud model402ofFIG. 4, and which may also or instead be used as a base object model by the image processing system300for detecting damage to an automobile of the same type. In particular, the base object model502includes contour lines drawn between adjacent points in the point cloud model402, forming triangles. Each triangle defines a particular surface segment of the automobile within the base object model502to thereby define the different flat surfaces (e.g., relatively flat or planar surfaces) of the base object. As will be understood, smaller triangles are associated with or are present at locations on the surface of the automobile with larger gradients of curvature, and larger triangles are associated with or are present at locations on the surface of the automobile with smaller gradients of curvature. That is, more curves or higher degrees of curvature in the surface of the automobile at a particular location tend to result in smaller area triangles being formed at or for that location within the base object model502. Thus, the base object model502ofFIG. 5is a triangulated surface model that illustrates the three dimensional surface contours of the vehicle being modeled. Each of the triangular surface areas of the model502ofFIG. 5may be considered a surface segment of the base object model502.

Moreover, as illustrated inFIG. 6, the base object model502(illustrated inFIG. 6with the triangular surface segments being slightly grayed) may include indications or outlines of various different body panels (indicated inFIG. 6by thicker lines) associated with the automobile being modeled. For example, body segments or body panels of the automobile depicted by or modeled by the base object model502may include a hood segment612, a front bumper segment614, a grill segment616, a driver's side front door segment618, a driver's side rear door segment620, a windshield segment622, a roof segment624, a front wheel segment626, a tire segment628, a driver's side rear view mirror segment630, etc. Of course, each of these segments may define or correspond to a different body panel or body part that is, for example, viewable from the outside of the automobile and thus that is present in the base object model502. The body segments may of course define other parts or components of the object being modeled, if so desired. In one case, each body segment may define a different body panel as defined by the manufacturer of the automobile, with each body panel being a unitary, single piece, and/or separable piece of the automobile that may be replaced or changed without changing the other body panels or segments.

FIG. 7illustrates an image or photo of a target (e.g., damaged) vehicle corresponding to the base object model ofFIGS. 4, 5, and 6. The photo image ofFIG. 7may be one of the target images obtained from a user of the system300which will be used, along with the base object model502, to determine a characterization of or an identification of the damage to (change in) the target vehicle depicted in the image ofFIG. 7with respect to the base vehicle as defined by the base object model502. As will be seen, the target vehicle or automobile depicted in the image ofFIG. 7has damage to the front end thereof which effects (has changed) at least the grill, the front hood, and the front bumper of the vehicle. Of course, there may be other damage to the target vehicle withinFIG. 7that is not visible in the image ofFIG. 7, but that may be visible in other images of that vehicle (e.g., in rear views, passenger side or front corner views, etc.) which images may be provided and processed separately.

Referring back toFIG. 3, a block308may operate to align the base object model502(or to align one more images obtained from the base object model502) with each of the selected images, such as the target vehicle image ofFIG. 7. Still further, the block308may correct for distortion in the target vehicle images (or a target vehicle model developed from those images), such as to correct for camera point of view distortions, camera lens and focal length distortions, etc. Generally speaking,FIG. 8illustrates a flowchart800, which may be used in, for example, the block308ofFIG. 3to perform target image/base object model alignment and correction of the target vehicle images. In particular, as illustrated inFIG. 8, the flowchart800includes a block802, which accesses the target vehicle images and which may analyze each of the target vehicle images separately to correct and align each target image with the base object model (or vice-versa). In particular, the block802may select a particular target vehicle image and provide control to a block804which processes the target vehicle image to detect one or more landmarks within the image. Generally speaking, landmarks are particular points on the object defined in the base object model. As an example,FIG. 9illustrates a base object model associated with the vehicle for which damage is to be detected, but having various landmarks defined therein. In this case, each landmark is designated with a plus sign (+). Generally speaking, any number of landmarks may be selected or predefined for each base object within a base object model, and these landmarks may be stored as part of or as associated with the base object model. In some cases, landmarks may be best located at the intersection of various different sections, parts, or components of the base object model to make such landmarks more easily detectable, although this suggestion need not be the case in all situations. In the example ofFIG. 9, landmarks may be defined in the base object model502to include positions at which the grill616of the vehicle intersects with the bumper614, at which the headlights intersect with the side panels of the vehicle, at the four corners of the windshield622, at the intersection of the driver's side front door panel with the front body panel, etc. Of course, any number of landmarks may be defined for and stored within or as part of the base object model502. Generally speaking, the routine800will determine the landmarks that are detectable within a selected target vehicle image, and will use those detected landmarks to then align the object depicted in the base object model with the object as depicted within the selected target vehicle image. In other cases, the routine800may detect landmarks in multiple ones of the target vehicle image in the process of creating a composite target vehicle model from those images.

To perform landmark detection, the block804may apply various ones of the correlation filters132to the target vehicle image (e.g., the image ofFIG. 7) to detect the landmarks within the target vehicle image which are defined in the base object model502. Basically, a single or different correlation filter132may be created for each of the landmarks within the base object model502. In one example, the correlation filters132may be, for example, two dimensional image filters (e.g., a 10 by 10 pixel filter, a 100 by 200 pixel filter, etc.) that measures the correlation of or matching of the surrounding pixels of a particular point in the target vehicle image to the surrounding pixels of the base object image at which the landmark is defined. A high degree of pixel matching (from a least squared error perspective, for example) results in a high correlation value for a particular pixel being analyzed (such as the center pixel of the correlation filter) resulting in a high likelihood that the particular pixel in the target vehicle image matches up with the landmark as defined in the base object model502.

In another example, the block804may use correlation filters that are applied to or that use any of various types of feature descriptors or feature vectors, such as a histogram of oriented gradients (HOG) feature vectors, a SIFT (scale-invariant feature transform) descriptor, a SURF(speeded up robust features) descriptor, a GLOH (gradient location and orientation histogram) descriptor, etc. As an example, the HOG feature descriptor is a feature descriptor based technique typically used in computer vision and image processing for the purpose of detecting persons or objects in images. Generally, this technique counts occurrences of gradient orientation in localized portions of an image and is thus similar to that of edge orientation histograms, scale-invariant feature transform descriptors, and shape contexts, but differs in that it is computed on a dense grid of uniformly spaced cells and uses overlapping local contrast normalization for improved accuracy. While typically used for pedestrian detection in static images, it has also been used for human detection in videos, to detect a variety of common animals and vehicles in static imagery, etc.

The basic principle behind the HOG feature descriptor is that local object appearance and shape within an image can be described by the distribution of intensity gradients or edge directions. The image is divided into small connected regions called cells, and the descriptor technique compiles a histogram of gradient directions for the pixels within each cell. The descriptor is the concatenation of these histograms. For improved accuracy, the local histograms can be contrast-normalized by calculating a measure of the intensity across a larger region of the image, called a block, and then using this value to normalize all cells within the block. This normalization results in better invariance to changes in illumination and shadowing. The HOG feature descriptor has a number of key advantages over other types of feature descriptors. Firstly, because the HOG descriptor operates on local cells, this technique is invariant to geometric and photometric transformations, except for object orientation. Such changes would only appear in larger spatial regions. Secondly, coarse spatial sampling, fine orientation sampling, and strong local photometric normalization permits the individual body movement of objects to be ignored so long as these objects maintain a roughly upright position.

The first step of feature vector calculation in many feature detector techniques is to ensure normalized color and gamma values. However, this step can be omitted in the HOG descriptor computation, as the ensuing descriptor normalization essentially achieves the same result. Instead, the first step of the HOG calculation is to compute the gradient values. The most common method is to apply the one dimensional centered, point discrete derivative mask in one or both of the horizontal and vertical directions. This method may require filtering the color or intensity data of the image with filter kernels such as:[−1, 0, 1] and [−1, 0, 1]T.

The second step of the calculation is to create the cell histograms. Each pixel within the cell casts a weighted vote for an orientation-based histogram channel based on the values found in the gradient computation. The cells themselves can be, for example, rectangular or radial in shape, and the histogram channels may be evenly spread over 0 to 180 degrees or 0 to 360 degrees, depending on whether the gradient is “unsigned” or “signed”. As for the vote weight, a pixel contribution can either be the gradient magnitude itself, or some function of the magnitude. Other options for the vote weight could include the square root or square of the gradient magnitude, or some clipped version of the magnitude.

To account for changes in illumination and contrast, the gradient strengths can be locally normalized, which requires grouping the cells together into larger, spatially connected blocks. The HOG descriptor is then comprised of the concatenated vector of the components of the normalized cell histograms from all of the block regions. These blocks typically overlap, meaning that each cell or at least some cells contribute more than once to the final descriptor. Two main block geometries exist, including rectangular R-HOG blocks and circular C-HOG blocks. R-HOG blocks are generally square grids, represented by three parameters, i.e., the number of cells per block, the number of pixels per cell, and the number of channels per cell histogram. In some cases, four 8×8 pixel cells per block (16×16 pixels per block) with nine histogram channels is deemed to be useful. Moreover, minor improvement in performance may be gained by applying a Gaussian spatial window within each block before tabulating histogram votes so as to weight pixels around the edge of the blocks less. The R-HOG blocks appear quite similar to the scale-invariant feature transform (SIFT) descriptors. However, despite their similar formation, R-HOG blocks are computed in dense grids at some single scale without orientation alignment, whereas SIFT descriptors are usually computed at sparse, scale-invariant key image points and are rotated to align orientation. In addition, the R-HOG blocks are used in conjunction to encode spatial form information, while SIFT descriptors are used singly. On the other hand, circular HOG blocks (C-HOG) can be found in two variants, i.e., those with a single, central cell, and those with an angularly divided central cell. In addition, C-HOG blocks can be described with four parameters, i.e., the number of angular and radial bins, the radius of the center bin, and the expansion factor for the radius of additional radial bins.

In any event, the final step in object recognition using HOG descriptors is to feed the descriptors into some recognition system based on supervised learning. As an example, the support vector machine (SVM) classifier is a binary classifier which looks for an optimal hyper-plane as a decision function. Once trained on images containing some particular object (such as the training images ofFIG. 1), the SVM classifier can make decisions regarding the presence of an object, or landmark, in additional images. The features of the gradient descriptors may also be fed into neural network classifiers which may provide more accuracy in the classification as compared to other classifiers. The neural classifiers can accept the descriptor feature as the binary function or the optimal function.

Thus, the block804ofFIG. 8may, for example, apply or scan each of the correlation filters132across a set of pixels of the target vehicle image being analyzed (which may be all of the pixels of the target vehicle image) to determine to a correlation value for each such pixel or pixel area. The block804may select the pixel or area that has the highest correlation value or classifier value for a particular correlation filter as the most likely position of the landmark associated with the correlation filter and may determine that the pixel is to be detected as the landmark if the correlation value of that pixel is over or above a certain amount (e.g., close to one, such as greater than 0.8 or 0.9 or 0.95 or 0.98, etc.). The block,804may then store the landmark detection results in a memory (e.g., may store an indication if a landmark was detected and the position of the landmark in the target vehicle image for detected landmarks). The block804may apply each of the correlation filters132to each of the selected set of pixels in the target vehicle image to look for each landmark identified in the base object model502. It is possible, in many cases, that the block804may not be able to find some or all of the landmarks due to the fact that, for example, some of the landmarks may be located in regions of the vehicle that are damaged in a manner that masks or prevents the detection of the landmark using the correct correlation filter132. It may also be possible that some of the landmarks are not visible in the image being analyzed, resulting in a low correlation value from the correlation filter for that landmark at all pixel locations. Still further, in some cases, the block804might have access to some information about the target vehicle image being processed that may limit the number of landmarks searched and thus the number of correlation filters132applied to that particular target vehicle image. For example, if the perspective or view of the target object depicted in the target image is identified in some manner, for example, it is known that the target vehicle image being analyzed depicts the right front corner of the target vehicle, then the block804may not search landmarks associated with positions on the target vehicle that are not normally visible from that perspective (i.e., the right front corner of the vehicle) when analyzing this particular target vehicle image.

Of course, when the block804is finished analyzing a selected target vehicle image with a particular correlation filter (CF)132, a block806may select the next correlation filter132and the block804may then analyze the target vehicle image for the next landmark associated with the newly selected correlation filter132, and store the results of the landmark detection in memory (e.g., if a landmark was detected and, if so, the location of the landmark in the damaged vehicle image).

As will be understood, a correlation filter132may generally be created for each of the identified landmarks during the training stage of the image processing system (e.g., by the image processing training routines130), in a manner that enables the landmark to be identified by examining the various pixel values near, at or surrounding the landmark. However, the correlation filters132may also be created during run-time of the image processing routine140from the base object model or from an image of the base object model. Generally speaking, during landmark detection on a damaged or target vehicle image, the correlation filter (which may define a two dimensional pixel area) is applied at various points or pixels throughout the selected target vehicle image, moving the center of the correlation filter on a pixel-by-pixel basis for example, to determine the correlation or registration between the correlation filter pixels and the pixels of the damaged vehicle image to thereby determine the likelihood that the correlation filter is centered at a landmark associated with the correlation filter. This comparison process may be repeated for each pixel or for various sets of pixels, such as every five pixels of the image, etc., for each correlation filter132, computing for each position at which the correlation filter is applied to the target vehicle image the likelihood that the correlation filter132is centered over the landmark associated with the correlation filter. In some cases, different correlation filters may be generated for the same landmark, and these correlation filters may be different in size, for example, to provide for better correlation analysis. In this case, the block804may run a first pass through the target vehicle image using a smaller correlation filter (e.g., a 10 by 10 pixel filter) for a particular landmark, looking for candidate matches, and may run a second pass over the locations in the image that had a high correlation in the first pass using a larger correlation filter (e.g., a 100 by 100 pixel filter) for the same landmark. Any number of such passes or iterations may be made to provide for better correlation analysis in a shorter amount of time or with less processing power.

The output of the correlation analysis at each pixel or location may be a value between, for example, zero and one, indicating the likelihood of correlation between the pixels of the correlation filter and the pixels of the damaged vehicle image currently covered by the correlation filter, with a number near one meaning that there is a high likelihood of match, and with a number close to zero meaning that this is a low likelihood of match. Of course, the block804may scan through the entire target vehicle image with a particular correlation filter, and may review the results of each correlation analysis of a particular correlation filter to determine whether there is a likelihood that the target vehicle image pixel being analyzed is the landmark associated with the correlation filter. While this description indicates that the block804uses one or more correlation filters to perform landmark detection, the block804could use other types of filters instead, such as other types of difference filters. Still further, the creation and use of correlation filters is known in the art and is described in more detail in, for example, Boddeti, Vishnu, Takeo Kanade, and B. V. K. Kumar, “Correlation Filters for Object Alignment,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013), the contents of which is hereby expressly incorporate by reference herein.

After analyzing a particular target vehicle image for each of the landmarks using the various different correlation filters132, a block808may determine if enough landmarks have been detected within the target vehicle image to perform further change processing thereon. Generally speaking, it is desirable to pick or analyze target vehicle images with multiple landmarks within them, as well as to pick a set of target vehicle images of the target vehicle that includes many or most of the landmarks available for or that have been pre-identified for the vehicle. For example, if no or only a small set of landmarks (e.g., one or two) are found in the target vehicle image, then a block810may discard that image for use in change processing as the analysis performed on the image may be of limited or no usefulness. At this point, further processing on that image that would usually be performed by the rest of the routine800is skipped, and control is provided back to a block314ofFIG. 3(which picks a new target vehicle image to analyze). Moreover, if not enough target vehicle images with a sufficient set of landmarks viewable therein are available, the block810may also instruct a user to obtain more target vehicle images. The block810could perform this function if, for example, a significant number landmarks are not available for one particular view, or if there are areas of the automobile that have no landmark coverage or very little landmark coverage, or if there is simply not enough landmarks and images to be able to perform further processing.

However, if enough (e.g., three or more) landmarks are detected in the damaged vehicle image, than a block811may use the detected landmarks to estimate or determine the position of the missing landmarks in that view. In one case, the block811may determine, from the detected landmarks, the expected position of the missing landmarks based on the relative three-dimensional positioning of these same landmarks in the base object model. In one case, the block811may use three dimensional interpolation techniques to determine the position of one or more of the missing landmarks in the target vehicle image based on the position of known or detected landmarks in the target vehicle image and the position of missing landmarks as defined in the base object model. In another case, a partial Bayesian inference technique may be used to determine or estimate missing landmarks based on detected landmarks and the shapes of the vehicle or components as defined by these landmarks. Such a partial Bayesian inference technique is described in Li, Yan, Li Gu, and Takeo Kanade, “A Robust Shape Model for Multi-View Car Alignment,” Computer Vision and Pattern Recognition CVPR 2009, IEEE Conference on IEEE (2009), the contents of which are expressly incorporated by reference herein. Thereafter, a block812uses the detected landmarks (and potentially the estimated landmarks) to align the base object image with the damaged vehicle image.

For example,FIG. 10depicts the damaged vehicle image ofFIG. 7with a set of the possible landmarks being detected. In particular, 13 landmarks have been detected in the image ofFIG. 10. As will be seen inFIG. 10, not all of the landmarks on the front of the vehicle (as depicted in the base object model ofFIG. 9) were detected due to the damage to various locations of the vehicle, but landmarks at relatively undamaged locations were detected. Thereafter, the block811may estimate the position of the missing landmarks by interpolating or inferring the position of these landmarks based on the positioning of the detected landmarks, as noted above.

Next, using the detected (and potentially the estimated) landmarks as determined by the blocks808and811, the block812can determine the angle and distance at which the camera that took the picture of the target vehicle as depicted inFIG. 7was located using standard image processing techniques. Still further, the block812can correct for internal and external camera distortions. With the corrected landmark information, the block812rotates and scales the base object model to match the perspective of the target object as depicted in the target vehicle image. That is, the block812can rotate the base object model to the same angle (in three dimensions for example), and can scale (move towards or away from the object) to match the position of the camera that took the damaged vehicle image. Generally speaking, the block812may determine, from the detected and estimated landmarks in the target vehicle image being analyzed, the camera model for the image. The camera model may include the camera positioning with respect to the target object and camera distortion. One manner of determining and calibrating for camera effects in the target vehicle images is described in more detail in Heikkila, Janne, Olli Silvén, “A Four-Step Camera Calibration Procedure with Implicit Image Correction,” Computer Vision and Pattern Recognition, 1997; Proceedings, 1997 IEEE Computer Society Conference on. IEEE (1997), the contents of which are hereby expressly incorporated by reference herein.

As noted above, the base object model may be based on or incorporate a three dimensional model that may viewed or manipulated in the three dimensional viewing program to be rotated or viewed from any angle, position, and/or distance. Thus, to properly align the base object model with the target vehicle image, as taken by or depicted in a two dimensional photograph, the perspective view of the object in the three dimensional model should be aligned and positioned to be the same as the target object as viewed in the target object image, i.e., so that the base object model is viewed from the position and camera angle as the camera that took the target vehicle image being processed. The block812can perform a comparison of the positions of the pre-identified landmarks in the base object model with the detected (and estimated) landmarks within the target vehicle image being processed to determine the camera position, view, and orientation of the base image object that best matches the positioning of the landmarks as detected in the target vehicle image being processed. The block812can then rotate and scale the base object image to that camera position, view, and orientation, thereby aligning the base object model with the target object as depicted in the target vehicle image. In this manner, the block812can rotate and scale the three dimensional base object model to produce an image that is of the same scale and that is taken from the same location or position as the target vehicle image being processed. Of course, if the target object images are used to create a composite target object model, this target object model can be rotated to the same position as the base object model.

Thereafter, the block812can create or capture a two dimensional image of the base object from the base object model from the determined camera position, so that the image of the object created from the base object model is aligned with and scaled to be the same size as the target object as depicted in the target vehicle image. Of course, if desired, the block812could instead process the target vehicle image to rotate and scale that image to match a particular view or alignment of the base object model to thereby align the base object model, or an image from the base object model, with the target object within the target object image. Thus, for example, the block812could process (rotate and scale, for example,) target object images so as to align the target object in these images with the base object model at a particular orientation, such that the change detection and the following prediction analyses may be based on a common reference frame (such as a 3D model, un-wrapped 2D cardboard representation, etc.) Thus, for example, it may be desirable to rotate and/or scale the target object within the target object image to a particular point of view or scale so as to match the point of view or scale of the object to that of the images that were used to develop the CNNs134(or other change/damage detectors) for that object during the training stage.

Still further, a block814may, based on a comparison between the positions of the landmarks in the aligned image of the base object model and the detected landmarks in the target vehicle image, correct for camera distortions that were introduced by the camera that actually took the damaged vehicle image. More particularly, the block814may, based on a comparison of the detected and estimated landmarks in the target vehicle image and the landmarks in the aligned image from the base object model, determine one or more distortion transfer functions that define the various types of distortion introduced by the camera that took the damaged vehicle image. In general, the landmarks in the aligned base object image and the landmarks in the target object image may not match each other exactly because the alignment calculations may determine the best or most likely camera angle and position in a statistical manner (e.g., by determining the camera angle and camera position of the base object that produces an image having the smallest or least squared error between the various pairs of landmarks in the base object image and the target object image). These differences may be the result of internal camera distortion effects, such as lens distortion effects. The block814may determine or calculate the camera distortion function based on these differences and may then correct the target object image to correct for or to remove the internal camera distortions. As such, the block814uses these distortion functions to correct for the distortion introduced into the target vehicle image by the camera. For example, the block814may determine an inverse of the internal camera distortion function and apply this inverse across the image pixels of the target vehicle image being processed to correct for this distortion. On the other hand, the block814could apply the distortion function to the aligned image from the base object model to distort that image in the same manner as the target vehicle image was distorted.

When the block814has corrected the target vehicle image for distortion, the system now has a base object image (produced from the base object model) that is aligned with a distortion corrected image of the target vehicle as depicted in the target vehicle image, so that the objects within the images are depicted from the same view angle, and are of the same scale and size.

Referring back now toFIG. 3, a block310operates on the corrected target vehicle image and the aligned object image from the base object model to eliminate background effects and/or other non-vehicle related content from the target vehicle image. In particular, the block310may overlay the corrected target vehicle image with the aligned object image from the base object model to define the outer contours of the vehicle (target object) within the target vehicle image.FIG. 11illustrates such an overlay between a segmented aligned version of the object image generated from the base object model and the distortion corrected target vehicle image, in which the segmented object image from the base object model is illustrated in dotted line. Thereafter, all pixels in the damaged vehicle image that fall outside of one of the segments or components of the aligned base object model image are identified as background image pixels and can be set to a neutral pixel value, such as zero or dark. This action prevents these pixels from being considered to be part of the target image in future calculations.FIG. 12illustrates the corrected target vehicle image ofFIG. 7with background image pixels removed for further processing of the target vehicle image. As can be seen inFIG. 12, some of the background image pixels (designated generally at1208) remain in the image ofFIG. 12as these pixels fall within the outline of the aligned base object image when overlaid with the target vehicle image as depicted inFIG. 11. Moreover, there are areas of the hood body panel of the target vehicle ofFIG. 12actually fall outside of the outer contour of the hood body panel of the base object image which were thus designated as background pixels inFIG. 12. This area is depicted with the reference numeral1210inFIG. 11. In any event,FIG. 12illustrates the manner in which the body panels or segments of the target vehicle, as defined in the base object model, overlay with the target vehicle as depicted in the corrected target vehicle image to define the contours of each of the body panels or segments of the target vehicle depicted in the target vehicle image. The block310may thus create a background eliminated, segmented, corrected damaged vehicle image that depicts the target vehicle with the contours of the body panels or other segments of the target vehicle defined and with the background image pixels removed.

Next, a block312determines or detects damage to the target vehicle as depicted in the background eliminated, segmented, corrected target vehicle image (also referred to herein as the segmented target vehicle image) for each of the individual body components on a body panel by body panel basis. In particular, the block312may apply one or more convolutional neural networks134to the pixels of each body panel of the segmented target vehicle image to determine potential damage, or likelihood of damage to these body panels, as defined by the trained convolutional neural networks (CNNs)134.

As will be understood, the block312may use one or more CNNs134for each body panel or segment. As noted above, the CNNs134are statistically based processing tools created using the training set of vehicle image files122and training information files124ofFIG. 1to determine, based on known and quantified damage (or change) to the various body components of the same type of vehicle, the likelihood of damage to a vehicle depicted in an image not used in the training set of data. Thus, each CNN134can be used to identify damage (e.g., the likelihood of damage and/or a quantification of damage such as amount of damage or a type of damage) at each part (e.g., pixel, flat surface area such as the triangular surface segments ofFIG. 5, etc.) of the body panel or segment being analyzed. In general, each convolutional neural network134defines a two dimensional area that overlays the pixels of the target vehicle image surrounding a pixel of the target vehicle image being analyzed for damage or change (e.g., the target vehicle image pixel that registers with the center pixel of the CNN), with each of the surrounding pixels and the pixel being analyzed being inputs to the CNN.FIG. 13, for example, illustrates a rectangular CNN1302applied to a pixel associated with the hood body panel1304of the segmented target vehicle image ofFIG. 12. Each of the elements (pixels or inputs) of the CNN1302cover or overlay one of the pixels of the hood body panel1304with the CNN1302calculating or determining a damage parameter value for the pixel (shown in black or as a dark element) at the center of the CNN1302. The CNN1302applies a series of weights and calculations (such weights being determined during the training of the CNN) to the inputs of the CNN1302to produce a damage parameter value or damage determination for the target vehicle image pixel being analyzed. The damage parameter value may represent a likelihood of damage, an amount of damage, and/or an indication of the type of damage, for example. The block312may scan the CNN1302over each pixel or other area (e.g., a 2 by 2 set of pixels, a 4 by 4 set of pixels, a particular segmented or flat surface area as defined by the base object model, etc.) in turn to determine a damage parameter value for each such entity (pixel, surface area, surface segment, etc.). For example, as illustrated in FIG. 13 by the arrows, the block312may scan the CNN1302over the target vehicle image covering each pixel or other area to which the CNN1302is applied, in a raster manner, to produce a damage parameter value at each such location or pixel of the body panel. The damage parameter values may take a range of values, such as a value on a scale of 0 to 100 or 0 to 10, etc.

As will be understood, each CNN will have been trained based on training images and data to define or detect damage to a particular body component based on pervious examples of quantified damage to the same body component as depicted within the training images in which the damage was known and quantified. Moreover, it will be understood that more than one CNN may be applied to each body panel or segment, and that these different CNNs may be trained to detect different types of damage, for example. In some cases, for example, one CNN may be trained to detect scratches, another CNN may be trained to detect dents, another CNN may be trained to detect folds, etc. Of course, a CNN may be trained to detect any type of damage (or other change) as long as the type of damage or change is quantified as such in the training set of data or images.

The block312may thus apply any number of CNNs to a body panel visible in the target vehicle image to produce one or more damage parameter values for each pixel or other surface element of a body panel. Likewise, the block312may run one or more CNNs over each separate body panel visible in the segmented target vehicle image being processed to thereby analyze each or a number of different body panels of the target vehicle in each image.

Of course, as illustrated inFIG. 3, a block314may select a new or the next target vehicle image to be processed and the routine300may repeat the steps308to312for each such target vehicle image until all of the target vehicle images are analyzed. Moreover, the block314may, upon processing any particular target vehicle image or after all of the target vehicle images are processed, determine if enough useful images of the target vehicle or if images covering all of the body panels of the target vehicle which need to be analyzed have been obtained, and if not, may instruct a user to obtain more or new images of the damaged vehicle. This operation may include indicating the body panels that need to be viewable in the new images (because, for example, change processing could not be performed on that body panel using the current set of target vehicle images, because the analysis on these body panels was indeterminate or inconsistent based on the current set of target vehicle images, etc.).

While this description particularly indicates that the block312uses CNNs134to detect damage to the target vehicle, it should be noted that the block312could use other types of statistical processing techniques to detect or classify damage or changes to particular target object components. For example, the block312may use type of deep learning network technique or machine learning technique, which includes the use of CNNs or ConvNet, artificial neural networks (ANNs), support vector machine (SVM) techniques, random forests techniques, multiple/multivariable linear regression techniques, general/multivariate linear regression techniques, generalized linear model (GLM) regression techniques, deep belief net (DBN) techniques, stacked auto-encoder techniques, convolutional deep belief network techniques, feed-forward ANN techniques, multi-layer perceptron (MLP) techniques, to name but a few. Moreover, the CNNs134or other deep learning tools may provide other possible outputs including, for example, a probability of a patch having damage (e.g., a number between 0 and 1), an indication of one or more types of damage detected (e.g., creases, dents, missing parts, cracks, scuffs, scrapes, scratches, etc.), an indication of damage severity (e.g., different damage levels based on, for example, the amount of labor hours required to repair or replace the component), an indication of a probability of hidden damage (e.g., damage not visible in the target images), an indication of the age of damage (e.g., whether the damage is prior damage or not), an indication of a repair cost for each involved body panel, an indication of a final repair cost, a confidence level with respect to prediction accuracy, etc. Still further, the CNNs or other deep learning technique may use, as inputs, a full set of target vehicle photos (not local patches), telematics data, video data, geographic variation data, etc.

As illustrated inFIG. 3, after one or more of the target vehicle images have been processed, a block316may display the results of the analysis to a user. One manner of providing such results is through the use of a visual or graphical “heat map” which illustrates the target vehicle with indications of damage as determined by the routine300displayed on the image, for example, such as previously described with respect to the block210. A heat map may be created, for example, by converting the damage parameter values at each image pixel or surface segment to colors (or some other visual indication) and displaying these indications on the target vehicle image, to illustrate the damage or change determination at each location of the target vehicle or other target object. As such, a heat map may be included in, superimposed on, overlaid on, or otherwise applied to an image of a vehicle to thereby indicate changes (e.g., damages) at various locations on the target vehicle. In an example, a heat map is applied to a base model image corresponding to the target vehicle, such as to the image502shown inFIGS. 5 and 6. In another example, a heat map is applied to an image of the damaged target vehicle, such as to a raw or source image received at the block202(e.g., the image ofFIG. 7). In yet another example, a heat map is applied to a damaged target vehicle image that has been generated by any one or more of the image processing techniques described herein, such as to the image shown inFIG. 12.FIG. 14, for example, illustrates a heat map applied to the background eliminated, corrected target vehicle image ofFIG. 12, wherein areas of higher damage (or likelihood of damage) are indicated by darker colors. Of course, the block316may use colors to indicate damage value gradations. For example, brighter or darker shades of red may be used to indicate areas, locations, or points of higher damage or higher amounts and/or greater severity of damage, or higher likelihoods of actual, occurred damage, with other shades of red or other colors used to illustrate areas with less damage or with less likelihood of damage. In cases in which the routine300determines types or probable types of damage, colors, icons, or other indicia may be used to indicate the type of damage (e.g., scratches, folds, dents, bends, etc.).FIG. 15illustrates an example set of heat maps for the target vehicle ofFIG. 14showing an analysis of each of the four corner images of the target vehicle as determined from target vehicle images. In any event, a heat map may illustrate or otherwise represent damage parameter values or other visual representations of the CNN output values in conjunction with the image of the damaged or target vehicle using different colors or other visible indicia superimposed onto the image of the target vehicle (or the object model image of the target vehicle), to define or illustrate the presence, severity, type, and/or likelihood of damage at each component, pixel, surface segment, or other part of the damaged component of the target vehicle. That is, the damage parameter values generated by the CNNs may be respectively mapped to particular colors and/or other characteristics utilized in the heat map. One or more heat maps may be included in, superimposed or overlaid onto, and/or otherwise applied to an image of a vehicle to thereby indicate damage parameter values corresponding to various locations on the vehicle. Additional discussion of heat maps is provided in a later section.

Moreover, as illustrated inFIG. 3, a block318may use the heat map (e.g., as generated at the block210ofFIG. 2) and/or may use the damage parameter values generated by the CNNs134(e.g., as generated at the block208ofFIG. 2) for each body panel to determine which body panels or other parts of the target vehicle are damaged, the type of damage, the extent of the damage, etc. The block318may analyze the damage parameter values from multiple images for the same body panel to determine the amount of damage to the entire panel. Based on this determination, the block318may determine whether an entire body panel needs to be replaced, whether the body panel needs to be repaired, the type and extent of repair that needs to be performed on the body panel, etc. In some cases, the block318may perform an analysis for each of the body panels within the target vehicle based on a single image or based on multiple images. Still further, the evaluation or determination of damage to the same body panel or body component from different images may be averaged, added or otherwise mathematically associated with one another to determine an overall evaluation of the damage to the body panel or component. This coordination may be necessary because, in some cases, the complete damage to a particular body panel may not be viewable or completely viewable from a single target vehicle image, and may instead only be determined from multiple different target vehicle images.

As illustrated inFIG. 3, a block320may then use the damage determination (e.g., the damage parameter values generated by the CNNs and/or the heat map) for secondary purposes. Such secondary considerations or evaluations may be related to or dependent on the damage determination, such as previously discussed with respect to the block212ofFIG. 2. For example, the block320may use the damage determination to determine each of the parts of the target vehicle that needs to be replaced, each of the parts that need to be repaired (and potentially the type of repair), etc. The block320may identify particular types of work that need to be performed with respect to various ones of the body panels, such as repainting, replacing, filling creases or gaps, etc. Still further, the block320may use the identification of the damage and/or of the types of damage, and/or amount of damage to determine an estimate of the repair costs associated with repairing each of the body components or panels, including costs associated with replacing such panels, the costs of repairing such panels, etc. For example, the identification of the damage and/or the types of damage detected for the target vehicle may be provided to a system and/or a method that automatically predicts and recommends parts, materials, and services needed for repairing the target vehicle, such as the systems and methods described in co-owned U.S. Pat. No. 9,218,626 entitled “Automatic Prediction and Recommendation of Parts, Materials, and Services for Vehicle Insurance Claim Estimates and Supplements,” the contents of which are hereby expressly incorporated by reference herein.

In one case, the quantification of damage as determined, for example, from the blocks312,316, and/or318, may be applied as an input to a statistically based estimating program or predictive model, such as an estimating program or predictive model that has been generated by one or more regression analyses performed on stored training data, which may include historical data related to damaged vehicles. The quantification of damage is applied as an input to the estimating program or predictive model to estimate and/or predict various parameters related to repairing the target vehicle, such as the cost of repair, the time it takes to repair the vehicle, the likelihood of a re-inspection of the vehicle, an insurance claim cost, etc. The statistically based estimating program or predictive model may be a program, routine, or model such as that described in co-owned U.S. patent application Ser. No. 12/792,104, entitled “Systems and Methods of Predicting Vehicle Claim Cost” and filed on Jun. 2, 2010; co-owned U.S. patent application Ser. No. 14/218,165, entitled “System and Method of Predicting Vehicle Claim Information Based on Image Attributes” and filed on Mar. 18, 2014; co-owned U.S. patent application Ser. No. 14/460,907, entitled “System and Method of Predicting Vehicle Claim Information based on Image Attributes” and filed on Aug. 15, 2014; co-owned U.S. patent application Ser. No. 14/168,345, entitled “System and Method of Predicting a Vehicle Claim Supplement between an Insurance Carrier and a Repair Facility” and filed on Jan. 30, 2014; co-owned U.S. patent application Ser. No. 14/168,327, entitled “Systems and Methods of Predicting Vehicle Claim Re-Inspections” and filed on Jan. 30, 2014; co-owned U.S. patent application Ser. No. 14/218,148, entitled “System and Method of Predicting Vehicle Claim Information based on Deformation Images” and filed on Mar. 18, 2014; co-owned U.S. patent application Ser. No. 15/016,756, entitled “Imaging Processing System for Identifying Parts for Repairing a Vehicle” and filed on Feb. 5, 2016; co-owned U.S. patent application Ser. No. 15/079,356, entitled “Image Processing System for Vehicle Damage” and filed Mar. 24, 2016; and/or co-owned U.S. patent application Ser. No. 15/079,380, entitled “Image Processing System for Vehicle Damage” and filed on Mar. 24, 2016, the contents of each of which are hereby expressly incorporated herein.

In some implementations, the statistically based estimating program or predictive model to which the quantification of damage is provided as input is generated based on historical data related to damaged vehicles. For example, the estimating program or predictive model may have been generated a priori (or, in some cases, may be generated in real-time or on-demand) based on one or more regression analyses, principal components analyses, cluster analyses, etc. of the historical data. The historical data related to damaged vehicles is stored in a data storage device or system which may be accessible to the image processing system100, and may include vehicle repair cost data, repair time data, re-inspection data, parts data, imaging data, insurance claim data, telematics data, etc. for various types of damage and/or previous repairs to the vehicles of the same make, model, year, type, etc. Further, the regression or other analyses that are utilized to generate the estimating program or predictive model can be run using many factors that account for many different variables that might affect the costs of repair or replacement, such as the city or state in which the vehicle is located, the type of vehicle, the identity of the insurance carrier or company paying for the components (which may have different methodologies of obtaining parts or components, such as using only new parts, using only replacement or used parts where possible), etc. At any rate, the estimating program or predictive model that receives the quantification of damage of the target vehicle as input may determine, based on the input, an estimate of damage or repair costs for the target vehicle, the likelihood that a component needs to be replaced, repainted, etc., the costs of such a repair or replacement, the time needed for repair or replacement, etc. In some scenarios, additional data corresponding to the target vehicle, such as telematics data associated with the damaged vehicle, is provided to the estimating program or predictive model as input in conjunction with the quantification of damage.

With further regard to the blocks318,320ofFIG. 3and the block212ofFIG. 2, a flow diagram of an example method1600for using an imaging processing system for repairing a damaged vehicle is depicted inFIG. 16. The method1600may be performed by any one or more portions of the image processing system100ofFIG. 1, the system200ofFIG. 2, or other systems. Additionally or alternatively, the method1600may operate in conjunction with the heat maps described inFIGS. 14 and 15, and/or with any one or more of the other methods described herein, or with other methods.

At a block1602, the method1600includes obtaining one or more source images of a target vehicle that has been damaged, e.g. in a manner such as that previously discussed with respect to the block302ofFIG. 3. For example, at the block1602, the one or more source images of the target vehicle may be communicatively received via a link or communication interface, the one or more images may be obtained from a data storage area, etc. The obtained images may include two-dimensional images and/or three-dimensional images.

At a block1605, the method1600includes applying an image processor to the source images to thereby generate a set of values corresponding to the damage of the target vehicle (e.g., to generate a set of damage parameter values). For example, one or more of the blocks202-208of the system200may be applied to the obtained source images of the damaged vehicle to determine occurrences of damage at various locations, points, or areas of the vehicle, and to generate a set of damage values corresponding to the determined damage. Each value included in the generated set of values may respectively correspond to respective damage at a respective location of the damaged vehicle. A particular damage value may correspond to a degree of severity of respective damage at a particular location on the target vehicle, a level of accuracy of the degree of severity of the respective damage at the particular location on the target vehicle, a type of damage at the particular location on the target vehicle, or a likelihood of an occurrence of any damage at the respective location on the target vehicle, for example.

In an example implementation, applying the image processor to the obtained source images of the target vehicle (block1605) includes evaluating the obtained source images and selecting one or more suitable images therefrom based on the evaluation, e.g., in a manner such as that previously discussed with respect to the block304. For example, the image collector/selector/evaluator202ofFIG. 2may evaluate a set of obtained images and select a subset of source images that are suitable for determining damage to the target vehicle.

In some configurations, the image processor that operates on the source images (block1605) includes a damage determiner (e.g., the block208ofFIG. 2) that applies one or more convolutional neural networks to one or more of the obtained source images of the target vehicle, or to a composite image of the target vehicle generated from the obtained source images to detect or determine damage at the target vehicle, and the set of values corresponding to the damage at the target vehicle are output by the one or more convolutional neural networks. For example, one or more convolutional neural networks are applied to one or more source images of the target vehicle in a manner such as previously discussed with respect to the block312ofFIG. 3. Each convolutional neural network may be applied to a different segment, panel, component, or portion of the target vehicle as represented or depicted within the vehicle image(s), if desired. In some scenarios, the block1605includes selecting a particular convolutional neural network that is to be applied to a particular segment of the target vehicle or representation thereof included within the obtained source images.

In some scenarios, applying the image processor to the obtained images (block1605) includes generating a composite image of the target vehicle from a plurality of obtained source images of the target vehicle, such as by using the composite image generator204ofFIG. 2. Additionally or alternatively, in some scenarios, applying the image processor to the obtained source images (block1605) includes aligning the composite image of the target vehicle with a base model image of vehicle to determine the damage and locations of the damage at the target vehicle, such as by using the image aligner206ofFIG. 2, and/or in a manner such as previously discussed with respect to the block308ofFIG. 3. For instance, at least some of the plurality of source images are two-dimensional images, and the composite image and the base model image are three-dimensional images.

In some scenarios, applying the image processor to the obtained images (block1605) includes generating a heat map that corresponds to damage to the target vehicle that has been determined by the image processor. For example, the image processor that operates on the obtained images (block1605) includes a heat map generator, such as the heat map generator210ofFIG. 2, which may generate the heat map in a manner such as previously discussed with respect to the block316ofFIG. 3.

At a block1608, the method1600includes determining or identifying, based on a particular value included in the set of damage parameter values generated by the image processor, one or more parts that are needed to repair the vehicle. The particular damage parameter value operated on at the block1608, may be directly obtained (e.g., via the change determiner208), or may be obtained from a generated heat map (e.g., via the heat map created by the heat map generator210). In an example implementation, in order to determine or identify the one or more parts are needed to repair the vehicle (block1608), the particular location corresponding to the obtained, particular damage parameter value is determined, and a larger or more comprehensive set of parts for the target vehicle is filtered (e.g., by using one or more of the filters218ofFIG. 2) based on the particular determined location, based on the particular component or segment of the target vehicle on which the particular location is disposed, and/or, in some cases, based on other components or segments of the vehicle that are adjacent to the particular component or segment. Examples of systems and methods for filtering a set of vehicle parts and making recommendations of parts therefrom based on detected damaged are described in aforementioned U.S. Pat. No. 9,218,626, which may operate in conjunction with any one or more portions of the present disclosure, including any one or more portions of the method1600.

One or more particular parts included in the filtered set of parts may be selected at the block1608, e.g., based upon the value of the particular damage parameter corresponding to the particular location. For instance, if the particular damage value indicates a severity of damage greater than a threshold, all parts in the filtered set may be selected. In another instance, if the particular damage value is within a certain range, a subset of the filtered set of parts may be selected. The selection of a specific part or a specific group of parts from the filtered set may be based upon the particular damage value as compared to one or more thresholds and/or to one or more ranges, for example. In some cases, the selection of a specific part or specific group of parts is additionally or alternatively based on respective damage values corresponding to other locations of the particular component or segment of the vehicle on which the particular location is disposed, and/or is based on respective damage values corresponding to one or more components or segments on the target vehicle that are adjacent to the particular component or segment on which the particular location is disposed. The filtering and selecting to determine/identify the one or more parts needed to repair the vehicle may be performed integrally or as sequential procedures, as desired. In some scenarios, filtering and selecting to determine/identify the one or more parts is performed iteratively.

In some situations, in addition to or as an alternative to filtering and selecting to determine/identify the one or more parts are needed to repair the vehicle, determining or identifying the one or more parts that are needed to repair the vehicle (block1608) includes predicting one or more parts that are needed to repair the vehicle. In an example, one or more of the damage values generated by the image processor and/or at least a portion of the heat map generated by the image processor (block1605) are provided to a predictive parts model that has been generated based on historical imaging data, parts data, telematics data, and/or repair data for a plurality of vehicles (e.g., one of the one or more of the predictive models215ofFIG. 2). The predictive parts model may have been generated a priori (or may be generated in real-time or on-demand) based on one or more regression analyses and/or other types of statistical analyses performed on the historical data, for instance. The predictive parts model receives one or more damage parameter values and/or the heat map as inputs, and generates therefrom as an output an indication of the one or more parts needed to repair the vehicle. Examples of the creation and usage of such predictive parts models may be found, for example, aforementioned U.S. patent application Ser. Nos. 14/218,165; 14/460,907; 14/218,148; 15/016,756; 15/079,356; and Ser. No. 15/079,380, any number of which may operate in conjunction with any one or more portions of the present disclosure, including any one or more portions of the method1600.

With further regard to the block1608, in some implementations, determining or identifying one or more parts needed to repair the vehicle includes selecting one or more parts that are needed to repair the vehicle based on an availability of parts, either individually and/or as an integral unit. For example, consider a scenario in which the set of damage parameter values generated at the block1605indicates that a driver door, driver door handle, driver door window, and mechanism for raising and lowering the window are damaged and need to be replaced. In this scenario, the block1608may include determining a respective availability of each of the individual parts, e.g. each of the driver door, driver door handle, driver door window, and the mechanism for raising and lowering the window. If one or more of these individual parts are not currently available or not available within a desired window of time, instead of selecting each part individually, the block1608may select an integral door unit that includes therein the handle, the window, the mechanism for raising and lowering the window, as well as other parts. In a similar manner, the block1608may additionally or alternatively include selecting one or more parts based on cost. For example, if the sum of the individual costs for the driver door, driver door handle, driver door window, and mechanism for raising and lowering the window is greater than the cost of integral door unit that includes the individual parts, the block1608may select the integral door unit instead of the individual parts.

At any rate, at a block1610, the method1600includes generating an indication of the one or parts needed to repair the vehicle, e.g., as determined/identified at the block1608. At a block1612, the indication is provided to at least one of a user interface or to another computing device. In some scenarios, providing the indication of the one or more parts needed to repair the vehicle (block1612) includes ordering the one or parts needed to repair the vehicle. For example, the indication of the one or more parts needed to repair the vehicle may be electronically transmitted to a parts ordering system, a parts procurement system, or a parts store.

In some scenarios, the method1600optionally includes determining one or secondary considerations (block1615), e.g., based on one or more values included in the set of values generated by the image processor (block1605), and/or based on the one or more parts that have been determined, identified, and/or selected (block1608). Examples of secondary considerations include the cost of repairing the damaged vehicle and/or of repairing a portion thereof, a likelihood that a part or component needs to be replaced, repainted, etc., an estimated time for completing the repair or replacement, an identification of the types of work that need to be performed to repair and/or replace one or more parts or components (e.g., repainting, replacing, filling creases or gaps, etc.), a likelihood of an occurrence of a re-inspection, an insurance cost corresponding to the damaged vehicle, and/or other secondary considerations. Examples of secondary considerations that are related to insurance costs corresponding to the damaged vehicle include a cost of an insurance claim, an out-of-pocket cost for an insured party, a cost of a supplement to the insurance claim, and/or other insurance costs, such as those described in aforementioned U.S. patent application Ser. Nos. 12/792,104; 14/218,165; 14/460,907; 14/168,345; 14/168,327; 14/218,148; 15/016,756; 15/079,356; and Ser. No. 15/079,380.

In some implementations, determining the one or more secondary considerations (block1615) includes predicting the one or more secondary considerations, such as in a manner previously discussed with respect to the block212ofFIG. 2and/or the block320ofFIG. 3. In an example, the block1615includes providing an indication of one or more damage parameter values and/or of one or more portions of the heat map generated by the image processor (block1605) to a predictive model generated based on one or more statistical analyses (e.g., one or more regression analyses, principal components analyses, etc.) performed on historical claim data, cost data, repair time data, re-inspection data, parts data, imaging data, etc. for a plurality of vehicles. For instance, the predictive model is one of the predictive models215ofFIG. 2. In some embodiments, the indication of one or more determined/identified/selected parts (block1608) is provided to the predictive model. The predictive model receives the indication of the one or more damage values, the one or more portions of the heat map, and/or the indication of the one or more parts as input, and outputs a resulting indication of predicted or estimated one or more secondary considerations, for example. Examples of the creation and usage of such predictive models may be found, for example, in aforementioned U.S. patent application Ser. Nos. 12/792,104; 14/218,165; 14/460,907; 14/168,345; 14/168,327; 14/218,148; 15/016,756; 15/079,356; and Ser. No. 15/079,380.

In an optional block1618, the method1600includes providing an indication of the determined secondary consideration(s) to a user interface and/or to another computing device.

It is noted that, in a preferred implementation, the method1600is performed without utilizing (e.g., exclusive of) any user input. That is, aside from optionally providing an initial set of source images of the damaged vehicle for image processing (block1602), other blocks (and in some cases, all blocks) of the method1600are performed by an image processing system, such as the image processing system100ofFIG. 1or the system200ofFIG. 2, and not by a human being or person.

Now turning again to heat maps (such as those previously described with respect to the block210, the block316, and elsewhere within the present disclosure),FIG. 17depicts a flow diagram of an example method1700for providing vehicle damage information via a heat map on a user interface and/or a display. The method1700may be performed in conjunction with any number of portions of the image processing system100ofFIG. 1, the system200ofFIG. 2, the heat maps described inFIGS. 14 and 15, and/or any one or more of the methods described herein.

At a block1702, the method1700includes generating a heat map corresponding to vehicle damage. The heat map includes a plurality of different colors, different gradations of colors, and/or other visual indicators, each of which represents or is indicative of a particular value corresponding to damage or change to a vehicle. For example, each different color, color gradation, and/or other visual indicator may represent or be indicative of a respective amount of damage, a level of accuracy of an estimated or predicted amount of damage, a type of damage, and/or a probability or likelihood of an occurrence of damage at a respective location, point, or area of or on the vehicle.

At a block1705, the method1700includes applying the heat map to an image corresponding to a target vehicle. The image corresponding to the target vehicle to which the heat map is applied (block1705) may be an image of the damaged target vehicle (such as a raw or source image as illustrated inFIG. 7, or a composite image generated via image processing as illustrated inFIG. 12). Alternatively, the image corresponding to the target vehicle to which the heat map is applied (block1705) may be a base model image of a vehicle that corresponds to the target vehicle, but does not include the damage incurred by the target vehicle (e.g., the base model image as illustrated inFIG. 5). Further, the image of the target vehicle to which the heat map is applied (block1705) may be a two-dimensional image or may be a three-dimensional image. For ease of reading, such an image to which a heat map is applied (block1705) (whether raw, source, or composite; and whether two-dimensional or three-dimensional) is generally referred to herein as a “target vehicle image” or an “image of a target vehicle.”

Applying the heat map to the image of the target vehicle (block1705) may be performed using any suitable technique, such as superimposing the heat map on the image of the target vehicle, overlaying the heat map onto the image of the target vehicle, incorporating the heat map into the image of the target vehicle, etc. At any rate, the heat map is applied to the target vehicle image (block1705) so that a respective color or characteristic at respective location on the heat map corresponds to respective damage at a corresponding location on the target vehicle.

For scenarios in which the image of the target vehicle to which the heat map is applied (block1705) is a composite image of the target vehicle, the method1700may optionally include generating the composite, target vehicle image (block1708). In an example, the method1700includes generating a composite, three-dimensional image of the target vehicle (block1708) from a plurality of two-dimensional raw or source images of the target vehicle. Additionally or alternatively, generating the composite image of the target vehicle (block1708) includes aligning one or more raw or source images of the target vehicle with a base model image corresponding to the target vehicle. Generally speaking, the image of the target vehicle may be generated (block1708) by using any one or more of the image processing techniques described herein, such as those previously described with respect to the blocks202-208ofFIG. 2.

In some scenarios, the method1700optionally includes generating values of damage or change parameters corresponding to the target vehicle (block1710). Generally, the values of damage parameters corresponding to the target vehicle may be generated by performing image processing on the target vehicle image. For example, the damage parameter values corresponding to the target vehicle may be generated by performing change detection image processing of the target vehicle image in which the target vehicle image is compared with a base model image corresponding to the target vehicle, e.g., in a manner such as previously described with respect to block208. In one of many implementations, one or more convolutional neural networks may be applied to one or more portions of the target vehicle image to detect the existence of change, the likelihood of the existence of change, the degree of change, and/or a quantification of change (e.g., damage) to the target vehicle with respect to the base model image. The values of the damage or change parameters corresponding to the target vehicle may be determined based on the output(s) of the CNN(s) applied to the target vehicle image, for example.

For scenarios in which the method1700includes the block1710, the damage parameter values generated at the block1710are provided to the block1702for generating the heat map corresponding to the target vehicle, as illustrated inFIG. 17. For example, generating the heat map of the target vehicle (block1702) includes mapping one or more damage parameter values generated at the block1710to a particular color, gradation of color, and/or other visual indicator, which may then be applied to particular pixels or locations on the heat map to visually indicate respective values of the one or more damage parameters corresponding to respective locations on the target vehicle.

At a block1712, the method1700includes presenting the image of the vehicle with the heat map on a user interface or display, thereby providing damage information corresponding to various locations on the vehicle on the user interface or display. For example, each respective color of the heat map at a respective location on the image of the target vehicle is indicative of respective damage information corresponding to a respective location on the target vehicle itself.

In some scenarios, the image of the vehicle presented with the heat map on the user interface (block1712) may include one or more user controls (not shown). In these scenarios, a user may be able to select a particular user control corresponding to a particular location on the displayed heat map. For example, a user may click on or hover over a desired particular location on the displayed heat map. Upon reception of the user selection of the particular location on the heat map, the method1700includes presenting additional information corresponding to the particular location of the target vehicle indicated by the user selection. The additional information may include, for example, a quantitative indication of a degree of severity of damage at the particular location of the target vehicle, a quantitative indication of an accuracy of a determined occurrence of the respective damage at the particular location of the vehicle, an indication or identification of particular types of damage at the particular location of the vehicle, an identification of one or more vehicle parts that are damaged and that correspond to the particular location of the vehicle, an identification of at least one vehicle part that corresponds to the particular location of the vehicle to order for repairing the vehicle, and/or other information corresponding to respective damage at the particular location of the vehicle. The additional information may be presented at the user interface or display in a pop-up window or other suitable presentation format.

In some scenarios, presenting the heat map at the user interface or on the display (block1712) includes presenting, at the user interface or on the display in conjunction with the heat map, a user control which, when activated, launches an application that utilizes at least some of the information that corresponds to the target vehicle and that is indicated by the heat map (not shown). In an example, the launched application, when executed, determines one or more secondary characteristics and/or features associated with the damaged vehicle based on the provided vehicle damage information, such as one or more cost estimates associated with repairing or replacing the damaged vehicle or parts of the damaged vehicle, an estimate of the time it may take to repair the damaged vehicle, a change in state of the damaged vehicle or a part thereof, etc. In some cases, the launched application performs additional post-processing on the determined secondary considerations. For example, the launched application may utilized the determined secondary considerations to estimate insurance settlements, estimate insurance payouts, order parts for pairing the damaged vehicle, and/or perform other post-processing on secondary consideration data generated by the image processing system100. Additional details on post-processing of data generated by the image processing systems and/or methods of the present disclosure are discussed elsewhere herein with respect to block212ofFIG. 2.

As previously discussed, in some scenarios, generating a heat map corresponding to the damaged vehicle (block1702) is based on performing one or more image processing techniques, e.g., by utilizing one or more of the image processing techniques discussed with respect to blocks202-208ofFIG. 2. Accordingly, in some embodiments of the method1700(not shown), the method1700may include determining a level of quality of the heat map generated at the block1702, and modifying the image processing of one or more images of the vehicle based on the determined level of quality of the generated heat map. The modifications may include, for example, selecting additional or alternative raw or source images of the vehicle for image processing, prompting the user to provide additional or alternative raw or source images of the vehicle, adjusting the size of a particular image area of a vehicle segment to which a CNN is applied (e.g., change from a 16 pixel square to a four pixel square), adjusting the scan distance between subsequent image areas to which a CNN is applied (e.g., change from scanning every fifth pixel to scanning every second pixel), and/or other modifications to the image processing techniques performed at one or more of the blocks202-208. A revised image of the damaged vehicle may be generated using the image processing modifications, and a revised heat map corresponding to the damaged vehicle may be generated based on the revised image of the damaged vehicle. The determination of the level of quality of the heat map, the determination of the particular modifications that are to be made to the image processing of the images of the vehicle, and/or the execution of the particular image processing modifications may be automatically performed without any user input, e.g., may by embodiments of the image processing systems and/or methods described herein.

FIG. 18illustrates a flow diagram of a method1800of analyzing images using image models, and facilitating various functionalities associated therewith. The method1800may be facilitated by one or more components of a system. For example, the method1800may be performed by any combination of the computing devices106, the image processing routines, the training routines, and/or other components of the image processing system100as discussed with respect toFIG. 1. For purposes of explanation, the method1800will be described as being performed by a server. It should be appreciated that the depicted functionalities of the method1800are merely exemplary and that additional or alternative functionalities are envisioned.

The method1800may begin when the server optionally accesses (block1805) telematics data associated with a target vehicle. In embodiments, the server may receive the telematics data from an electronic device associated with the target vehicle (e.g., an on-board electronic device of the target vehicle or an electronic device of an operator of the target vehicle such as one of the user interface devices102as discussed with respect toFIG. 1), or may receive or access the telematics data from another source. The server may also optionally identify or estimate (block1810) any damage to the target vehicle from the telematics data. In particular, the server may analyze the telematics data to identify any specific damage to the target vehicle as well as characteristics thereof, such as a degree of damage, locations of the damage, and/or other information.

Based on the telematics data, the server may optionally determine (block1815) whether a first notice of loss (FNOL) event has occurred, where the FNOL event may represent a partial or full loss of an insured product such as the target vehicle. In an implementation, the server may receive a notice of FNOL separate from accessing or analyzing the telematics data, such as by receiving a notice from an insurance provider or from an individual associated with the target vehicle. Otherwise, to determine whether an FNOL event has occurred, the system may reference the damage assessment of block1810.

Generally, if the telematics data indicates damage or loss to the target vehicle, then the server may deem that an FNOL has occurred (“YES”), and processing may proceed to block1820. In contrast, if the telematics data does not indicate damage or loss to the target vehicle, then the server may deem that an FNOL has not occurred (“NO”), and processing may end or proceed to other functionality. In an implementation, the server may automatically and consistently poll for updated telematics data to determine whether an FNOL has occurred.

At block1820, the server may optionally request an electronic device to transmit a set of images. It should be appreciated that the electronic device may be the same device as or a different device from the device from which the server received the telematics data. In an implementation, the server may request the user interface device102as discussed with respect toFIG. 1to transmit the set of images. In embodiments, the request may include information for an individual to reference in association with capturing the set of images, including which views or perspectives of the target vehicle are needed (e.g., images taken of the target vehicle from each of the “four corners”), which vehicle components need to be depicted (e.g., hood, trunk, bumper, doors, etc.), and/or other information.

The server may receive (block1825), from the electronic device via a network connection, a set of images depicting the target vehicle. In embodiments, the server may receive the set of images regardless of whether the server accesses or receives any telematics data associated with the target vehicle. In an implementation, the individual associated with the target vehicle may use the electronic device to capture the set of images and transmit the set of images to the server. An application that may be installed on the electronic device may guide the individual through the image capture, such as by displaying a set of interfaces and instructions associated therewith that instruct the individual to capture any needed image perspectives and vehicle components.

The server may examine the set of images and identify (block1830) at least a portion of the set of images, where the portion of the set of images may include or indicate a certain characteristic(s) or quality(ies). In particular, the portion of the set of images may, in comparison to the whole of the set of images, be of higher quality, depict certain vehicle components, be captured from certain perspectives, and/or have other relevant characteristics or qualities. The server may also identify (block1835) the target vehicle is that is depicted in the portion of the set of images. In identifying the target vehicle, the server may use any image recognition or analysis technology to ascertain the make, model, and/or year of the target vehicle.

The server may also identify (block1840), from the portion of the set of images, a set of information associated with the target vehicle and/or associated with the portion of the set of images themselves. In an embodiment, the server may analyze the portion of the set of images to determine or identify any damage to the target vehicle that is depicted in the portion of the set of images. In the same or an alternative embodiment, the server may analyze the portion of the set of images to identify a general image quality, based on factors such as sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, exposure accuracy, lateral chromatic aberration, lens flare, color moiré, artifacts, and/or other factors. The server may also identify additional information from the portion of the set of images, as later discussed.

In an implementation, if the server does not access any telematics data in block1805, processing may proceed to “A” as discussed with respect toFIG. 18B. In contrast, if the server accesses telematics data in block1805, the server may optionally determine (block1845) whether the damage indicated in the telematics data (as determined in block1810) is depicted in the portion of the set of images. In an implementation, the server may determine whether a threshold percentage of the damage indicated in the telematics data is depicted in the portion of the set of images. In another implementation, the server may determine whether there are any discrepancies between the damage indicated in the telematics data and the damage depicted in the portion of the set of images.

For example, both the telematics data and the portion of the set of images may indicate damage to the front bumper of the target vehicle, in which case the server may deem that the depicted damage matches the indicated damage. For further example, the telematics data may indicate damage exclusively to the front bumper without any damage to the passenger's door (but that the telematics data would otherwise detect such door damage), and the portion of the set of images may depict damage to both the front bumper and the passenger's door, in which case the server may deem that the depicted damage does not match the indicated damage. If the server determines that the damage matches (“YES”), processing may proceed to “D” as discussed with respect toFIG. 18B. If the server determines that the damage does not match (“NO”), processing may proceed to “B” as discussed with respect toFIG. 18B.

FIG. 18Bdepicts additional functionality of the method1800as discussed with respect toFIG. 18A.FIG. 18Bmay begin at “A” at which the server may identify (block1850) a set of perspectives included in the portion of the set of images depicting the target vehicle, where the set of perspectives may correspond to vantage points from which the portion of the set of images were captured. For example, the set of perspectives may be the “four corners” (i.e., “front right,” “front left,” “back right,” and “back left”). It should be appreciated that alternative or additional perspectives are envisioned. The server may also identify (block1855) a set of components of the target vehicle that may be depicted in the portion of the set of images. According to embodiments, the set of components may include any element or component of the vehicle that may be depicted in the portion of the set of images including, for example, a front bumper, a hood, headlights, wheels and tires, a back bumper, a trunk, a windshield, a sunroof, front and back windows, a tailgate, doors, a steering wheel, a dashboard, a grill, seats, body panels, and/or other components.

The server may access (block1860) a base image model corresponding to the target vehicle. In particular, the server may access a base image model that corresponds to the make, model, and/or year of the target vehicle that was identified in block1835. Generally, the base image model may depict a base vehicle in its pristine or pre-changed condition, where the base vehicle may have the same make, model, and/or year of the target vehicle. In embodiments, the server may remotely or locally access the base image model, such as via a local database or via a remote storage entity.

The server may also analyze (block1865) the portion of the set of images based on the base image model. In embodiments, the server may determine the need for additional images depicting the target vehicle or additional information associated with the target vehicle. In particular, the server may determine whether any additional perspective(s) is needed, whether any components are not depicted in the portion of the set of images, whether the general image quality is too low, and/or whether the portion of the set of images is deficient in another regard(s). In some situations, the server may optionally ask for or request zoom-in image(s) of a damaged or changed area of the target vehicle to further identify and quantify the damage type/severity, or could ask for or request zoom-out image(s) if not enough of the target vehicle is depicted in a particular image or if damage spans to the edge of the image.

Generally, in analyzing the portion of the set of images, the server may calculate or determine a general quality and completeness of the portion of the set of images, such as by accounting for any need for additional images. The server may then determine (block1870) whether the determined quality or completeness of the portion of the set of images at least meets a given threshold criteria. Generally, the threshold criteria may account some combination of a sufficient amount and type of image perspective(s), a sufficient amount and type of identified vehicle components, a sufficient image quality, and/or other characteristics. For example, a threshold criteria for a given base image model may specify that “front right,” “back right,” “front left,” and “back left” image perspectives are required, and that a vehicle hood and four (4) doors must be depicted in the portion of the set of images. It should be appreciated that the threshold criteria may vary depending on the base image model.

If the determined quality or completeness of the portion of the set of images at least meets the threshold criteria (“YES”), processing may end or proceed to other functionality. In contrast, if the determined quality or completeness of the portion of the set of images does not at least meet the threshold criteria (“NO”), the server may determine (block1875) any additional or supplemental information that may be needed to at least meet the threshold criteria. For example, the server may determine that the “front right” perspective of the target vehicle is missing. For further example, the server may determine that the driver's side door (which may be a required component) is unable to be identified from the portion of the set of images.

At block1880, the server may generate a notification. In particular, the server may generate a notification that includes any information determined in block1875and/or other information. Generally, the notification may inform the individual associated with the electronic device that additional image data is needed to properly perform the image analysis functionalities. In some scenarios, such as if the server determines that the damage indicated by the telematics data is not depicted in the portion of the set of images (block1845ofFIG. 18A, to “B”), the server may generate a notification that instructs the individual to capture images of any or all of the damage that is indicated by the telematics data (or the damage indicated by the telematics data that is not depicted in the initial set of images). In this regard, the damage that may be depicted in any additionally-captured images may confirm (or not confirm) the damage indicated by the telematics data. Further, in this regard, any inaccurate or fraudulent damage information may be determined or identified.

After generating the notification, the server may transmit (block1885) the notification to the electronic device via the network connection. Accordingly, the individual associated with the electronic device may review the notification, ascertain any additional information or images that may be needed, and take necessary action to fulfill the needed additional information or images (e.g., capture additional images). Processing may then proceed to “C” as shown inFIG. 18A, at which the server may receive (block1825) any additional images from the electronic device. According to embodiments, the server may perform processing on the additional images as depicted inFIGS. 18Aand B, and as discussed herein, until the server identifies a portion of the received images that have qualities and characteristics that at least meet the threshold criteria of the base image model. At this point, the server may facilitate the additional image processing functionalities as discussed herein.

FIGS. 19A-19Dillustrate exemplary interfaces associated with capturing images of a target vehicle. One or more electronic devices (e.g., a mobile device, such as a smartphone) may be configured to display the interfaces and/or receive selections and inputs via the interfaces, where the electronic device(s) may be used by an individual associated with the target vehicle. For example, a dedicated application that is configured to operate on the electronic device may display the interfaces. The electronic device may communicate with a server via a network connection. It should be appreciated that the interfaces are merely exemplary and that alternative or additional content is envisioned.

FIG. 19Aillustrates an example interface1950depicting a target vehicle and associated image capturing functionalities. The interface1950may include a “live view” of image data that is captured, in real-time, by an image sensor of the associated electronic device. Further, the interface1950may include an instruction1951that instructs the individual to capture a front right view of the target vehicle, as well as a shutter selection1952that, upon selection, causes the electronic device to generate an image from the image data captured by the image sensor. In the example embodiment as illustrated inFIG. 19A, the “live view” may depict the front right view of the target vehicle and, in response to the user selecting the shutter selection1952, the interface1950may display a confirmation1953that the image was captured. The interface1950may further include a transmit selection1954that, upon selection, may cause the electronic device to transmit the captured image (as well as any previously-captured images) to the server via the network connection.

In response to the individual selecting the transmit selection1954, the electronic device may display an interface1955as depicted inFIG. 19B. The interface1955may include a status1956indicating that the electronic device is transmitting the captured image(s). After receiving the image(s) from the electronic device, the server may analyze the image(s) to identify the target vehicle depicted in the image(s) and determine whether the quality and/or characteristics of the image(s) at least meet the threshold criteria for the corresponding base image model, as discussed herein.

FIG. 19Cillustrates an example interface1960depicting a notification1961that the electronic device may receive from the server as a result of the image analysis performed by the server. In particular, the notification1961may instruct the individual to capture an image of the front left view of the target vehicle. In this scenario, the server may have determined that the front left view of the target vehicle either was missing from the initially-transmitted image(s) or that an image of the front left view of the target vehicle was of unacceptable quality (or was deficient in some other regard), and that a proper front left view is needed for proper image analysis according to the corresponding base image model.

The interface1960may include an “okay” selection1962that, upon selection, may cause the electronic device to display an interface1965as illustrated inFIG. 19D. Similar to the interface1950, the interface1965may include a “live view” of image data that is captured, in real-time, by an image sensor of the electronic device. Further, the interface1965may include an instruction1966that instructs the individual to capture a front left view of the target vehicle, as well as a shutter selection1967that, upon selection, causes the electronic device to generate an image from the image data captured by the image sensor. In the example embodiment as illustrated inFIG. 19D, the “live view” may depict the front left view of the target vehicle and, in response to the user selecting the shutter selection1967, the interface1965may display a confirmation1968that the image was captured. The interface1965may further include a transmit selection1969that, upon selection, may cause the electronic device to transmit the captured image to the server via the network connection. Accordingly, the server may analyze the captured image to determine whether the threshold criteria for the corresponding base image model is fulfilled.

Accordingly, the hardware terms used herein should be understood to encompass tangible entities, be that entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.

Hardware and software modules or routines can provide information to, and receive information from, other hardware and/or software modules and routines. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple of such hardware or software modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits, lines and buses) that connect the hardware or software modules. In embodiments in which multiple hardware modules or software are configured or instantiated at different times, communications between such hardware or software modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware or software modules have access. For example, one hardware or software module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware or software module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware and software modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

In addition, use of “a” or “an” is employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the description. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs may be used for implementing an image processing application and system for configuring and executing the change detection techniques disclosed herein. Thus, while particular embodiments and applications have been illustrated and described herein, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the methods and structure disclosed herein without departing from the spirit and scope defined in the claims.