Patent Description:
Imaging technologies including radiography, computed tomography, magnetic resonance imaging (MRI), positron emission tomography (PET), and many others have found widespread use in applications as diverse as medical imaging and cargo inspection. X-ray radiography is the principal method of Non-Intrusive Inspection (NII) of cargo today and is used in conjunction with human inspections.

<CIT> discloses systems and methods for identifying and reporting contents of a tanker, container or vehicle. Programmatic tools are provided to assist an operator in analyzing contents of a tanker, container or vehicle. Manifest data is automatically imported into the system for each shipment, thereby helping security personnel to quickly determine container contents. In case of a mismatch between container contents shown by manifest data and the contents as ascertained from the scanning system, the container or vehicle may be withheld for further inspection.

According to a first aspect of the present invention, there is provided a computing-device implemented method for validating a cargo manifest associated with a shipping container according to claim <NUM>. Optional features of the method are outlined in claims <NUM> to <NUM>.

According to a second aspect of the present invention, there is provided a system for validating a cargo manifest associated with a shipping container according to claim <NUM>. Optional features of the system are outlined in claims <NUM> to <NUM>.

According to a third aspect of the present invention, there is provided a non-transitory computer readable medium storing instructions executable by a processor for validating a cargo manifest associated with a shipping container according to claim <NUM>.

The skilled artisan will understand that the drawings are primarily for illustrative purposes and are not intended to limit the scope of the subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar or structurally similar elements).

The foregoing and other features and advantages provided by the present disclosure will be more fully understood from the following description of exemplary embodiments when read together with the accompanying drawings, in which:.

Today, images of cargo are evaluated during inspections of cargo by security personnel for threat detection, smuggling of illegal substances and evasion of customs duty. Scanners are used to assist in inspection of cargo but have traditionally been used to examine only a small percentage of cargo traffic, as scanner-rendered images conventionally still require significant human resources to properly interpret. In cases where cargo does get NII scrutiny, conventional image analysis relies on human perception, decision-making and judgment. Customs and Border Protection agency identified operator vigilance and variability in skills between operators as the main risks that can undermine the inspection process. Embodiments of the present invention help to address these risks by introducing machine-learning algorithm(s) for cargo inspection that can replace or aid operators in order to be able to achieve <NUM>% inspection of all cargo.

Described herein are image processing systems, methods, and computer readable media for processing images to verify container contents and identify anomalies in the images and manifests. Embodiments provide an image processing system that includes a machine-learning framework to automatically analyze the images and the manifests. In some embodiments, the images include radiographic images that may be obtained via a scanner or screening device at a security checkpoint. The shipping document, typically called the shipping manifest, is a mandatory document that accompanies a cargo container during the shipping and describes the container content and quantity, the origin, the destination, the shipper, shipping dates. The shipping document can also be the customs declaration or any similar document that can reveal the container content. As used herein, 'manifest' may denote other documents that describe cargo content in a shipping container in addition to, or in place of, a traditional shipping manifest.

One difficulty with conventional inspection approaches is the vigilance of human operators under heavy inspection load and variability in the decision between different operators. The task of a human operator is to understand the manifest text, determine the type of cargo that is being shipped, evaluate the radiographic image, and determine if the pattern in the image is consistent with the expected pattern. Inspection may be difficult as, for example, the shipping document (manifest) may contain trade names, foreign language words, and/or professional (chemistry) terms that may not be understandable to the operator. Therefore, the operator may not know the type of cargo in the container under inspection. Another challenge is to understand how the x-ray image pattern should look for a given cargo type. Containers are typically completely filled with cargo to optimize the shipping cost. Individual objects overlap with other objects in the radiographic transmission image, therefore, shapes of individual objects are often not recognizable and only the group pattern of the cargo is used to evaluate consistency. For example, an operator may recognize the shape of a single glass bottle in the radiographic image, but in a full container load of empty glass bottles individual bottle shapes are not recognizable, but the operator must rely on the group pattern.

Those skilled in art understand the difficulty of analyzing radiographic images compared to photographic images. Radiographic images are formed by creating x-rays with an x-ray source that is collimated to form a narrow fan beam, passing cargo through the beam, and detecting x-rays that pass through the cargo. Since the object image is created from x-rays that transmit through multiple cargo objects, the image contains overlapping radiographic profiles of all objects in the beam at a given time. Therefore, unlike a photographic image that shows only objects in the front, a radiographic images show multiple overlapping object that makes recognition of individual object more challenging.

In exemplary embodiments, the image processing system includes a computing device equipped with a processor in communication with a scanner, such as but not limited to an x-ray radiography scanner, configured to render radiographic images. As discussed further below, the computing device is configured to train a machine-learning algorithm(s) with a training data set that includes at least one radiographic image to generate at least one feature vector. The radiographic image includes at least one image of a test container storing objects. The test container may contain one or more objects or be empty. The objects may include one object or many objects of the same type. In exemplary embodiments, the machine-learning algorithm(s) is an autoencoder neural network (<NPL>). After the machine-learning algorithm(s) is trained as described below, the computing device receives at least one rendered radiographic image from the scanner. The rendered radiographic image depicts a container that contains objects of a certain type or may be empty. In one embodiment, the computing device verifies a shipping manifest document listing the object(s) in the container, verifies that the container is empty, or identifies an object or an anomaly using the at least one feature vector. The computing device generates an automated detection indication.

A machine-learning algorithm(s)(s) for image analysis of radiographic images of shipping containers is executed on a machine-learning framework or platform employing an autoencoder neural network. Some advantages of using autoencoder type neural network framework over other methods is that it requires a smaller number of images for training, can be trained on live data in real-time, can be used to label images, and can be used for self-learning. Accordingly, the machine-learning algorithm(s) is based on an autoencoder neural network and uses radiographic images of containers for the training data. The training minimizes the difference between the input image vector and the reconstructed image in each training sample, <MAT>, where A and B are the encoder and decoder parts of the network, respectively, x is the input image vector (see <FIG> discussed below), and Δ is Euclidean distance squared in the feature vector space that is used as the loss function in the training. Parameters of A and B are determined in the training using a standard backpropagation method. In the second part of the training, the feature vector of containers with same object type is averaged to obtain the calibration data. Object types that are averaged may include, for example, containers with beer cases, refrigerators, or motorcycles. However, the object type can be any object or objects transportable in a container. The calibration data contains the mean, 〈A(x)〉, and the standard deviation, 〈〈A<NUM>(x)〉 - 〈A(x)〉<NUM>〉, of the feature vectors for the sample, where the averaging is denoted with angle brackets. This procedure assumes that the feature parameters are described with the normal (Gaussian) distribution. If this is not the case the same-cargo sample is described with more than one mean and standard deviation such that the probability density function of the sample is approximated with a series of Gaussian approximations. In this way, even complex shapes can be described with a relatively small number of parameters.

The trained machine-learning algorithm(s) generates one or more feature vectors based on the training data set. Feature vectors are used to represent numeric or symbolic characteristics, called features, of an object in a mathematical way, and are used for pattern processing. The training data set is used to train the image processing system to identify patterns associated with the object type. The generated patterns are stored, and used to analyze feature vectors of one or more input images. The input images are images of objects obtained via a radiographic machine. Based on the feature vectors, in one embodiment, the image processing system is capable of identifying anomalies.

A second type of artificial neural network algorithm(s) is used to analyze the shipping documentation (manifest) using natural language processing algorithms. In exemplary embodiments, the algorithm is based on a word-to-vector type of neural network, also known as word embedding, (such as, but not limited to, word-to-vector types such as those described in <NPL>); see <FIG> discussed further below. The approach creates a dictionary of manifest words and associated labels, and finds correlation between the words and the labels. The words are transformed into a new feature vector space where correlated words have similar feature vectors (the hidden layer in <FIG>). The training procedure minimizes the loss function, for example, defined as: <MAT>, where x' is the sum of output word vectors from the manifest sample n, A and B are encoder and decoder parts of the network that convert the input word vector as x' = B · A · x, and y is the word vector for a label, s is the softmax function <MAT>. The training minimizes the loss function using standard backpropagation and gradient descent methods in order to determine parameters of the matrices A and B. Those skilled in art know that instead of using raw manifest words it is better to use combination of words or word segments. Common words that appear in all cargo categories ('the', 'origin', 'destination', 'weight'. ) have little discriminating power and can be ignored.

The image and/or manifest processing system described herein analyze radiographic images and/or manifests to verify contents of containers and identify anomalies in the images and/or manifests. Embodiments may perform manifest verification (MV) to verify that objects listed on the manifest correspond with objects in the containers, to perform empty container verification (ECV) to verify that containers are empty, to determine a cargo type or to perform object type detection, to search for specific objects and/or threats, and/or to perform anomaly detection to search for odd patterns. Each type of analysis is explained further below.

It should be appreciated that the radiographic image analysis of shipping containers described herein may be performed in isolation or in conjunction with cargo manifest analysis. In some embodiments, the techniques for cargo manifest analysis described herein, such as, but not limited to, those using word embedding, may also be performed in isolation (i.e. without performing radiographic image analysis) to determine the contents of shipping containers.

Manifest verification (MV) attempts to verify that what is in the shipping container is the same as what the manifest indicates is supposed to be in the shipping container. MV is performed by processing the shipping manifest text with the manifest processing system and processing the radiographic image with the image processing system. Both the image and manifest processing systems derive feature vectors that are similar for the same cargo category. For example, image feature vectors for cargo with empty bottles are similar, while different from other cargo due to specific pattern that empty bottles make in the radiographic image. Description of empty bottle cargo in the manifest is similar for empty bottle shipments so the manifest feature vector is specific to this cargo category. MV compares the correlation between image and manifest feature vectors with historic data.

In some embodiments, the manifest is analyzed first to determine the cargo category. The image processing system checks image's pattern(s) for consistency with historic images for this cargo category resulting in the chi-square value, as described above. The algorithm can choose to validate shipments with chi-square value less than a threshold. The threshold can be set such that the inspection meets required false alarm rate for given verification efficiency rate. For example, an implementation of the MV algorithm requires that at <NUM>% of manifest verification rate <NUM>% of falsely documented cargo should be interdicted. The Receiver-Operator-Characteristics curves for <NUM> cargo categories of a seaport (depicted in <FIG>) show the relationship between the efficiency and the interdiction rate.

In some embodiments, the image processing is done first in order to determine the cargo category, and the manifest processing derives a feature vector that is used to compare with the historic distribution.

In some embodiments, the cargo category is not assigned, but manifest and image feature vectors are combined into a large space. For example, image feature vector with <NUM>-dimensional space and the manifest feature vector in <NUM>-dimensional space are combined in a <NUM>-dimensional space. Instead of looking for clusters separately in manifest and image spaces, a check is performed that the data is consistent with historic clean scans in <NUM>-dimensional space. Not having to place the cargo into a category allows semi-supervised and continuous learning.

In some embodiments, the image processing system uses the feature vector to rank radiographic images for operator inspection.

The image processing system determines a statistical probability that the shipping container content matches the manifest by comparing the feature vector extracted from the radiographic image of the shipping container against a historic distribution of prior feature vectors of radiographic images of prior shipping containers associated with cargo manifests listing a like object to those supposedly in the shipping container under examination. As a non-limiting example, the system <NUM> may generate a statistical probability of the shipping container content matching the listed objects in the associated cargo manifest based on the degree of matching between the extracted feature vector and the historic distribution of prior feature vectors using pre-determined criteria. For example an <NUM>% match between the extracted feature vector and the historic distribution of feature vectors may directly equate to the system <NUM> assigning an <NUM>% probability of the contents of the shipping container matching the associated cargo manifest or may equate to a lesser or greater probability being assigned based on previous historical data acquired by the system.

In some embodiments, the image processing system determines that a parameter quantifying the container content (e.g., objects of an object type) matches the manifest. The parameter is displayed graphically against a distribution of the historic parameters for the object type and presented to the operator.

In some embodiments, the manifest processing system helps an operator to understand the meaning of the manifest text. As mentioned before, the manifest may contain trade or professional terms that are not known to the operator. Manifest processing determines the cargo category based on the proximity of the feature vector with historic distribution of labeled cargo manifests. Cargo category determination is a part of Operator Assistance.

In some embodiments, the image processing system helps operators to understand the expected pattern in the radiographic image. As described above, containers are typically full of cargo and individual object shapes may not be recognizable, but instead operators rely on the group pattern. Image processing finds the expected image pattern for the cargo category based on historic data. Display of similar historic images is a part of an operator assistance function that may display randomly chosen historic images for this cargo category or images that are closest to the image under inspection (see <FIG> for example).

In some embodiments, the image processing system updates the historic distribution of feature vectors in real time with every new image.

In some embodiments, the historic distribution of feature vectors is based on less than <NUM> radiographic scans and corresponding manifests.

The artificial neural network is an autoencoder neural network.

In some embodiments, the artificial neural network is an unsupervised neural network.

Empty Container Verification is performed by the image processing system described herein by inspecting radiographic images of an empty container. The image processing system receives a rendered radiographic image that includes a container that is declared empty. The image processing system divides the image into one or more segments that are analyzed using an autoencoder type neural network to extract a feature vector. The image processing system compares the feature vector against historic distribution of feature vectors for the corresponding container segment. Empty Container Verification searches for objects inside an otherwise empty container. The image processing system uses the feature vector to confirm that the declared-empty container is indeed empty.

In some embodiments, the image processing system uses the feature vector to rank radiographic images of declared-empty containers for operator inspection.

In some embodiments, the image processing system assigns a statistical probability that the container is empty.

In some embodiments, the image processing system determines a parameter quantifying that the container is empty, which is displayed graphically against a distribution of historic parameters for empty containers and presented to the operator.

In some embodiments, the image processing system updates the historic distribution of feature vectors for each segment in real time with every empty container radiographic image.

In some embodiments, the historic distribution of feature vectors is based on less than <NUM> radiographic scans.

In some embodiments, the objects found in declared-empty containers are marked for operator verification (see for example, <FIG>).

Anomaly detection is performed by the image processing system described herein and by inspecting radiographic images of a container to detect an anomaly. The image processing system divides the image into one or more segments (for example, as shown in <FIG>) that are analyzed using an autoencoder type neural network to extract a feature vectors for each segment. The image processing system compares each feature vector against the distribution of other feature vectors from the container. The image processing system uses the feature vector to identify segments that are significantly different from the rest of the image. For example, <FIG>, described further below, shows chi-square difference of the feature vector from the average of other segments.

In some embodiments, the image processing system assigns a statistical probability that the container segment has an anomaly.

In some embodiments, the image processing system determines a parameter quantifying a probability that the container segment has an anomaly, which is displayed graphically against the distribution of parameters for the container segments and presented to the operator.

Data quality monitoring is performed by the image processing system described herein by analyzing radiographic images for data quality monitoring. The image processing system divides an image into one or more segments that are analyzed using an autoencoder type neural network to extract one or more feature vectors for each segment. The image processing system compares each feature vector against historic distribution feature vectors for the segment. The image processing system uses the feature vectors to evaluate data quality of the radiographic image such as bad detector channels, cut off images, and images with incorrect radiation dose.

Cargo type detection is performed by the image processing system described herein by analyzing radiographic images of one or more objects under inspection. The image processing system analyzes the images using an artificial type neural network to extract a feature vector. Each feature vector is compared against historic distribution feature vectors for the segment and the feature vectors are used to classify each of the one or more objects being scanned.

In each of the actions described above, a label may be included in an output image generated by the image processing system identifying an anomaly or object in the analyzed image. In some embodiments, the location of the anomaly or object is indicated or highlighted in an output image. In another embodiment, the image processing system may be used to visually or audibly indicate to security personnel that the container includes an anomaly and/or does not match the manifest.

Embodiments may be used in various security applications, such as but not limited to, applications where security personnel are evaluating images of containers for threat. For example, the image processing system described herein can be used by Customs and Border Protection at ports for checking containers at a security checkpoint. The image processing system may also be used by Customs and Border Protection to scan containers. The image processing system may also be used to check cargo on transport or delivery vehicles.

Having described an overview and benefits of embodiments above, more specific details of the embodiments are now provided with respect to the figures below.

<FIG> is a block diagram showing an image processing system <NUM> in terms of modules according to an example embodiment. One or more of the modules may be implemented using device <NUM>, scanning device <NUM>, server <NUM>, and/or database <NUM>, as shown in <FIG>. The modules include a machine-learning algorithm module <NUM>, a training data set module <NUM>, an input data module <NUM>, and an output data module <NUM>. The modules may include various circuits, circuitry and one or more software components, programs, applications, or other units of code base or instructions configured to be executed by one or more processors. In some embodiments, one or more of modules <NUM>, <NUM>, <NUM>, <NUM> may be included in server <NUM>, while other of the modules <NUM>, <NUM>, <NUM>, <NUM> are provided in device <NUM> or scanning device <NUM>. Although modules <NUM>, <NUM>, <NUM>, and <NUM> are shown as distinct modules in <FIG>, it should be understood that modules <NUM>, <NUM>, <NUM>, and <NUM> may be implemented as fewer or more modules than illustrated. It should be understood that any of modules <NUM>, <NUM>, <NUM>, and <NUM> may communicate with one or more components included in system <NUM> (<FIG>), such as device <NUM>, scanning device <NUM>, server <NUM>, or database(s) <NUM>.

The machine-learning algorithm module <NUM> may be a software and/or hardware-implemented module configured to execute or run a machine-learning algorithm(s), and may store and manage parameters, variables, data, and other components needed to execute the machine-learning algorithm(s). The machine-learning algorithm(s) is an autoencoder type neural network 110A used to perform image analysis of radiographic images of a shipping container or a recurrent neural network 110B used to perform analysis of cargo manifests associated with a shipping container.

The training data set module <NUM> may be a software and/or hardware-implemented module configured to manage and store a training data set for the machine-learning algorithm(s) employed by the image processing system <NUM>. In an example embodiment, the training data set includes images of containers obtained using a radiographic screening machine or device. The images may depict containers may storing one or more objects.

The input data module <NUM> may be a software and/or hardware-implemented module configured to manage and store input images that are analyzed by the image processing system <NUM>.

The output data module <NUM> may be a software and/or hardware-implemented module configured to manage and store the output of the machine-learning algorithm(s). In some embodiments, the output is an indication whether the input image includes an anomaly. In an example embodiment, the output is an output image indicating the presence of the anomaly by visually highlighting or emphasizing the anomaly in the output image. In one embodiment, the output image may include a label identifying the anomaly. In another example embodiment, the output is an alarm or alert generated at a security screening machine or device to indicate to an operator or security personnel that the container includes an anomaly and/or does not match the manifest based on the analysis of the images.

It should be appreciated that the modules of <FIG> may provide some or all of the functionality discussed above with respect to radiographic image and manifest analysis. As discussed above, the machine-learning algorithm(s) is based on an autoencoder neural network and uses radiographic images of containers for the training data. The training minimizes the difference between the input image vector and the reconstructed image in each training sample, <MAT>, where A and B are the encoder and decoder parts of the network, respectively, x is the input image vector (see <FIG> discussed below), and Δ is Euclidean distance squared in the feature vector space that is used as the loss function in the training. Parameters of A and B are determined in the training using a standard back propagation method. In the second part of the training, the feature vector of containers with same object type is averaged to obtain the calibration data. Object types that are averaged may include, for example, containers with beer cases, refrigerators, or motorcycles. However, the object type can be any object or objects transportable in a container. The calibration data contains the mean, 〈A(x)〉, and the standard deviation, 〈〈A<NUM>(x)〉 - (A(x)〉<NUM>〉, of the feature vectors for the sample, where the averaging is denoted with angle brackets. This procedure assumes that the feature parameters are described with the normal (Gaussian) distribution. If this is not the case the same-cargo sample is described with more than one mean and standard deviation such that the probability density function of the sample is approximated with a series of Gaussian approximations. In this way, even complex shapes can be described with a relatively small number of parameters.

A second type of artificial neural network algorithm(s) is used to analyze the shipping documentation (manifest) using natural language processing algorithms. In exemplary embodiments, the algorithm is based on a word-to-vector type of neural network, also known as word embedding, (such as, but not limited to, word-to-vector types such as those described in <NPL>); see <FIG> discussed further below. The approach creates a dictionary of manifest words and associated labels, and finds correlation between the words and the labels. The words are transformed into a new feature vector space where correlated words have similar feature vectors (the hidden layer in <FIG>). The training procedure minimizes the loss function, for example, defined as: <MAT>, where x' is the sum of output word vectors from the manifest sample n, A and B are encoder and decoder parts of the network that convert the input word vector as x' = B ·A ·x, and y is the word vector for a label, s is the softmax function <MAT>. The training minimizes the loss function using standard back propagation and gradient descent methods in order to determine parameters of the matrices A and B. Those skilled in art know that instead of using raw manifest words it is better to use combination of words or word segments. Common words that appear in all cargo categories ('the', 'origin', 'destination', 'weight'. ) have little discriminating power and can be ignored.

The image and/or manifest processing system described herein analyze radiographic images and/or manifests to verify contents of containers and identify anomalies in the images and/or manifests. Embodiments may perform manifest verification (MV) to verify that objects listed on the manifest correspond with objects in the containers, to perform empty container verification (ECV) to verify that containers are empty, to determine a cargo type or to perform object type detection, to search for specific objects and/or threats, and/or to perform anomaly detection to search for odd patterns.

<FIG> is a flowchart illustrating an exemplary method <NUM> employed by the image processing system, according to an example embodiment. The method <NUM> may be performed using or one or more modules of system <NUM> described above.

At block <NUM>, the machine-learning algorithm(s) employed in the image processing system <NUM> is trained to analyze a specific object type within a container using a training data set. The machine-learning algorithm module <NUM> may be configured to execute the machine-learning algorithm(s). The training data module <NUM> may be configured to provide the training data set to the machine-learning algorithm module <NUM> for training.

In an example embodiment, the training data set includes at least one radiographic image of a container containing a specified object type. For example, the training data set may include a radiographic image of a container containing refrigerators. The training data set can be used to train the machine-learning algorithm(s) to identify an anomaly in a radiographic image. For example, the training data set can be used to train the machine-learning algorithm(s) to determine whether a radiographic image of a container containing refrigerators contains an anomaly.

In a non-limiting example, the images for the training data set are obtained from a scanning device installed at a security checkpoint in a facility, such as a port or border control, a highway way station, and the like.

In an example embodiment, the training data set may include 2D or 3D images. In another example, the training data set may include multiple 2D images representing different layers of an object. The 2D images may be cross-sectional images or slices of an object. In yet another example, the training data set may include multiple images each representing a different perspective of an object.

At block <NUM>, the input data module <NUM> receives at least one rendered radiographic image that includes a container.

In one embodiment, the image processing system <NUM> analyzes the input image and identifies a presence of an anomaly by dividing the input image into one or more segments that are analyzed using an autoencoder type neural network to extract feature vectors for each segment. Each feature vector is compared against the distribution of other feature vectors from this container. The feature vector is used to identify segments that are significantly different from the rest of the image.

At block <NUM>, the machine-learning algorithm module <NUM> is configured to analyze the input image received at block <NUM>, and automatically determine whether the input image is consistent with a manifest document listing the object type in the container, whether the input image is consistent with an empty container, whether the input image includes an object, or whether the input image includes an anomaly. For example, the image processing system <NUM> may determine that the input image includes an anomaly within a refrigerator.

At block <NUM>, the output data module <NUM> generates an automated detection indication based on the analysis of step <NUM>. For example, the automated detection indication may be associated with an anomaly identified in the input image. In an example embodiment, the automated detection indication is a text label or other graphical indication that identifies the anomaly in the input image. The automated detection indication may be displayed on or included in an output image provided at a display of a user device. The output image may be the input image (e.g., a rendered radiographic image) with an indication of the presence of the anomaly. As described herein, the user device where the output or the automated detection indication is provided to the user may include a display device communicatively coupled to security scanning device, a computing device, or a server. In another embodiment, the automated detection indication may be a visual indication indicating the location of the anomaly in the input image. Such visual indication may be a colored box shape enclosing the anomaly. In another embodiment, the automated detection indication may be a graphical indication and/or an audible indication alerting a user that an anomaly is detected in the input image. In another embodiment, the automated detection indication may indicate whether the input image is consistent with the manifest document or whether the input image is consistent with an empty container.

The automated detection indication may be transmitted to a user device or a computing device coupled to a security scanning device in a facility. In some embodiments, the security scanning device may be an x-ray screening machine at a port of entry or other secure facility. The automated detection indication may be generated and transmitted in real-time or near real-time with respect to when the input image is received at the image processing system <NUM>.

In some embodiments, the image processing system <NUM> determines that the identified anomaly at block <NUM> is a common false-alarm anomaly for an object type. In an example embodiment, a list of common false-alarm anomalies is stored in a database. In an example embodiment, anomalies that may be identified by the image processing system but are common false-alarms anomalies may be aggregated to generate a list of common false-alarm anomalies to enable a particular machine-learning algorithm(s) to account for the false alarm anomalies.

To implement the image processing system described herein, an exemplary embodiment employs a computing device running LINUX, one or more graphics processing cards or units (GPUs), one or more images for training, and an autoencoder framework.

<FIG> illustrates an exemplary autoencoder neural network topology employed by the image processing system, according to an example embodiment. An autoencoder is a neural network that has three layer stages: an input layer stage <NUM>, a hidden (encoding) layer stage <NUM>, and a decoding layer stage <NUM>. The network is trained to reconstruct its inputs, which forces the hidden layer stage to try to learn good representations of the inputs. Images are reduced from the original size to a <NUM>-dimensional feature vector through a series of hidden layers. The network is trained by expanding the feature vector to the original size and applying a loss function to restore the original image.

The mapping learned by the encoder part of the autoencoder neural network is useful for extracting features from data. Each neuron in the encoder has a set of weights associated with it, which will be tuned to respond to a particular visual feature. The features learned by the autoencoder neural network represent patterns from the digital images. The training data set is used to train the image processing system to identify patterns associated with the object type. The generated patterns are stored, and used to analyze feature vectors of one or more input images, as described above.

<FIG> shows an output image provided by the image processing system, in accordance with an exemplary embodiment. <FIG> illustrates radiographic images <NUM> of containers containing common object types, including refrigerators, motorcycles, beer, tobacco, and polyethylene terephthalate (PET), and an empty container.

In order to quantify dissimilarity between object types, the system performs an image analysis using an autoencoder type neural network with topology as shown in <FIG>. The object type within the container under inspection is compared with contours of historic distribution of a same object type, which are then plotted as shown in <FIG>.

<FIG> shows feature vectors from around <NUM> image segments of four object types (tobacco, beer, refrigerators, and motorcycles) plotted on a graph, as well as image segments for an empty container and PET container. Clustering of points from the same object type 430A, 430B, 430C, 430D, 430E and 430F indicates that the object type can be distinguished based on the image features.

Container content verification can be achieved by comparing manifest data and radiographic images. In some embodiments, the system parses the manifest and automatically verifies that the image pattern matches objects listed in the manifest. In other embodiments, operators at security checkpoints may receive radiographic images of containers and/or a manifest with content information for each container. In such an embodiment, the operator verifies that the image pattern matches objects listed in the manifest.

<FIG> illustrates a network diagram depicting a system <NUM> for implementing the image processing system, according to an example embodiment. The system <NUM> can include a network <NUM>, multiple devices, for example, device <NUM>, scanning device <NUM>, server <NUM>, and a database(s) <NUM>. Each of the devices <NUM>, <NUM>, server <NUM>, and database(s) <NUM> is in communication with the network <NUM>.

In an example embodiment, one or more portions of network <NUM> may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, a wireless network, a Wi-Fi network, a WiMax network, any other type of network, or a combination of two or more such networks.

The device <NUM> may include, but are not limited to, work stations, computers, general purpose computers, Internet appliances, hand-held devices, wireless devices, portable devices, wearable computers, cellular or mobile phones, portable digital assistants (PDAs), smart phones, tablets, ultrabooks, netbooks, laptops, desktops, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, mini-computers, and the like. Device <NUM> may connect to network <NUM> via a wired or wireless connection.

The scanning device <NUM> includes a radiographic machine or system to scan containers, and may produce rendered radiographic images of scanned containers on a display device coupled to the scanning device <NUM>. In an example embodiment, the scanning device <NUM> is a security screening system at a checkpoint. The scanning device <NUM> may be used to scan shipping containers for security purposes. In an example embodiment, the images produced by the scanning device <NUM> are used as the dataset to train the machine-learning algorithm(s) as described herein. The scanning device <NUM> produces the input image for the image processing system <NUM>. After the input image is processed, the output image may be displayed at the display device coupled to the scanning device <NUM>, where the output image may include a label for an anomaly and an indication of the location of the anomaly within the image.

As an add-on module, the image processing system described herein can be loaded as a software module onto an existing security checkpoint device or existing server networked to one or more devices. The image processing system generates an alarm and alerts the operator of the security checkpoint device to detected anomalies, in addition to any other detection technology that may be installed on those devices or servers.

One schooled in the art can appreciate that the add-on capability may be added to a variety of radiographic imaging devices. In this way, the alerts generated by the image processing system add-on described herein may be displayed on the screen along with alarms generated by other detection software.

In an example embodiment, the devices <NUM>, <NUM> may perform one or more of the functionalities of the image processing system <NUM> described herein. The device <NUM>, <NUM> can include one or more components of computing device <NUM> of <FIG>. The device <NUM>, <NUM> may be used to train the machine-learning algorithm(s), and then use the trained algorithm on input images to identify presence of an anomaly. For example, device <NUM>, <NUM> may store and execute first neural network <NUM> to perform radiographic image analysis and second neural network <NUM> to perform cargo manifest analysis for a shipping container.

In an example embodiment, the image processing system <NUM> may be included on the server <NUM>, and the server <NUM> performs one or more of the functionalities of the image processing system <NUM> described herein. In some embodiments, the devices <NUM>, <NUM> may perform some of the functionalities, and the server <NUM> performs the other functionalities described herein.

Each of the database(s) <NUM> and server <NUM> is connected to the network <NUM> via a wired or wireless connection. Server <NUM> includes one or more computers or processors configured to communicate with devices <NUM>, <NUM> via network <NUM>. The server <NUM> can include one or more components of device <NUM> of <FIG>. Server <NUM> hosts one or more applications or websites, including the image processing system described herein, accessed by devices <NUM>, <NUM> and/or facilitates access to the content of database(s) <NUM>. Database(s) <NUM> include one or more storage devices for storing data and/or instructions (or code) for use by server <NUM>, and/or devices <NUM>, <NUM>. Database(s) <NUM> and server <NUM> may be located at one or more geographically distributed locations from each other or from devices <NUM>, <NUM>. Alternatively, database(s) <NUM> may be included within server <NUM>.

The system <NUM> includes a first neural network <NUM> and a second neural network <NUM>. The first neural network <NUM> is used to extract feature vectors from radiographic images, as described herein. The second neural network <NUM> is used for cargo manifest text analysis, as described herein. In some embodiments, the device <NUM> and/or the server <NUM> utilizes the first neural network <NUM> and the second neural network <NUM> for processing images and cargo manifests to verify container contents and/or identify anomalies in the images and cargo manifests, as described herein.

<FIG> is a block diagram of an exemplary computing device <NUM> that can be used to perform one or more steps of the methods provided by exemplary embodiments. For example, computing device <NUM> may be, but is not limited to device <NUM>, <NUM> and server <NUM> as described in <FIG>. The computing device <NUM> includes one or more non-transitory computer-readable media for storing one or more computer-executable instructions or software for implementing exemplary embodiments. The non-transitory computer-readable media can include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more USB flash drives), and the like. For example, memory <NUM> included in the computing device <NUM> can store computer-readable and computer-executable instructions or software for implementing exemplary embodiments. The computing device <NUM> also includes processor <NUM> and associated core <NUM>, and optionally, one or more additional processor(s) <NUM>' and associated core(s) <NUM>' (for example, in the case of computer systems having multiple processors/cores), for executing computer-readable and computer-executable instructions or software stored in the memory <NUM> and other programs for controlling system hardware. Processor <NUM> and processor(s) <NUM>' can each be a single core processor or multiple core (<NUM> and <NUM>') processor. The computing device <NUM> also includes a graphics processing unit (GPU) <NUM>. In some embodiments, the computing device <NUM> includes multiple GPUs.

Virtualization can be employed in the computing device <NUM> so that infrastructure and resources in the computing device can be shared dynamically. A virtual machine <NUM> can be provided to handle a process running on multiple processors so that the process appears to be using only one computing resource rather than multiple computing resources. Multiple virtual machines can also be used with one processor.

Memory <NUM> can include a computer system memory or random access memory, such as DRAM, SRAM, EDO RAM, and the like. Memory <NUM> can include other types of memory as well, or combinations thereof. A user can interact with the computing device <NUM> through a visual display device <NUM>, such as a touch screen display or computer monitor, which can display one or more user interfaces <NUM>. The visual display device <NUM> can also display other aspects, elements and/or information or data associated with exemplary embodiments. The computing device <NUM> can include other I/O devices for receiving input from a user, for example, a keyboard or any suitable multi-point touch interface <NUM>, a pointing device <NUM> (e.g., a pen, stylus, mouse, or trackpad). The keyboard <NUM> and the pointing device <NUM> can be coupled to the visual display device <NUM>. The computing device <NUM> can include other suitable conventional I/O peripherals.

The computing device <NUM> can also include one or more storage devices <NUM>, such as a hard-drive, CD-ROM, or other computer readable media, for storing data and computer-readable instructions and/or software, such as one or more modules of the system <NUM> shown in <FIG> that implements exemplary embodiments of the notification system as described herein, or portions thereof, which can be executed to generate user interface <NUM> on display <NUM>. Exemplary storage device <NUM> can also store one or more databases for storing any suitable information required to implement exemplary embodiments. The databases can be updated by a user or automatically at any suitable time to add, delete, or update one or more objects in the databases. Exemplary storage device <NUM> can store some or all of image processing system <NUM> including training data 100A, first neural network 100B, second neural network 100C and/or algorithms 110D (such as but not limited to a manifest verification algorithm, an empty container verification algorithm, a cargo type detection algorithm, an anomaly detection algorithm and/or a data quality monitoring algorithm).

The computing device <NUM> can include a network interface <NUM> configured to interface via one or more network devices <NUM> with one or more networks, for example, Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, <NUM>, T1, T3, 56kb, X. <NUM>), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections, controller area network (CAN), or some combination of any or all of the above. The network interface <NUM> can include a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device <NUM> to any type of network capable of communication and performing the operations described herein. Moreover, the computing device <NUM> can be any computer system, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer (e.g., the iPad® tablet computer), mobile computing or communication device (e.g., the iPhone® communication device), or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.

The computing device <NUM> can run any operating system <NUM>, such as any of the versions of the Microsoft® Windows® operating systems, mobile OS, Android and iOS, the different releases of the Unix and Linux operating systems, any version of the MacOS® for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating systems for mobile computing devices, or any other operating system capable of running on the computing device and performing the operations described herein. In exemplary embodiments, the operating system <NUM> can be run in native mode or emulated mode. In an exemplary embodiment, the operating system <NUM> can be run on one or more cloud machine instances.

<FIG> illustrates an exemplary neural network architecture for manifest text analysis, in accordance with an exemplary embodiment. Input neurons are word segments (n-grams) found in manifest data including cargo type labels. The purpose of the neural net training is to find correlation between word segments that occur in the same context such that related words have similar feature vectors. The training uses both encoder and decoder part of the network, while the manifest evaluation uses only the encoder part that calculates the feature vector. The cargo is classified by selecting a cargo label that has closes feature vector with the average of manifest text.

<FIG> illustrates an exemplary implementation of a manifest verification (MV) algorithm to assist an operator in an inspection decision, in accordance with an exemplary embodiment. The operator views an image under inspection <NUM> and/or raw manifest text. The MV algorithm results <NUM> are displayed to the operator. As shown at results <NUM>, the MV interprets the manifest and assigns a label <NUM> that explains the cargo content to the operator. The MV also displays images <NUM> of previous scans with the same cargo content. The MV displays a confidence level <NUM> that the image matches the manifest. The MV algorithm results <NUM> are typically displayed via a user interface.

<FIG> illustrates an exemplary Receiver-Operator-Characteristics (ROC) curve <NUM> that relates the efficiency of manifest verification to interdiction of incorrectly documented cargo in accordance with an exemplary embodiment. The efficiency of manifest verification is defined as the fraction of correctly documented cargo that is cleared by the algorithm. The interdiction rate is the fraction of incorrectly documented cargo that is flagged by the algorithm. Each ROC curve belongs to a cargo category and the plot shows approximately <NUM> categories. In the exemplary implementation, all cargo categories have at least <NUM>% interdiction rate at <NUM>% of manifest clearing rate.

<FIG> illustrates an implementation of an empty container validation (ECV) algorithm that flagged a duffel bag filled with currency in a declared-empty container, in accordance with an exemplary embodiment. A suspect region <NUM> is marked in the image for operator attention. A photograph <NUM> is added to the figure to show the test target.

<FIG> illustrates container segments <NUM> that are used for anomaly detection, in accordance with an exemplary embodiment. Each segment <NUM> (as illustrated through the use of a grid overlay to explain segmentation) is evaluated with an autoencoder neural network that provides a feature vector, in accordance with an exemplary embodiment. The result is to identify segments <NUM> that are significantly different from the rest of segments. As depicted in <FIG> some segments <NUM> have been identified that indicate an anomaly.

<FIG> illustrates results of an exemplary implementation of an anomaly detection algorithm, in accordance with an exemplary embodiment. The neural network generates a feature vector for each segment, and the output module makes the chi-square heat map (by comparing feature vectors). In some embodiments, colors denote chi-square values of the segment under inspection from the other segments. The chi-square is calculated as the distance-squared between the segment feature vector and the average of feature vectors of other segments, divided by the standard deviation squared of feature vectors of other segments. This chi-square is normalized to the degree of freedom i.e. the number of features minus one. The results indicate an anomaly <NUM> detected in certain segments of the shipping container (as shown in <FIG>). For instance, the anomaly may indicate an object in a container that is supposed to be empty, or a different type of object than that listed on the cargo manifest.

<FIG> is a flowchart illustrating an exemplary method to validate cargo manifests employed by the image processing system, according to an example embodiment. At step <NUM>, a computing device receives a rendered radiographic image from a scanner that includes a shipping container in the image. At step <NUM>, a feature vector is extracted from the radiographic image of the scanned shipping container using an autoencoder neural network. At step <NUM>, the image processing system assigns a statistical probability that the container matches a listed object on an associated cargo manifest for the scanned shipping container. As discussed above, in step <NUM>, the statistical probability is further determined by performing an additional analysis on words in cargo manifests using a second artificial neural network. In step <NUM>, an automated detection indication related to the scanned shipping container is then generated by the image processing system based on the assigned statistical probability. The detection indication may take a number of forms such as, but not limited to, generating warning displays or audible alarms.

<FIG> is a flowchart illustrating an exemplary method employed by the image processing system to confirm that designated empty shipping containers are in fact empty, according to an example embodiment. At step <NUM>, a computing device receives a rendered radiographic image of a shipping container. At step <NUM>, a feature vector is extracted from the image using an autoencoder neural network. At step <NUM>, the image processing system assigns a statistical probability of the emptiness of the scanned shipping container based on comparison of the extracted feature vector against a historic distribution of feature vectors extracted from radiographic images associated with prior empty shipping containers.

Claim 1:
A computing-device implemented method for validating a cargo manifest associated with a shipping container, the method comprising:
receiving, at a computing device (<NUM>), a rendered radiographic image of a scanned shipping container from a scanner (<NUM>) configured to scan the shipping container and render the radiographic image;
extracting, with an autoencoder neural network (<NUM>), a feature vector from the radiographic image;
determining a statistical probability that an object in the scanned shipping container matches a listed object on a cargo manifest associated with the scanned shipping container based at least in part on:
a comparison of the extracted feature vector against a historic distribution of prior feature vectors of radiographic images of prior shipping containers associated with cargo manifests containing a like object; and
generating an automated detection indication related to the scanned shipping container based on the determined statistical probability, the automated detection indication:
providing a graphical indicator of a degree of consistency of the content of the scanned container with the associated cargo manifest, and/or
providing a display of a ranking of radiographic images, and/or
being a graphical indication or an audible indication alerting a user that an anomaly is detected in the radiographic image, and/or
being a text label identifying the anomaly, and/or
being a colored box shape enclosing the anomaly, and/or
indicating whether the radiographic image is consistent with an empty container;
characterized in that the statistical probability is further determined based at least in part on natural language processing of the cargo manifest by a second artificial neural network.