Patent Description:
Known methods and systems of inspection use transmission of radiation through a container. It is sometimes difficult to detect threats (such as weapons or explosives) or contraband (such as cigarettes or drugs) when the threats or the contraband are placed in a container which has a complex pattern when inspected by transmission, such as a refrigeration unit or an engine of a vehicle.

<CIT> discloses a vehicle inspection technique.

<CIT> discloses a vehicle identification technique.

Aspects of the invention are recited in the appended claims.

In the Figures like reference numerals are used to indicate like elements.

Embodiments of the disclosure relate to a method for inspecting a container. An inspection image of the container is classified in a class among predetermined classes of containers of interest. A model of the inspection image is compared to models of reference images contained within the class in which the inspection image has been classified. The closest reference image among the reference images is associated with the inspection image and registered with respect to the inspection image, to map differences between the inspection image and the closest reference image. In some examples, a classifier is applied to the mapped differences and a probability that the differences correspond to objects of interest is determined.

<FIG> illustrates an analyser <NUM> configured to inspect an inspection image <NUM> of a container <NUM>.

Each of the images <NUM> may be generated by an inspection system <NUM> described in greater detail below.

As will be apparent in more detail below, the analyser <NUM> may be configured to receive the one or more images <NUM> from the system <NUM>, for example over a communication network <NUM> which may be wired and/or may be wireless. The analyser <NUM> conventionally comprises at least a processor and a memory in order to carry out an example method according to the disclosure.

As explained in further detail below in relation to <FIG> and <FIG>, the inspection system <NUM> is configured to inspect the container <NUM> by transmission of inspection radiation <NUM> from an inspection radiation source <NUM> to an inspection radiation detector <NUM> through the container <NUM>.

<FIG> and <FIG> illustrate that the container <NUM> may comprise a shipping container and/or a vehicle (such as a car and/or a van and/or a bus and/or a train and/or a truck) and/or a part of a shipping container and/or a vehicle (such as an engine and/or a boot lid and/or a hood and/or a roof and/or a floor and/or a wheel and/or a trailer and/or a refrigeration unit). It is appreciated that the container <NUM> may be any type of container, and thus may be a suitcase in some examples.

The radiation source <NUM> is configured to cause inspection of the container <NUM> through the material (usually steel) of walls of the container <NUM>, e.g. for detection and/or identification of a load of the container and/or an object of interest. The object of interest may comprise threats (such as weapons or explosives) and/or contraband (such as cigarettes or drugs, etc.), as non-limiting examples.

The system <NUM> is configured to, in the inspection mode, cause inspection of the container <NUM>, in totality (i.e. the whole container <NUM> is inspected) or partially (i.e. only a chosen part of the container is inspected). Partial inspection may be advantageous e.g., typically, when inspecting a vehicle, as a cabin of the vehicle may not be inspected to protect a driver of the vehicle from radiation, whereas a rear part of the vehicle is inspected.

In the example illustrated by <FIG>, the inspection system <NUM> may be mobile and may be transported from a location to another location (the system <NUM> may comprise an automotive vehicle), and in the example illustrated by <FIG>, the inspection system <NUM> may be static with respect to the ground and cannot be displaced.

A type of the inspection system <NUM> may be characterized by an energy and/or a dose of the inspection radiation <NUM>.

In the examples illustrated by the Figures, the inspection radiation source <NUM> comprises an X-ray generator. The energy of the X-rays may be comprised between 300keV and 15MeV, and the dose may be comprised between 2mGy and 20Gy (Gray).

In the example illustrated by <FIG>, the power of the X-ray source <NUM> may be e.g., between 500keV and <NUM>. 0MeV, typically e.g., 2MeV, <NUM>. 5MeV, 4MeV, or 6MeV, for a steel penetration capacity e.g., between <NUM> to <NUM>, typically e.g., <NUM> (<NUM>. In the example illustrated by <FIG>, the dose may be e.g., between 20mGy and 50mGy.

In the example illustrated by <FIG>, the power of the X-ray source <NUM> may be e.g., between 4MeV and 10MeV, typically e.g., 9MeV, for a steel penetration capacity e.g., between <NUM> to <NUM>, typically e.g., <NUM> (<NUM>. In the example illustrated by <FIG>, the dose may be 17Gy.

In the examples illustrated by the Figures, the inspection radiation detector <NUM> comprises, amongst other conventional electrical elements, radiation detection lines <NUM>, such as X-ray detection lines. The inspection radiation detector <NUM> may further comprise other types of detectors, such as optional gamma and/or neutrons detectors, e.g., adapted to detect the presence of radioactive gamma and/or neutrons emitting materials within the container <NUM>, e.g., simultaneously to the X-ray inspection.

In the example illustrated in <FIG>, the inspection radiation detector <NUM> may also comprise an electro-hydraulic boom <NUM> which can operate in a retracted position in a transport mode (not illustrated in the Figures) and in an inspection position (<FIG>). The boom <NUM> may be operated by hydraulic activators (such as hydraulic cylinders).

In the example illustrated in <FIG>, the inspection radiation detector <NUM> may also comprise a structure and/or gantry <NUM>.

The detection lines <NUM> may be mounted on the boom <NUM> (<FIG>) or structure and/or gantry <NUM> (<FIG>), facing the source <NUM> on the other side of the container <NUM>.

In order to inspect the container <NUM>, in the example illustrated by <FIG>, the system <NUM> may comprise a motion generation device so that the system <NUM> may be displaced, the container <NUM> being static (this mode is sometimes referred to as a 'scanning' mode). Alternatively or additionally, the motion generation device may cause the container <NUM> to be displaced, the system <NUM> being static with respect to the ground (<FIG>). Alternatively or additionally, in a 'pass-through' mode the system <NUM> does not comprise a motion generation device and the container moves with respect to the system <NUM>, the system <NUM> being static with respect to the ground.

The example method illustrated by <FIG>, <FIG> and <FIG> comprises, at S1, classifying an inspection image <NUM> of the container <NUM> in a matching class Cm of one or more predetermined classes Cn of containers of interest.

In the example of <FIG>, the type of containers of interest comprises a refrigeration unit of a vehicle. It should be understood that the disclosure may be applied to inspection images of other containers, such as images of cars for example.

There are several types of refrigeration units around the world. In Europe for example, ISO refrigeration units may be divided into <NUM> to <NUM> classes, and most common truck refrigeration unit may be divided into <NUM> classes. In the example of <FIG>, n is comprised between <NUM> and <NUM>, and the matching class Cm is the predetermined class C1.

As illustrated in <FIG>, each predetermined class Cn comprises reference images 2R associated with a type of containers of interest.

As illustrated in <FIG> and <FIG>, the method also comprises, at S2, comparing a shape model of the inspection image <NUM> to corresponding shape models associated with reference images 2R within the matching class Cm.

As illustrated in <FIG> and <FIG>, the method also comprises, at S3, associating the inspection image <NUM> with a matching reference image 2RM, based on the comparison.

As illustrated in <FIG> and <FIG>, the method also comprises, at S4, registering one or more zones <NUM>, <NUM> and <NUM> of the inspection image <NUM> with corresponding one or more zones Z1, Z2 and Z3 of the matching reference image 2RM.

As illustrated in <FIG> and <FIG>, the method also comprises, at S5, mapping differences <NUM> between the inspection image <NUM> and the matching reference image 2RM, based on the registration.

As illustrated in <FIG> and <FIG>, classifying at S1 may comprise:.

In some examples, extracting the image features comprises using a pre-trained Convolutional Neural Network.

In some examples, the classifier trained at S13 and applied at S15 comprises a multi-class Support Vector Machine, SVM. The number of classes of the SVM is equal to the number of predetermined classes.

As shown by <FIG>, once the inspection image <NUM> is classified in a matching class, there is still intra-class variability. Therefore, comparing, at S2, the shape models comprises, using a statistical shape model, SSM.

In some examples, the SSM may comprise at least one of Active Shape Models, ASM, and/or Active Appearance Models, AAM, and/or Constrained Local Models, CLM.

Examples of SSM are known, for example from <NPL>, from <NPL>, and from <NPL>.

In <FIG> and <FIG> S21 illustrates training the SSM on the reference images 2R to obtain vectors of parameters of the reference images 2R. The vectors of parameters of e.g. N reference images 2R are referenced by pi, with i comprised between <NUM> and N.

In some examples, as shown in <FIG>, training the SSM comprises, for each reference image 2R, annotating the reference image 2R by indicating one or more landmarks LM in the reference image 2R.

In some examples, training at S21 the SSM further comprises obtaining the vector of parameters pi based on the annotation of the reference image.

Obtaining the vector of parameters pi of an image, for example a reference image 2R, comprises:.

In some examples, the coordinates may be coordinates (x, y) of each landmark in the reference image 2R.

Examples of Grey Level Profiles are known, for example from from <NPL>. As shown in <FIG>, each Grey Level Profile may correspond to samples of a grey level profile along the normal to a shape defined by a landmark LMi.

In some examples, obtaining the vector of parameters pi may further comprise applying Procrustes Analysis prior to approximating the distribution model. <FIG> shows shape representations in <NUM> modes, obtained after Procrustes Analysis.

As shown in <FIG> and <FIG>, using the SSM comprises at S22, applying the trained SSM to the inspection image <NUM> to obtain a vector of parameters of the inspection image. The vector of parameters of the inspection image <NUM> is referenced by pIm. Obtaining the vector of parameters pIm of the inspection image <NUM> comprises the same steps as the steps performed to obtain the vector of parameters pi of the reference images 2R.

In some examples, associating, at S3, the inspection image <NUM> with the matching reference image 2RM comprises applying a k-nearest neighbours, k-NN, algorithm on the obtained shape vectors pi and pIm, i.e. the obtained shape vector pIm of the inspection image and the shape vectors pi of the reference images.

The obtained vectors of parameters have a low dimension (for example <NUM> or <NUM> dimensions). Each of the vectors pi can thus be compared with the vector pIm, and the nearest neighbour pNN among the vectors pi can be identified.

In some examples, in the registration performed at S4 the inspection image <NUM> is considered as a fixed image, while the matching reference image 2RM is considered as a moving image. This may reduce the risk that an object of interest in the inspection could be deformed or attenuated.

In some examples, both fixed and moving images may be split into different zones, and the registration performed at S4 may thus be assimilated to a piecewise-rigid model.

As shown in <FIG> and <FIG>, registering, at S4, the one or more zones (e.g. zones <NUM>, <NUM> or <NUM>) of the inspection image <NUM> with the corresponding one or more zones (e.g. zones Z1, Z2 or Z3) of the matching reference image 2RM comprises, for each zone of the inspection image <NUM> and for each corresponding zone of the matching reference image 2RM:.

In some examples, the feature points <NUM> may comprise at least one of Scale-Invariant Feature Transform (SIFT) points or Speeded Up Robust Features (SURF) points. Iteratively, SIFT or SURF feature points are generated, and a correspondence is determined for corresponding zones of the inspection image and of the matching reference image. In some example, a filtering algorithm, such as the algorithm known as RANdom SAmple Consensus (RANSAC) algorithm may be used to filter outlier correspondences.

In some examples, registering, at S4, may further comprise for each zone of the matching reference 2RM:.

An example of the determined displacement vector field is shown in <FIG>. The displacement vector field may be dense, with discontinuities at boundaries between the zones Z1, Z2 and Z3 in the matching reference image 2RM.

The Direction Diffusion method applied at S45 enables to obtain a smoother field. An example of Direction Diffusion method is known from <NPL>. It may normalize the displacement vectors and evolve simultaneously the isotropic diffusion equation for both X and Y components of the field I: <MAT> for i=<NUM> and i=<NUM>, where i=<NUM> is associated with the X component of the displacement vector field, and i=<NUM> is associated with the Y component of the displacement vector field.

The application of a Gaussian kernel with a maximum value at the boundaries may also avoid the normalization at other regions of the image 2RM.

In some examples, as shown in <FIG>, mapping, at S5, the differences <NUM> comprises:.

In some examples, extracting at S51 may comprise extracting an image Diff of the difference between the inspection image <NUM> and the registered matching reference 2RM image by determining: <MAT> where IReg is associated with the registered matching reference image 2RM, and Ifixed is associated with the inspection image <NUM>.

Alternatively or additionally, extracting at S51 may comprise extracting an image Div of the division of the inspection image <NUM> by the registered matching reference image 2RM by determining: <MAT> <MAT>.

The image Div derives from Lambert's law.

An example of the image Diff is shown in <FIG>.

As shown by <FIG>, <FIG>, the method may further comprise obtaining, at S52, the binarized map of the mapped differences. In some examples obtaining the binarized map at S52 comprises applying an adaptive thresholding to the extracted image (for example the image Diff illustrated, e.g. in <FIG> - but this could also be applied to the image Div), to obtain the image illustrated in <FIG>. In some examples, the method may further comprise, at S52, applying a morphological operation to the extracted image after the adaptive threshold has been applied, i.e. the image illustrated in <FIG>, to obtain the image of the binarized mapped differences <NUM> illustrated in <FIG>.

Each binarized mapped difference <NUM> in the image shown in <FIG> corresponds to a region containing a potential object of interest, such as a threat or contraband.

Applying at S53 the trained SVM may comprise applying the binary SVM on the mapped differences <NUM> obtained by the binarization performed at S52, as shown in <FIG>.

In some examples, training, the binary SVM comprises:.

In some examples, the criteria comprise at least one of: a size, a shape, an intensity or a texture. During the training, the SVM will learn to discard binarized mapped differences <NUM> as false alarms if the binarized mapped differences <NUM> do not meet the criteria, for example because they are too small (such as smaller than 10cmx10cm, depending on the container and the objects of interest) or are too faint on the image, or because they are located in certain locations (this relies on the knowledge about specific locations of the containers which are unlikely to contain e.g. illicit products).

After the trained SVM is applied at S53, a probability that the mapped difference corresponds to an object of interest, based on the application of the SVM, is determined at S54.

As shown by <FIG>, the method may further comprise outputting a signal, based on the mapping, for example derived from the steps performed at S54. The signal may be configured to cause a visual alarm, such as the display of the square <NUM> on the inspection image <NUM>, indicating the location of an object of interest. Alternatively or additionally, the signal may be configured to cause an aural alarm, to attract attention of a user when an object of interest is found. A user may thus search the container for checks.

In some examples, the method may comprise, prior to the classifying performed at S1, determining that the inspection image <NUM> corresponds to a container of interest by:.

Examples of Histograms of Gradients are known from <NPL>. Examples are also shown in <FIG>. In <FIG>, the Histogram of Gradients corresponds to a container of interest (e.g. a refrigeration unit), whereas in <FIG>, the Histogram of Gradients does not correspond to a container of interest. The comparing may comprise using a trained SVM.

In the disclosure above, the inspection image initially corresponds to a container to be inspected, such as an initial image of a refrigeration unit. As illustrated in <FIG>, it may be the case that the initial inspection image does not correspond to the container to be inspected, for example an initial image of a truck is inspected and only the refrigeration unit of the truck is of interest. In such examples, the method may further comprise generating the inspection image by selecting a zone of interest in an initial image of the container. As illustrated in <FIG>, the selecting may comprise:.

As shown in <FIG>, it will be appreciated that in examples of the method in accordance with the disclosure, the analyser <NUM> may be configured to retrieve the reference images from a database <NUM> over a communication network <NUM>, thanks to a communication server <NUM> configured to provide a remote data management system. Alternatively or additionally, the database <NUM> may be at least partially located in the analyser <NUM>.

In the example illustrated by <FIG>, the server <NUM> may also provide access to the database <NUM> to a plurality of geographically distributed analysers <NUM>, over the network <NUM>.

In some examples, the database <NUM> may be populated by inspection images inspected by the inspection system <NUM>. This enables enriching and/or updating the database <NUM>.

Alternatively or additionally, the inspection image <NUM> may be retrieved by the analyser <NUM> from the database <NUM>.

The disclosure may be applied to situations where a container (such as shipping container or a vehicle) is inspected by a system <NUM> (for example defined by an energy spectrum and a geometry), at a first location. The analyser <NUM> may perform the comparison of the inspection image <NUM> with the reference images 2R stored in the database <NUM>, regardless of the location of the analyser <NUM> and/or the database <NUM> with respect to the system <NUM>. In some examples, the analyser <NUM> and/or the database <NUM> may be at a second location, different from the first location. In some examples, the reference images 2R correspond to inspection images inspected by a system similar or identical, in terms of energy spectrum and/or geometry, to the system <NUM>.

Other variations and modifications of the system or the analyser will be apparent to the skilled in the art in the context of the present disclosure, and various features described above may have advantages with or without other features described above.

For example, the analyser <NUM> and/or the database <NUM> may, at least partly, form a part of the inspection system <NUM>.

It is understood that the inspection radiation source may comprise sources of other radiation, such as gamma rays or neutrons. The inspection radiation source may also comprise sources which are not adapted to be activated by a power supply, such as radioactive sources, such as using Co60 or Cs137.

As one possibility, there is provided a computer program, computer program product, or computer readable medium, comprising computer program instructions to cause a programmable computer to carry out any one or more of the methods described herein. In example implementations, at least some portions of the activities related to the analyser <NUM> and/or the communications networks <NUM> and/or <NUM> herein may be implemented in software. It is appreciated that software components of the present disclosure may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques.

In some examples, components of the analyser <NUM> and/or the communications networks <NUM> and/or <NUM> may use specialized applications and hardware.

As will be apparent to the skilled in the art, the server <NUM> should not be understood as a single entity, but rather refers to a physical and/or virtual device comprising at least a processor and a memory, the memory may be comprised in one or more servers which can be located in a single location or can be remote from each other to form a distributed network (such as "server farms", e.g., using wired or wireless technology).

In some examples, one or more memory elements (e.g., the database <NUM> and/or the memory of the processor) can store data used for the operations described herein. This includes the memory element being able to store software, logic, code, or processor instructions that are executed to carry out the activities described in the disclosure.

A processor can execute any type of instructions associated with the data to achieve the operations detailed herein in the disclosure. In one example, the processor could transform an element or an article (e.g., data) from one state or thing to another state or thing. In another example, the activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software/computer instructions executed by a processor) and the elements identified herein could be some type of a programmable processor, programmable digital logic (e.g., a field programmable gate array (FPGA), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM)), an ASIC that includes digital logic, software, code, electronic instructions, flash memory, optical disks, CD-ROMs, DVD ROMs, magnetic or optical cards, other types of machine-readable mediums suitable for storing electronic instructions, or any suitable combination thereof.

The communications network <NUM> and the communications network <NUM> may form only one network.

The data received by the analyser <NUM> may be typically received over a range of possible communications networks <NUM> and/or <NUM> at least such as: a satellite based communications network; a cable based communications network; a telephony based communications network; a mobile-telephony based communications network; an Internet Protocol (IP) communications network; and/or a computer based communications network.

In some examples, the communications networks <NUM> and/or <NUM> and/or the analyser <NUM> may comprise one or more networks. Networks may be provisioned in any form including, but not limited to, local area networks (LANs), wireless local area networks (WLANs), virtual local area networks (VLANs), metropolitan area networks (MANs), wide area networks (WANs), virtual private networks (VPNs), Intranet, Extranet, any other appropriate architecture or system, or any combination thereof that facilitates communications in a network.

Claim 1:
A computer-implemented method for inspecting a container, comprising:
classifying an inspection image of the container in a matching class of one or more predetermined classes of containers of interest, each predetermined class comprising reference images associated with a type of containers of interest,
wherein the inspection image is generated using transmission of inspection radiation through the container,
the classifying comprising applying a trained classifier to a classification descriptor of the inspection image generated by extracting image features of the inspection image,
wherein the classifier has been trained by applying the classifier to a classification descriptor of each of the reference images, the classification descriptor of each of the reference images being generated by extracting image features of the reference images;
comparing a shape model of the inspection image to corresponding shape models associated with reference images within the matching class,
wherein comparing the shape models comprises applying a trained statistical shape model, SSM, to the inspection image to obtain a vector of parameters of the inspection image, and
wherein the SSM has been trained on the reference images to obtain vectors of parameters of the reference images;
associating the inspection image with a matching reference image, based on the comparison between each of the vectors of parameters of the reference images and the vector of parameters of the inspection image;
registering one or more zones of the inspection image with corresponding one or more zones of the matching reference image; and
mapping differences between the inspection image and the matching reference image, based on the registration.