Patent Description:
<NPL> discloses random projection techniques to reduce dimensionality of visual content. A conceptual space, induced from training data, is used to perform automatic image annotation.

Features of the present disclosure are illustrated by way of examples shown in the following figures. In the following figures, like numerals indicate like elements, in which:.

For simplicity and illustrative purposes, the present disclosure is described by referring to examples thereof. It will be readily apparent however that the present disclosure may be practiced without limitation to these specific details.

An automatic image annotation system that receives a reference image that includes one or more annotations along with at least one query image and automatically annotates the query images is disclosed. The reference image and the at least one query image can include objects of the same or different classes. But the reference image and the query image also include regions of interest with a similarity that can be identified by the human eye as well as by image analysis machines. The annotation(s) on the reference image can include one or more markings in the form of open or closed geometric figures that highlight the ROI which can include specific objects or portions of specific objects captured in the reference image. The automatic image annotation system provides an annotation that is substantially similar in size, appearance, and position to the annotation in the reference image. Therefore, an annotation is automatically generated for a region in the query image that is identified as being similar to the region highlighted by the annotations in the reference image.

The reference image and the query image are initially processed via SVD to obtain a corresponding singular value diagonal matrix or the S-matrix for each of the reference image and the one or more query images. Also, a lower-dimensional reference image and a lower-dimensional query image are generated by a pre-trained deep learning model. A target annotation2vec which is a vector representation for the reference image is generated from the S-matrix of the reference image and the low dimensional reference image. Similarly, a query image2vec which is a vector representation of the query image is also generated from the S-matrix of the query image and the low dimensional query image. A distance metric is calculated between the vector representation of the reference image and the vector representation of the query image. As a result, a preliminary output image is generated which includes a preliminary annotation on the query image. However, the preliminary annotation may not accurately delineate the entire ROI in the query image in a manner that is substantially similar to the annotations in the reference image. Rather, if the preliminary annotation is a closed geometric figure, e.g., a boundary box, it may only partially enclose the ROI. Therefore, the preliminary annotation is adjusted or resized to more accurately cover the ROI in the query image that corresponds to the annotated ROI in the reference image. An improper preliminary annotation that does not adequately cover the ROI in the query image can be optimized via implementing the expectation maximization (EM) algorithm which is an iterative process that enables identifying the area around and outside the preliminary annotation which can be included within the annotation. The EM algorithm is based on a threshold value or epsilon value that can be determined either manually or automatically via an ML-based model which can include linear regression models or non-linear regression models. The ML-based model is trained on training data which can include different images wherein each image includes one or more optimized annotations and the training data further includes threshold values corresponding to the optimized annotations.

Training of ML models for various purposes such as object detection, image segmentation, image classification or image to text conversion, etc., requires labeled training data which may include annotated and/or labeled images, for the supervised ML purposes. However, generating the labeled training data requires time and effort. Furthermore, if the training data includes errors, the ML models may not achieve sufficient accuracy to be used in production systems. The automatic image annotation system disclosed herein enables a technical solution for generating training data on a large scale within a short period by automatically annotating images. The automatic annotation of images not only produces a large volume of training data but also mitigates errors that might arise in the training data. The automatic image annotation system improves techniques such as template matching which simply identify locations of a given template image within a larger image. More particularly, the automatic image annotation system provides for an unsupervised process of annotating images. A single annotated reference image provides the necessary input for the automatic image annotation system to process the image embeddings as well as the area to be annotated irrespective of the position of the ROI in the query images. Furthermore, the automated annotation system employs lightweight algorithms so that a single input annotation is adequate to find similar objects. The accuracy of the automated annotation system is further enhanced via the use of Bayesian Statistical techniques such as the EM. The automated annotation system can be made available not only as a cloud service shared between users but also as a stand-alone service for specific users.

<FIG> shows a block diagram of the automatic image annotation system <NUM> in accordance with the examples disclosed herein. In an example, the automatic image annotation system <NUM> can be hosted on a cloud server <NUM> and is accessible to different user devices 192a, 192b, etc. The automatic image annotation system <NUM> can be configured to receive from one of the user devices, e.g., the user device 192a, a reference image <NUM> wherein at least a portion of an imaged object is annotated (i.e., the ROI) and at least one query image <NUM> without any annotations. The automatic image annotation system <NUM> analyzes the reference image <NUM> with the annotation and the query image <NUM> as detailed herein to automatically generate one or more annotations on the query image <NUM> in a manner that is similar to the one or more annotations included in the reference image <NUM>. The reference image <NUM> and the query image <NUM> can include objects which may belong to similar or different classes. However, the reference image <NUM> and the query image <NUM> include at least certain similar image regions wherein the image region in the reference image <NUM> similar to an image region of the query image <NUM> is annotated and forms the ROI. An example with objects of the same class can include the reference image <NUM> with an annotated backside of a car of a particular make and model e.g., a Honda hatchback with a number plate while the query image <NUM> includes a car of a different make and model such as a sportscar e.g., a Ferrari. The automatic image annotation system <NUM> is configured to identify annotate the backside of the Ferrari in the query image <NUM> based on an input of a single reference image. In another example, the reference image <NUM> can include an image of a door with a number while the query image <NUM> includes a street sign and/or a letterbox with a number. Here, the objects in the images are of different classes while having similar regions of interest that include numbers. While only a single reference image with the annotation needs to be provided to the automatic image annotation system <NUM>, one or more query images each having different objects but regions of interest similar to the ROI of the reference image <NUM> may be supplied for annotation. It can be appreciated that the objects and models are discussed herein for illustration purposes only and that the automatic image annotation system <NUM> can be configured to receive a reference image and one or more query images of any living or non-living objects for annotation as described herein.

The automatic image annotation system <NUM> includes an input receiver <NUM>, an image analyzer <NUM>, an image annotator <NUM>, and an annotation optimizer <NUM>. The input receiver <NUM> receives the reference image <NUM> including the annotation and the query image <NUM> uploaded by the user device 192a. The reference image <NUM> and the query image <NUM> can include image data of different formats, e.g., jpeg,. gif, etc. The image analyzer <NUM> is configured to generate a sigma matrix or an S-matrix for each of the reference image <NUM> and the query image <NUM> via singular value decomposition. Accordingly, two S-matrices <NUM> and <NUM> can be generated by the image analyzer <NUM> and stored in a data store <NUM> that is used to store received information and information that is generated during the operation of the automatic image annotation system <NUM>. If more than one query image is received, then an S-matrix is generated for each of the query images. Furthermore, the image analyzer <NUM> is also configured to generate a lower-dimensional image for each of the received images. Accordingly, a lower-dimensional reference image <NUM> corresponding to the reference image <NUM> and a lower-dimensional query image <NUM> corresponding to the at least one query image <NUM> are generated. In an example, the images received from the user device 192a can be provided to a pre-trained deep learning network to generate the lower-dimensional images.

The S-matrices <NUM> and <NUM> and the lower-dimensional images <NUM> and <NUM> are used to generate vector representations of the received images including the reference image <NUM> and the query image <NUM>. Therefore, a target annotation2vec and a query image2vec are generated as vector representations <NUM> corresponding to the reference image <NUM> and the query image <NUM>. The vector representations <NUM> including the target annotation2vec and the query image2vec are supplied to the image annotator <NUM>. The image annotator <NUM> determines a distance metric between the reference image <NUM> and the query image <NUM> based on the vector representations <NUM> - the target annotation2vec and the query image2vec. Various distance metrics as detailed herein can be used. Based on the distance metric, a preliminary output image <NUM> is generated wherein the preliminary output image <NUM> includes the query image <NUM> with a preliminary output annotation that covers at least a portion of the ROI in the query image <NUM> which is similar to the annotated ROI in the reference image <NUM>.

The preliminary output annotation in the preliminary output image <NUM> may not accurately delineate the ROI in the query image. For example, the preliminary output annotation may cover a larger or smaller area than the ROI in the query image <NUM>. Therefore, the annotation optimizer <NUM> employs optimization procedures to adjust the size of the preliminary output annotation to generate a finalized output image <NUM> with a finalized annotation that accurately covers the entire ROI in the query image <NUM>. In an example, the annotation optimizer <NUM> can implement an optimization procedure based for example, on EM.

<FIG> shows a block diagram of the image analyzer <NUM> in accordance with the examples disclosed herein. The image analyzer <NUM> includes a matrix extractor <NUM>, a reduced image generator <NUM>, and a vector generator <NUM>. The matrix extractor <NUM> extracts the S-matrix <NUM> of the reference image <NUM> and the S-matrix <NUM> of the query image <NUM> from the SVD of the corresponding image matrices. Each Sigma or S-matrix contains the topic "singular values" in a square diagonal matrix. The S-matrix can be indicative of the amount of information that is captured.

The reduced image generator <NUM> generates lower-dimensional images which include the lower-dimensional reference image <NUM> representative of the reference image <NUM> and the lower-dimensional query image <NUM> corresponding to the query image <NUM>. The reduced image generator <NUM> includes a pre-trained deep learning model <NUM> which requires no further training to generate the lower-dimensional images. Rather, the terminal layer of the pre-trained deep learning model <NUM> is discarded in order to obtain the lower-dimensional images. In an example, the pre-trained deep learning model <NUM> can include a <NUM>-layer Residual Neural Network (ResNet). The vector generator <NUM> accesses the S-matrix <NUM> and the S-matrix <NUM> along with the lower-dimensional reference image <NUM> and the lower-dimensional query image <NUM> to obtain the vector representations <NUM> which include the target annotation2vector corresponding to the reference image <NUM> and the query image2vector corresponding to the query image <NUM>. More particularly, the vector generator <NUM> can combine the S-matrix <NUM> with a matrix representation (e.g., an image matrix) associated with the lower-dimensional reference image <NUM> via a matrix operation (e.g., matrix multiplication) to generate the target annotation2vector corresponding to the reference image <NUM>. The query image2 vector corresponding to the query image <NUM> may also be generated similarly via a matrix operation between the S-matrix <NUM> and the lower-dimensional query image <NUM>.

<FIG> shows a block diagram of the image annotator <NUM> in accordance with the examples disclosed herein. The image annotator <NUM> includes a distance calculator <NUM>, an ROI identifier <NUM>, and an annotation generator <NUM>. The image annotator <NUM> accesses the vector representations <NUM> and calculates a distance metric <NUM> between the vector representations <NUM>. The distance metric <NUM> is a measure of different between two entities that can be represented or obtained using various methods. The simplest example of a distance metric might be the absolute difference between two numbers representing the two different entities or properties or images in the present instance. A metric or distance function is a function d(x,y) that defines the distance between elements of a set as a non-negative real number. If the distance is zero, both elements are equivalent under that specific metric. A typical distance for real numbers is the absolute difference,
<MAT>.

After processing the reference image <NUM> and the query image <NUM>, the vector representations <NUM> of real numbers are produced. Different parts of the images can be represented as different components of the corresponding vector representations that represent the images in n-dimensional Euclidean space. Similar images are closer together in the vector space. In an example, cosine distance measure can be employed by the distance calculator <NUM> to obtain the distance metric. Based on the distance metric, between the various components of the vector representations <NUM>, the ROI identifier <NUM> is enabled to identify at least a portion of the ROI i.e., the preliminary ROI <NUM> from the query image <NUM> that would be similar (or closer to) the annotated part of the object in the reference image <NUM>. In an example, a pixel-by-pixel distance comparisons can be executed by the ROI identifier <NUM> to identify the preliminary ROI <NUM> from the query image <NUM>. In an example, a preset threshold distance can be used by the ROI identifier <NUM> to determine the similarity between the pixels in the query image <NUM> to the pixels in the reference image <NUM>. The positions of the similar pixels from the query image <NUM> can be provided to the annotation generator <NUM> to produce a preliminary output image <NUM> with a preliminary annotation <NUM> around the preliminary ROI <NUM>.

The preliminary annotation <NUM> can be similar in shape and/or color to the reference annotation <NUM> of the reference image while the position of the preliminary annotation <NUM> within the preliminary output image <NUM> depends on the image region that is identified as the preliminary ROI <NUM>. If the annotation of the reference image <NUM> is an open figure such as an arrow, a line, etc., that points to the ROI as opposed to enclosing the ROI in the reference image <NUM>, then the process of automatic annotation generation may be terminated with the generation of the preliminary annotation <NUM>. However, if the annotation in the reference image <NUM> encloses the ROI such as the reference annotation <NUM>, then further optimization may be required. This is because the preliminary ROI <NUM> is identified based on the regional similarity between the annotated part of the reference image <NUM> and corresponding portions of the query image <NUM>. The preliminary ROI <NUM> is therefore delineated by the preliminary annotation <NUM> which should ideally cover the entire ROI corresponding to the query image <NUM> in the preliminary output image <NUM>. However, due to different reasons the preliminary annotation <NUM> may not delineate the entire ROI in the query image <NUM> as accurately done in the reference image <NUM>. This is shown, for example, in the comparison of the details of the reference image <NUM> and the preliminary output image <NUM> wherein a reference annotation <NUM> covers an entire backside of the car in the reference image <NUM> whereas the preliminary annotation <NUM> covers only the back windshield of the car in the preliminary output image <NUM>. The annotation optimizer <NUM> optimizes the preliminary annotation <NUM> so that the ROI is identified more accurately as detailed herein.

<FIG> shows a block diagram of the annotation optimizer <NUM> in accordance with the examples disclosed herein. The annotation optimizer <NUM> includes a threshold generator <NUM>, a threshold receiver <NUM>, and an annotation adjuster <NUM>. Optimization is the process of modifying a parameter or target variable to achieve optimum value. In an example, the annotation optimizer <NUM> can implement an optimization process based on EM. The EM algorithm enables finding maximum-likelihood estimates for model parameters when the data is incomplete, has missing data points, or has unobserved (hidden) latent variables. It is an iterative way to approximate the maximum likelihood function. More specifically, the annotation optimizer <NUM> is configured to find the area around the preliminary annotation <NUM> (or the boundary box) which covers the similar distribution of pixel values as the area within the reference annotation <NUM>. Thus, the boundary box or the preliminary annotation <NUM> is resized to accommodate the entire ROI of the query image. The optimization can be controlled using a threshold <NUM> or an 'epsilon' parameter which is the measure of the difference between the distributions that represent the images. In an example, the annotation optimizer <NUM> uses the absolute difference between the mean of the distributions. However, the choice is not limited to this particular selection. Any metric can be used for this purpose including KL-Divergence, for instance, to estimate the similarity/difference of both distributions. The threshold <NUM> value depends on an object captured in the images. Different objects such as cars, houses, dogs, balls, etc., can have different thresholds. The threshold <NUM> can determine the extent to which the preliminary annotation <NUM> needs to be adjusted to optimally cover the ROI of the query image to the extent covered by the reference annotation <NUM>. In an example, the value for the threshold <NUM> can be received from a user <NUM> in response to a display of the preliminary output image <NUM> using a dashboard which is one of the user interfaces that may be put forth by the automatic image annotation system <NUM> for user interactions that enable receiving data and outputting the results. When the threshold is provided by the user <NUM>, the threshold receiver <NUM> obtains the threshold <NUM> and provides it to the annotation adjuster <NUM> to adjust the preliminary annotation <NUM>. However, threshold <NUM> can also be automatically generated by the threshold generator <NUM>. In an example, the threshold generator <NUM> can include an ML-based model <NUM> which is trained using training data <NUM>. The training data <NUM> can include various images with annotations and the corresponding thresholds (or epsilon values) to be used to generate the annotations accurately. The threshold <NUM> thus generated can be provided to the annotation adjuster <NUM> which causes resizing of the annotation to generate a finalized annotation <NUM>. A finalized output image <NUM> is provided with the finalized annotation <NUM> that accurately covers the entire ROI <NUM> in a manner that is similar to the ROI covered by the reference annotation <NUM> in the reference image <NUM>.

<FIG> shows a flowchart <NUM> that details a method of automatically annotating images in accordance with examples disclosed herein. The method begins at <NUM> wherein the reference image <NUM> with the reference annotation <NUM> and the query image <NUM> with no annotations are received. While only one annotated reference image is needed, a plurality of query images can be uploaded to the automatic image annotation system <NUM> for automatic annotation as described herein. At <NUM>, the S-matrix <NUM> (or the singular value diagonal matrix) for the reference image <NUM> and the S-matrix <NUM> the query image <NUM> are obtained via the SVD process. At <NUM>, the lower-dimensional reference image <NUM> for the reference image <NUM> and the lower-dimensional query image <NUM> for the query image <NUM> are generated by the pre-trained deep learning model <NUM>. The S-matrices <NUM>, <NUM>, and the corresponding lower-dimensional images <NUM>, <NUM> are used at <NUM> to generate the vector representations <NUM> which include the target annotation2vector and query image2vector for the reference image <NUM> and the query image <NUM> respectively. The distance metric <NUM> is determined at <NUM> between the target annotation2vector and the query image2vector using one of the various distance metric measures such as Euclidean distance metric, Cosine distance metric, etc. Based on the distance metric <NUM>, the preliminary ROI <NUM> is initially identified in the query image <NUM> at <NUM>. The preliminary annotation <NUM> around the preliminary ROI <NUM> is automatically generated by the annotation generator <NUM> at <NUM>. The preliminary annotation <NUM> is further optimized at <NUM> to produce the finalized output image <NUM> at <NUM> with the finalized annotation <NUM>.

<FIG> shows a flowchart <NUM> that details a method of optimizing the annotation via accurately identifying the ROI in accordance with the examples disclosed herein. The method begins at <NUM> wherein the preliminary output image <NUM> with the preliminary annotation <NUM> and the preliminary ROI <NUM> is accessed. At <NUM>, the threshold <NUM> or the epsilon value is retrieved. The threshold <NUM> retrieved at <NUM> can be supplied manually by the user <NUM> or may be automatically generated by the ML-based model <NUM>. The threshold <NUM> is applied to the boundary of the preliminary annotation <NUM> at <NUM>. More particularly, the pixels beyond the boundary up to the threshold limit can be compared with the pixels within the preliminary ROI <NUM>. It is determined at <NUM> if the pixels within the preliminary ROI <NUM> are identical to the pixels beyond the boundary of the preliminary annotation <NUM> and up to the threshold limit. If it is determined at <NUM> that the pixels beyond the preliminary ROI <NUM> are not identical, then the pixels are discarded from consideration at <NUM> and the process terminates on the end block. If it is determined at <NUM> that the pixels beyond the preliminary ROI <NUM> are identical to the pixels in the preliminary ROI <NUM>, then the preliminary annotation <NUM> is resized or expanded at <NUM> to include the identical pixels. The method moves to <NUM> to determine if further pixels remain to be processed. If no further pixels remain for processing, the method terminates on the end block. If it is determined at <NUM> that further pixels remain for processing, then the method returns to <NUM> to apply the threshold to the boundary of the expanded or resized annotation.

<FIG> shows an example reference image <NUM> and an example query image <NUM> in accordance with the examples disclosed herein. The example reference image <NUM> includes an annotation which forms a bounding box <NUM> that covers the trunk of the car. Before being used for automatic annotation, the example reference image <NUM> can be processed to extract the annotated portion <NUM> of the example reference image <NUM>. The annotated portion <NUM> and the example query image <NUM> can be processed as described herein so that the automatic image annotation system <NUM> can identify and annotate the rear end of the car shown in the example query image <NUM> like the annotation in the example reference image <NUM>.

<FIG> shows a block diagram that illustrates the usage of SVD for automatic annotation of images in accordance with the examples disclosed herein. SVD is a dimensionality reduction technique and is also a factorization of a real or complex matrix, e.g., the data matrix A <NUM>, that generalizes the eigen decomposition of a square normal matrix to any m*n matrix via an extension of the polar decomposition. Specifically, the SVD of an m*n real or complex matrix A <NUM> is a factorization of the form U Sigma V*, where U is an m*m real or complex unitary matrix <NUM>, Sigma is an m*n rectangular diagonal matrix <NUM> with non-negative real numbers on the diagonal, and V is an n*n real or complex unitary matrix <NUM>.

Calculating the SVD consists of finding the eigenvalues and eigenvectors of AAT and ATA, wherein T is the transpose matrix of A. The eigenvectors of ATA make up the columns of the n*n real or complex unitary matrix V <NUM>, the eigenvectors of AAT make up the columns of U the m*m real or complex unitary matrix <NUM>. Also, the singular values in S are square roots of eigenvalues from AAT or ATA. The singular values are the diagonal entries of the S-matrix and are arranged in descending order. The singular values are always real numbers. If the matrix A is a real matrix, then U and V are also real.

The SVD transformation is invoked mainly in several applications, such as image compression, image hiding, watermarking, noise reduction, and image watermarking. The SVD of an M × M matrix referred to as Rm, which represents the input image, is a decomposition of the form Rm = USV. As mentioned above, U and V are orthogonal matrices, and S is a diagonal matrix consisting of the singular values of Rm. The singular values s1 ≥ s2 ≥··· ≥ sm ≥ <NUM> are in descending order along the main diagonal of S. These singular values are obtained by calculation of the square root of the eigenvalues of RmRmT and RmTRm. The singular values are unique; however, the matrices U and V are not unique. Hence, the matrices U <NUM> and V <NUM> are discarded from further processing by the automatic image annotation system <NUM> in the automatic image annotation process. The middle singular matrix or the S-matrix can be extracted using a function:.

<FIG> shows a schematic diagram <NUM> of the pre-trained deep learning model <NUM> used for generating the lower-dimensional images in accordance with the examples disclosed herein. The convolution-based deep learning models can be employed for most of the tasks related to images including classification, segmentation, etc. These models follow a hierarchical structure called layers as shown in representation <NUM>. These layers capture the features of the input images <NUM> e.g. the reference image <NUM> and the query image <NUM> and encode the input images into an N-dimensional Euclidean space. The vector obtained from the last layer of the model encapsulates the features of the input images and could serve as low dimensional representation for subsequent processing.

The schematic diagram <NUM> shows a <NUM> layer ResNet <NUM> from which the final layer <NUM> is discarded. The output of the last MaxPooling2D operation can include one or more feature maps, e.g., feature map <NUM>. Using the transfer learning from the pre-trained deep learning model <NUM> enables using the latent feature maps, extracting them and then finding the mutual features between two images which enables automatically annotating the query image <NUM> based on the reference image <NUM>. The automatic image annotation system <NUM> combines the feature maps with SVD to match the ROIs on the reference images and query images. Therefore, the mutual information between the two images is maximized before estimating the distance metric.

<FIG> shows the various distance metrics that can be employed by the automatic image annotation system <NUM> in accordance with the examples disclosed herein. Some example distance plots and/or equations are shown. The Euclidean distance metric <NUM> determines the straight line distance between two points in the Euclidean space. The Euclidean distance between two points is the length of the line segment connecting the two points. The Manhattan distance metric <NUM> is another metric that can be employed by the automatic image annotation system <NUM> for the automatic annotation generation. Manhattan distance is the distance between two points measured along the axes at right angles. The Minkowski distance <NUM> is a metric in a normed vector space which can be considered as a generalization of both the Euclidean distance and the Manhattan distance. The Chebyshev distance <NUM> is the maximum metric, or L∞ metric is a metric defined on a vector space where the distance between two vectors is the greatest of their differences along any coordinate dimension. Cosine similarity <NUM> measures the similarity between two vectors of an inner product space. It is measured by the cosine of the angle between two vectors. While cosine similarity is often used to measure document similarity in text analysis, the automatic image annotation system <NUM> uses cosine similarity to measure the distance between vector representations obtained from the S-matrices and the lower-dimensional images of the reference image and the query images. Hamming distance <NUM> is another distance metric that can be employed by the automatic image annotation system <NUM>. Hamming distance <NUM> between two strings of equal length is the number of positions at which the corresponding symbols are different.

<FIG> shows the annotations that are generated at various steps during the automatic annotation generation process and the various bounding boxes that are generated during the optimization process for the various threshold values in accordance with the examples disclosed herein. Different annotations are shown in the example query image <NUM>. The SVD output annotation <NUM> which is obtained automatically using only the S-matrix from the SVD does not cover the rear end of the car in the example query image <NUM>. The SVD+pre-trained output <NUM> which is obtained via combining the S-matrix <NUM> with the lower-dimensional query image <NUM> covers the rear end of the car adequately. However, when compared to the annotation in the example reference image <NUM>, the SVD+pre-trained output <NUM> can be further optimized to better cover the rear end of the car like the annotation in the example reference image <NUM>. Finally, the optimized output <NUM> from the expectation maximization accurately covers the rear part of the car.

Thresholding is a hyperparameter that is denoted by epsilon 'ε' which can be determined by a user or may be predefined within the automatic image annotation system <NUM> so that the threshold value can be selected by the ML-based model <NUM>. The various bounding boxes that are generated for different values of epsilon and the corresponding values of the epsilon are shown in the schematic figures <NUM>. As seen from the schematic figures <NUM>, figure <NUM> has the best annotation <NUM> which is optimized to enclose the ROI <NUM>.

<FIG> illustrates a computer system <NUM> that may be used to implement the automatic image annotation system <NUM>. More particularly, computing machines such as desktops, laptops, smartphones, tablets, and wearables which may be used to generate or access the data from the automatic image annotation system <NUM> may have the structure of the computer system <NUM>. The computer system <NUM> may include additional components not shown and that some of the process components described may be removed and/or modified. In another example, a computer system <NUM> can sit on external-cloud platforms such as Amazon Web Services, AZURE® cloud or internal corporate cloud computing clusters, or organizational computing resources, etc..

The computer system <NUM> includes processor(s) <NUM>, such as a central processing unit, ASIC or another type of processing circuit, input/output devices <NUM>, such as a display, mouse keyboard, etc., a network interface <NUM>, such as a Local Area Network (LAN), a wireless <NUM>. 12x LAN, a <NUM>, <NUM> or <NUM> mobile WAN or a WiMax WAN, and a processor-readable medium <NUM>. Each of these components may be operatively coupled to a bus <NUM>. The computer-readable medium <NUM> may be any suitable medium that participates in providing instructions to the processor(s) <NUM> for execution. For example, the processor-readable medium <NUM> may be non-transitory or non-volatile medium, such as a magnetic disk or solid-state non-volatile memory or volatile medium such as RAM. The instructions or modules stored on the processor-readable medium <NUM> may include machine-readable instructions <NUM> executed by the processor(s) <NUM> that cause the processor(s) <NUM> to perform the methods and functions of the automatic image annotation system <NUM>.

The automatic image annotation system <NUM> may be implemented as software stored on a non-transitory processor-readable medium and executed by the one or more processors <NUM>. For example, the processor-readable medium <NUM> may store an operating system <NUM>, such as MAC OS, MS WINDOWS, UNIX, or LINUX, and code <NUM> for the automatic image annotation system <NUM>. The operating system <NUM> may be multi-user, multiprocessing, multitasking, multithreading, real-time, and the like. For example, during runtime, the operating system <NUM> is running and the code for the automatic image annotation system <NUM> is executed by the processor(s) <NUM>.

The computer system <NUM> may include a data storage <NUM>, which may include non-volatile data storage. The data storage <NUM> stores any data used by the automatic image annotation system <NUM>. The data storage <NUM> may be used to store the S-matrices, the lower-dimensional images, the vector representations, etc., and other data that is used or generated by the automatic image annotation system <NUM> during the course of operation.

The network interface <NUM> connects the computer system <NUM> to internal systems for example, via a LAN. Also, the network interface <NUM> may connect the computer system <NUM> to the Internet. For example, the computer system <NUM> may connect to web browsers and other external applications and systems via the network interface <NUM>.

Claim 1:
An automatic image annotation system (<NUM>), comprising:
at least one processor (<NUM>);
a non-transitory processor-readable medium (<NUM>) storing machine-readable instructions that cause the processor to:
receive a reference image (<NUM>; <NUM>) and at least one query image (<NUM>; <NUM>), wherein the reference image and the at least one query image include similar regions of interest (<NUM>; <NUM>) and the reference image further includes at least one annotation (<NUM>; <NUM>) highlighting a region of interest, ROI, therein;
generate an S-matrix for the reference image (<NUM>) and an S-matrix for the at least one query image (<NUM>), wherein the S-matrix for the reference image includes singular values for the reference image and the S-matrix for the at least one query image includes singular values for the at least one query image, and wherein singular value decomposition is employed to generate the S-matrix of the reference image and the S-matrix for the at least one query image;
extract a low dimensional reference image with the at least one annotation from the reference image (<NUM>) using a pre-trained deep learning model (<NUM>), wherein the pre-trained deep learning model includes a residual neural network, ResNet, (<NUM>) without terminal layers (<NUM>);
extract a low dimensional query image from the at least one query image (<NUM>) using the pre-trained deep learning model;
obtain a vector representation of the reference image (<NUM>) using a combination of the low dimensional reference image with the at least one annotation and the S-matrix of the reference image;
obtain a vector representation of the query image (<NUM>) using a combination of the low dimensional query image and the S-matrix of the at least one query image;
determine a distance metric (<NUM>) between the vector representation of the reference image with the at least one annotation and the vector representation of the query image;
identify a preliminary ROI (<NUM>) within the at least one query image based on the distance metric, wherein the preliminary ROI includes at least a portion of the ROI in the reference image delineated by the at least one annotation; and
provide a preliminary output image (<NUM>; <NUM>) based on the distance metric wherein the preliminary output image includes the at least one query image with a preliminary annotation that covers at least a portion of the preliminary ROI in the query image.