Patent Description:
<CIT> discloses a system that selects image frames from a video feed for recognition of objects (such as physical objects, text characters, or the like) within the image frames. The individual frames are selected using historical metrics that compare individual metrics of the particular image (such as focus, motion, intensity, etc.) to similar metrics of previous image frames in the video feed. The system will select the image frame for object recognition if the image frame is relatively high quality, that is the image frame is suitable for a later object recognition processing.

<CIT> discloses methods and systems of image pre-processing. The subject technology classifies an image received from a camera of a mobile computing device into one or more classes: <NUM>) normal background, <NUM>) textured background, <NUM>) image with text, <NUM>) image with barcode, <NUM>) image with QR code, and/or <NUM>) image with clutter or "garbage. " Based on the classes associated with the image, the subject technology may forgo certain image processing operations, when the image is not associated with a particular class.

An invention is set out in the independent claims appended hereto, and embodiments are indicated in the dependent claims that follow.

This specification describes technologies relating to efficient image analysis that enables electronic devices to recognize objects in images and/or extract information from images, and present information or other content related to the images while reducing the consumption of computational and electrical power of the electronic devices.

A first aspect of the invention described in this specification is embodied in devices that include image sensor configured to output image pixel data, an image buffer configured to temporarily store the image pixel data, an environment sensor, and an image processing apparatus. The device can also include a controller configured to determine an expected image quality of the image pixel data based on a signal output by the environment sensor and provide the image pixel data to the image processing apparatus selectively, according to the expected image quality. Other implementations of this aspect include corresponding apparatus, methods, systems, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

In some aspects, if the controller determines the expected image quality of the image pixel data to be below a preset minimum quality threshold, the image pixel data are not provided to the image processing apparatus. If the controller determines the expected image quality of the image pixel data to be equal to or above a preset minimum quality threshold, the image pixel data are provided to the image processing apparatus.

Some aspects include a selected frame buffer. The controller is configured to copy the image pixel data from the image buffer to the selected frame buffer or update a selected frame pointer to the image buffer, according to the expected image quality of the image pixel data and to provide the image processing apparatus with the image pixel data stored in the selected frame buffer.

In some aspects, the controller is configured to compare the expected image quality of the image pixel data in the image buffer to the expected image quality of the image pixel data of the selected frame buffer, and copy the image pixel data from the image buffer to the selected frame buffer or update a selected frame pointer to the image buffer if the expected image quality of the image pixel data in the image buffer equals or exceeds the expected image quality of the selected frame buffer.

In some aspects, the environment sensor includes an inertial sensor, and the controller is configured to determine the expected image quality based on a sensed movement of the apparatus. In some aspects, the controller is configured to determine the expected image quality based on the signal output by the environment sensor and a signal output by the image sensor. In some aspects, the signal output by the image sensor includes at least one of brightness information, focus information or histogram data relating to the image pixel data output by the image sensor.

In some aspects, if the expected image quality of the image pixel data is determined to be below a preset minimum quality threshold, the image pixel data are not provided for image processing.

Some aspects include copying the image pixel data from the image buffer to a selected frame buffer or updating a pointer to the image buffer, according to the expected image quality of the image pixel data. The image pixel data stored in the selected frame buffer is provided for image processing.

In some aspects, the environmental signal includes an inertial signal, and expected image quality is determined based on a sensed movement. The expected image quality is determined based on the environmental signal and an image information signal generated with the image pixel data. The image information signal includes at least one of brightness information, focus information or histogram data relating to the generated image pixel data.

A second aspect aspect of the invention described in this specification is embodied in image processing apparatus that include one or more image recognition modules corresponding to respective object classes. Each image recognition module is configured to identify one or more objects in the respective object class. The image processing apparatus includes a coarse recognition module configured to receive an input image, determine whether the input image includes an image feature which identifies one of the object classes, and provide the input image for processing by the image recognition module which corresponds to the identified object class. Other implementations of this aspect include corresponding systems, methods, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

In some aspects, the coarse recognition module is configured to provide the image recognition module with information indicating a position and/or orientation of the object. In some aspects, the image recognition modules correspond to one or more of a text object class, a landmark object class, a barcode object class, a media object class and an artwork object class.

In some aspects, the one or more image recognition modules are configured to output an identified object in the respective object class with an associated confidence score. The one or more image recognition modules are configured to adjust an output confidence score for an identified object based on a previous output of the image recognition module.

In some aspects, the output confidence score is adjusted based on at least one of an edit distance between the identified object or text within the identified object, and a previously identified object, a location of the previously identified object and the confidence score associated with the previously identified object. Some aspects include a communication unit. The coarse recognition module can be arranged remotely from the one or more image recognition modules.

In some aspects, the coarse recognition module is configured to provide a reduced version of the input image for processing by the image recognition module. The reduced version of the input image is one of a low resolution version, a cropped version, or a vector representation of the input image.

In general, another aspect of the subject matter described in this specification is embodied in image processing methods that include receiving an input image, determining whether the input image includes an image feature which identifies one of one or more object classes, and image processing the input image to identify one or more objects in the identified object class. Other implementations of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

Some aspects include determining information indicating a position and/or orientation of the object. Image processing the input image to identify one or more objects is based on the determined information indicating the position and/or orientation of the object. The object classes include one or more of a text object class, a landmark object class, a barcode object class, a media object class and an artwork object class.

Some aspects include outputting an identified object with an associated confidence score and adjusting an output confidence score for an identified object based on a previous output. In some aspects, the output confidence score is adjusted based on at least one of an edit distance between the identified object and a previously identified object, a location of the previously identified object and the confidence score associated with the previously identified object.

Some aspects include generating a reduced version of the input image for image processing to identify one or more objects. The reduced version of the input image is one of a low resolution version, a cropped version, or a vector representation of the input image.

The first and second aspect of the invention described in this specification is embodied in an image processing system as defined in claim <NUM> that includes a camera configured to capture images, one or more environment sensors configured to detect movement of the camera, a data processing apparatus, and a memory storage apparatus in data communication with the data processing apparatus. The memory storage apparatus stores instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations including accessing, for each of a multitude of images captured by a mobile device camera, data indicative of movement of the camera at a time at which the camera captured the image. The data processing apparatus selects, from the images, a particular image for analysis based on the data indicative of the movement of the camera for each image. The data processing apparatus analyzes the particular image to recognize one or more objects depicted in the particular image and present content related to the one or more recognized objects. Other implementations of this aspect include corresponding methods, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

The operations further include analyzing the particular image using a coarse classifier to detect presence of one or more classes of objects depicted in the particular image. The particular image is analyzed to recognize one or more objects depicted in the particular image in response to detecting the presence of the one or more classes of objects in the particular image.

In some aspects, analyzing the image to recognize one or more objects depicted by the image includes analyzing the image using one or more computer vision techniques. The data indicative of the movement of the mobile device include at least one of (i) inertial acceleration measurements received from an accelerometer sensor of the mobile device or (ii) orientation data received from a gyroscope of the mobile device.

In some aspects, selecting a particular image for analysis includes selecting the particular image based on the particular image having a least amount of movement at the time the camera captured the particular image relative to each an amount of movement at the time the camera captures each other image in the multitude of images. The data indicative of the movement of the mobile device for each image include data describing rotational motion of the mobile device at the time at which the image was captured. In some aspects, selecting a particular image for analysis includes selecting the particular based on the rotational motion of the mobile device at the time at which the particular image was captured being less than a threshold amount of rotational motion.

In some aspects, the operations include receiving a request for an image to be analyzed by the coarse classifier following completion of analysis of a previous image by the coarse classifier. Selecting a particular image for analysis includes selecting the particular image based on the particular image having a least amount of movement relative to other images in a set of images captured by the camera following the previous image being sent to the coarse classifier for analysis.

Analyzing the particular image using the coarse classifier includes initiating analysis by the coarse classifier periodically based on a processing rate. The operations further include adjusting the processing rate based on whether presence of one or more classes of objects has been detected in one or more images analyzed by the coarse classifier over a previous time period. The processing rate can be increased such that more images are analyzed by the coarse classifier per unit time in response to detecting presence of one or more classes of objects in at least one image over the previous time period. The processing rate can be reduced such that fewer images are analyzed by the coarse classifier per unit time in response to not detecting presence of one or more classes of objects in at least one image over the previous time period.

In some aspects, presenting content related to the one or more identified objects can include presenting a results page that include results that include links to resources related to the one or more identified objects. Presenting content related to the one or more identified objects can include presenting, for each identified object, content related to the object in an overlay over the object. Presenting content related to the one or more identified objects can include presenting a selectable user interface element that, when selected by a user, initiates a particular action associated with the one or more identified objects. Presenting content related to the one or more identified objects can include selecting, for each class for which presence of an object of the class was detected by the coarse classifier, a content item related to the class and presenting a content item for each class for which presence of an object of the class was detected by the coarse classifier.

In some aspects, selecting a particular image for analysis based on the data indicative of the movement the movement of the mobile device camera can include selecting the particular image independent of data indicative of visual characteristics of the images. Selecting a particular image for analysis based on the data indicative of the movement of the mobile device camera can include selecting the particular image based on the data indicative of the movement of the mobile device camera for each image in combination with data indicative of visual characteristics of each image. The data indicative of the visual characteristics of each images can include at least one of (i) brightness data, (ii) focus data, or (iii) histogram data.

In some aspects, selecting a particular image for analysis can include determining, for each image, an expected image quality of the image based on the data indicative of the movement of the mobile device camera for the image and selecting, as the particular image, an image having a highest expected image quality. Selecting a particular image for analysis can include determining, for a first image, a first expected image quality of the first image based on the data indicative of the movement of the mobile device camera for the first image. Image data for the first image can be stored in a selected frame buffer based on the first expected image quality of the first image. A second expected image quality of a second image can be determined based on the data indicative of the movement of the mobile device camera for the second image. A determination can be made to replace, in the selected frame buffer, the image data for the first image with the image data for the second image based on the second expected image quality being greater than the first expected image quality.

Some aspects can include analyzing a second image using a coarse classifier to detect presence of one or more classes of objects depicted in the particular image and determining to not analyze the second image to recognize one or more objects depicted in the particular image in response to failure to detect the presence of the one or more classes of objects in the particular image using the coarse classifier.

The invention described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. By selecting images for analysis based on data indicative of the movement of a camera capturing the images, higher quality images can be selected for more detailed analysis quickly and using fewer computing resources and less electrical power than selecting the images based on an analysis of visual characteristics of each image. Storing only a single best image (or fewer than a threshold number of images) of a stream of images for subsequent analysis reduces the amount of consumed memory, freeing up memory space for other applications. These features also ensure that a higher quality image is used in the computer vision analysis, resulting in more accurate vision analysis results and more relevant content being provided based on objects recognized in the image.

Using coarse classifiers to identify the presence of particular classes of objects and only performing further computer vision analysis on images that are classified as depicting an object of one of the classes reduces the number of images analyzed using computer vision analysis techniques. Additionally, a multi-class coarse classifier can limit additional processing to the fine classifier(s) matching the type(s) detected by the coarse classifier. As later-stage computer vision analysis can be more computationally intensive than the coarse classification and the image selection, this greatly reduces the demand placed on computing resources and conserves computational and electrical power for use by other applications. This also can improve the speed at which images are analyzed as it reduces the number of images placed into a queue for analysis. The image selection and analysis techniques described herein also allow for modeless integration of visual analysis into existing applications without compromising the applications' primary behavior (e.g., taking pictures with a camera).

Various features and advantages of the foregoing subject matter are described below with respect to the figures. Additional features and advantages are apparent from the subject matter described herein and the claims.

In general, systems and techniques described herein can reduce the consumption of computing resources and electrical power used to analyze images, while also improving the accuracy of the image analysis, by selectively analyzing images that are expected to be high quality. For example, images may be analyzed to recognize objects depicted in the images and to extract information about objects depicted in the images for the purpose of providing additional information about the objects. A user of a mobile device, e.g., a smart phone or tablet computing device, may capture images of objects using a camera of the mobile device. One or more of the images can be selected for analysis, e.g., based on an expected quality of the images determined based on data indicative of the movement of the camera at the time the images were captured. The selected images can be analyzed to provide additional content for presentation at the mobile device. For example, the content may be displayed with the image in a user interface of the mobile device or results from the analysis may be stored for later use.

Mobile devices typically have less computing power and data storage capacity than desktop computers and servers. Mobile devices also typically have limited battery power and thermal dissipation capabilities. Thus, techniques for performing image analysis in a way that reduces the utilization of computing resources and battery power can improve the functioning and the performance of the mobile devices by preserving the limited computing power and electrical power for other applications and/or processes, and reducing the amount of heat generated by the mobile devices.

In some implementations, image analysis is performed on images of a sequence of images such as a video stream. For example, a user may activate a camera mode that provides content related to objects depicted in images and point the camera at various objects. In another example, a camera application may generally have this feature active when the camera application is active. In these examples, some (e.g., less than all) of the images in the sequence of images may be analyzed to recognize object(s) depicted in the images. The techniques described herein can use one or more low power consuming stages to select images for further analysis to reduce the number of images analyzed and to reduce the amount of computing power and electrical power consumed by the analysis process.

In some implementations, a low power consuming stage can determine an expected image quality of an image. For example, an expected image quality can be determined based on an environment sensor of the device. The determination of an expected image quality may require fewer processing resources than the further analysis to recognize object(s) depicted, and thus less power is consumed. In this way, unnecessary consumption of resources to analyze images having a low image quality may be avoided.

In some implementations, a low power consuming stage can determine whether an image includes an image feature which identifies an object class. An image including such an image feature can be provided for further analysis by an image recognition module which corresponds to the identified class. The determination of such an image feature may require fewer processing resources than the further analysis to recognize the object(s) depicted, and thus less power is consumed. In this way, unnecessary consumption of resources to analyze images which do not include any object(s) may be avoided. Furthermore, by providing an image for further analysis by an image recognition module which corresponds to the identified class, the image recognition modules can be made more efficient, and further reductions in the processing requirements of the system can be provided.

<FIG> is a block diagram of an environment <NUM> in which an example mobile device <NUM> analyzes images and presents content related to one or more objects depicted in the images. The mobile device <NUM> may be a smart phone, tablet computer, laptop computer, wearable device, or other appropriate type of mobile device. Although a mobile device <NUM> is illustrated in <FIG> and described herein, the components of the mobile device <NUM> may be included in other types of electronic devices (e.g., desktop computers) and the techniques performed by the mobile device <NUM> may be performed by other electronic devices.

The mobile device <NUM> includes a camera <NUM> and a camera application <NUM>. The camera <NUM> capture still images (e.g., digital photos) and videos. The camera application <NUM> may be a native application developed for use on a particular platform or a particular device. The camera application <NUM> enables a user to control the camera <NUM> and to view images and video captured by the camera <NUM>. The camera <NUM> can include an image sensor that is configured to output image data (e.g., image pixel data) captured by the image sensor. The pixel data for each pixel of the image can specify one or more visual characteristics of the pixel, e.g., the color of the pixel.

The camera application <NUM> can also perform image selection techniques and/or image analysis techniques on captured images to provide content related to objects recognized in the images. In some implementations, the camera application <NUM> captures an image and analyzes the image in response to a user selecting an icon. For example, the user can point the camera <NUM> at an object and select the icon to capture an image of the object and receive content related to the object.

In some implementations, the camera application <NUM> performs the image analysis process automatically, e.g., in an always-on state, on a stream of images being captured by the camera <NUM>. For example, the camera application <NUM> can select images from a video stream and analyze the selected images to recognize objects depicted in the images and provide content related to the recognized objects. In a particular example, the camera application <NUM> can select the images from a sequence of images captured by the camera <NUM> while the user is pointing the camera <NUM> at an object. In another example, the camera application <NUM> can use the camera <NUM> to capture images of a scene that is visible in a viewfinder of the camera <NUM> (or the mobile device <NUM>).

Although the following description is in terms of the camera application <NUM> performing the image selection and image analysis techniques on the images, the techniques (or a portion thereof) can be performed by another application, (e.g., a camera-first application that can access and/or control the camera to present content related to images captured by the camera), hardware circuitry of the mobile device, a controller, or another appropriate hardware and/or software component.

The camera application <NUM> can use one or more low power consuming stages to select images that are analyzed for object recognition. The one or more stages can ensure that the images that are analyzed in later stages are of sufficient quality and depict one or more objects that may be of interest to a user. This allows the mobile device <NUM> to not waste computational resources and battery power on images for which objects may not be recognized or for which content may not be identified.

The camera application <NUM> includes an image selector <NUM> that selects an image from an image stream for further processing based on data indicative of the movement of the camera <NUM> ("movement data") at the time at which the image was captured. The movement data at the time at which the image was captured can include movement data for the camera <NUM> for a time window that begins a specified period of time before the image was captured to a specified period of time after the image was captured. For example, if the camera <NUM> is moving while capturing an image, the image is more likely to be low quality (e.g., blurry) than if the camera <NUM> is still. By using movement data for an image instead of visual characteristics of the images, the selection can be performed more quickly and by using fewer computing resources. For example, the camera application <NUM> can select an image from multiple images without detecting and evaluating any visual characteristics of the images and just using movement data for the images. As described below, the camera application <NUM> can also use visual characteristics of images in combination with the movement data for the images to select an image for further processing.

The camera application <NUM> can obtain or receive the movement data for an image from an inertial measurement unit (IMU) <NUM> of the mobile device <NUM> or another type of environment sensor. For example, the camera application <NUM> can obtain the movement data for an image when the camera application <NUM> causes the camera <NUM> to capture the image. An IMU is an electronic device that may include one or more accelerometers, one or more gyroscopes, and/or one or more magnetometers. The IMU data can be in the form of a game rotation vector. In some implementations, the mobile device <NUM> may include separate accelerometers, gyroscopes, and/or magnetometers. The movement data for an image can include inertial acceleration measurements received from the IMU <NUM>, and/or orientation data received from the IMU <NUM> or a separate gyroscope.

In some implementations, the movement data for an image only includes orientation data received from a gyroscope. For example, the orientation data from the gyroscope can be used to determine orientation changes that occurred at the time at which the image was captured.

The camera application <NUM> can use the movement data for an image (e.g., the data received from the IMU <NUM>) to determine a quality score for the image. The quality score for an image can represent an expected image quality of the image. The quality score for an image can be based on (e.g., inversely proportional to) a measure of rotational motion of the camera <NUM> at the time the image was captured, which can be determined using the movement data. The quality score for an image can be based on orientation changes of the camera <NUM> at the time at which the image was captured. For example, orientation data that specifies the orientation of the mobile device <NUM> before, during, and after the image was captured can be used to determine whether the orientation of the camera <NUM> was changing at the time the image was captured.

The camera application <NUM> can select, from multiple images, one or more images for further analysis based on the quality scores for the images. For example, the camera application <NUM> may select the image having the highest quality score (e.g., the least amount of rotational motion or least amount of movement) for further analysis. The camera application <NUM> can also use the recency of the images to determine which to use for further processing. The camera application <NUM> can then send image data for the selected image for further processing, e.g., by a coarse classifier <NUM> described below. The camera application <NUM> can then capture additional images and select, from the additional images, another image for further analysis. In this example, the coarse classifier <NUM> can request another image or notify the camera application <NUM> that it is ready for another image. In response, the camera application <NUM> can send the selected image to the coarse classifier <NUM>.

The camera application <NUM> can continuously analyze images and store the best image while waiting for the coarse classifier <NUM> to finish processing a previous image or otherwise become ready to analyze another image. For example, the camera application <NUM> may store image data for a highest quality image that has an associated highest quality score among a set of images that have been captured since image data for a previous image was sent to the coarse classifier <NUM>. While waiting, the camera application <NUM> may receive another image. The camera application <NUM> can determine a quality score (e.g., a measure of rotational motion) for the newly received image. The camera application <NUM> can compare the quality score for the stored best image to the quality score for the newly received image. If the quality score for the newly received image is greater than the quality score for the stored best image, the camera application <NUM> can replace the stored image data for the highest quality image with the image data for the newly received image. In this way, the camera application <NUM> is storing the image data for the highest quality image received while waiting until time for the subsequent analysis to be performed. By only storing the image data for the highest quality image, the amount of memory used to store image data is reduced relative to storing image data for multiple images.

The camera application <NUM> can use buffers to store the image data. For example, the camera application <NUM> can store image data for a newly received image in an image buffer. The camera application <NUM> can also store image data for the highest quality image in a selected image buffer. If a newly received image has a higher quality score than the image for which image data is stored in the selected image buffer, the camera application <NUM> can replace the image data in the selected image buffer with the image data for the newly received image. In another example, if a newly received image has a higher quality score than the image for which image data is stored in the selected image buffer, the camera application <NUM> can update the selected frame pointer to point to the image buffer. The camera application <NUM> can provide the image data stored in the selected image buffer to the coarse classifier <NUM>, e.g., after a predetermined time interval or when the coarse classifier <NUM> is ready to process another image.

In some implementations, the camera application <NUM> selects, from multiple images, an image for further analysis based on data indicative of the movement of the camera <NUM> (e.g., rotational motion) at the time each image was captured in combination with a time at which each image was captured. For example, a more recently received image having a slightly lower quality score (e.g., within a threshold amount) than the quality score for an older image may be preferred and therefore selected.

A three-tier strategy may be used to select an image from multiple images. In this example, if the image has a quality score that is less than a first threshold, the image may not be used at all as the quality of the image not be sufficient to detect or recognize objects depicted in the image. By not sending such low quality images to the subsequent stages, computing power and electrical power that would otherwise be used is avoided. If one or more images have a quality score that is greater than a second threshold (which is higher than the first threshold), the camera application <NUM> can select the most recent image that has a quality score that is greater than the second threshold. If no image of the multiple images has a quality score that is greater than the second threshold, but one or more images have a quality score that is between the two thresholds, the camera application <NUM> can select one of the one or more images that has the highest quality score.

In some implementations, the camera application <NUM> can select an image for further analysis based on data indicative of the movement of the camera <NUM> at the time at which the image was captured and visual characteristics of the image. Using data indicative of the movement of the camera <NUM> can reduce the number of visual characteristics and/or the types of visual characteristics needed to determine the quality of an image relative to using visual characteristics alone.

In some implementations, the camera application <NUM> can select an image for further analysis based on light features of the images. The light features are those that are computed based on visual input, but in a less computationally intensive manner than examining the image using vision analysis. For example, the quality score for an image may be based on a combination of the data indicative of the movement of the camera <NUM> and light features of the image. The light features may be specified in metadata for the image. The light features can include brightness information, focus information, histograms that indicate characteristics including high level contrast within the image, and/or movement of objects within the image (e.g., based on the location of an object within a previous image and the location of the same object in this current image). Similar to light data, other features of the image that are already specified in metadata can be used as this data does not require further image analysis.

In some implementations, the image selection process is performed in hardware, e.g., a hardware circuit or controller separate from the processor of the mobile device <NUM>. In this way, the mobile device's processor does not have to process any data or execute any instructions for the image selection process, resulting in even less demand being placed on the processor. For example, the hardware circuit may receive movement data from the IMU, gyroscope, or other sensor and use the data to determine whether the image has sufficient quality (e.g., less than a threshold amount of movement or rotational jitter) to be sent to the coarse classifier <NUM>. If an image is detected by the hardware circuit as having sufficient quality, the hardware circuit can wake the processor and cause the processor to perform coarse classification on the image.

The coarse classifier <NUM>, which can include multiple coarse classifiers, detects the presence of one or more classes of objects depicted in an image. The coarse classifier <NUM> can detect the presence of a class of objects based on whether or not the image includes one or more features that are indicative of the class of objects. The coarse classifier <NUM> can include a light-weight model to perform a low computational analysis to detect the presence of objects within its class(es) of objects. For example, the coarse classifier <NUM> can detect, for each class of objects, a limited set of visual features depicted in the image to determine whether the image depicts an object that falls within the class of objects. In a particular example, the coarse classifier <NUM> can detect whether an image depicts an object that is classified in one or more of the following classes: text, barcode, landmark, media object (e.g., album cover, movie poster, etc.), or artwork object (e.g., painting, sculpture, etc.). For barcodes, the coarse classifier <NUM> can determine whether the image includes parallel lines with different widths.

In some implementations, the coarse classifier <NUM> uses a trained machine learning model (e.g., a convolutional neural network) to classify images based on visual features of the images. For example, the machine learning model can be trained using labeled images that are labeled with their respective class(es). The machine learning model can be trained to classify images into zero or more of a particular set of classes of objects. The machine learning model can receive, as inputs, data related to the visual features of an image and output a classification into zero or more of the classes of objects in the particular set of classes of objects.

The coarse classifier <NUM> can output data specifying whether a class of object has been detected in the image. The coarse classifier <NUM> can also output a confidence value that indicates the confidence that the presence of a class of object has been detected in the image and/or a confidence value that indicates the confidence that an actual object, e.g., the Eiffel Tower, is depicted in the image.

In some implementations, the camera application <NUM> includes multiple coarse classifiers. In this example, each coarse classifier can detect the presence of a particular class of objects and output a confidence score for the particular class. Each of the multiple coarse classifiers can detect the presence of a different class of objects than each other coarse classifier.

In some implementations, the coarse classifier <NUM> is a composite coarse classifier <NUM> that can determine the confidence score of multiple coarse classifiers, e.g., simultaneously. For example, a composite coarse classifier can determine, for an image, the confidence score for multiple different classes of objects simultaneously. The composite coarse classifier can include a core portion that's common to each classification and several modules that each determine a per-class probability for the classes of objects. This can reduce the overall computation performed by the camera application <NUM> to detect the presence of one or more classes of objects depicted in the image, for example, by reducing redundant computations.

The coarse classifier <NUM> can also output annotations that specify data about the objects detected in the image and provide the data to one or more vision analyzers <NUM>. For example, if the coarse classifier <NUM> detects the presence of text in the image, the coarse classifier <NUM> can provide annotations that specify where the text is located and the orientation of the text. The coarse classifier <NUM> can also provide annotations that specify types of text detected, e.g., phone numbers, addresses, etc. Such data can save on computational costs and time of performing OCR to recognize the actual text later by a text analyzer <NUM>. Similarly, if the coarse classifier <NUM> detects a barcode in the image, the coarse classifier <NUM> can provide annotations that specify the location and type of the barcode to save on computational costs and time of reading/decoding the barcode by a barcode analyzer <NUM>.

As described above, an image can be selected for coarse classification at an appropriate time, such as when coarse classification has been completed for a previous image. In some implementations, the camera application <NUM> may perform coarse classification on an image periodically based on a processing rate. For example, the camera application <NUM> can select an image (or retrieve a stored best image from a selected frame buffer) every second (or some other time period) and perform coarse classification on the image.

In some implementations, the camera application <NUM> can dynamically and automatically adjust the processing rate based on whether the presence of one or more classes of objects has been detected by the coarse classifier <NUM> in one or more previously analyzed images. For example, if the camera <NUM> is not capturing images of objects that are classified within one of the classes of objects for which the coarse classifier <NUM> is configured to detect, the camera application <NUM> may reduce the rate at which images are being analyzed by the coarse classifier <NUM>. In this example, if the presence of a class of objects was not detected in a previous image (or a threshold number of previous images), the camera application <NUM> may reduce the processing rate to increase the period of time between each coarse classification. This can reduce the amount of computing power and electrical power consumed in performing the coarse classification when images that do not depict relevant objects are being captured. For example, this processing rate adjustment can result in fewer CPU cycles used to classify images.

Similarly, the camera application <NUM> can increase the processing rate to reduce the period of time between each coarse classification if the presence of at least one class of objects has been detected in a previous image (or at least a threshold number of previous images). In this way, when the user is using the camera <NUM> to capture images of relevant objects, the camera application <NUM> can identify and present useful and relevant content to the user. This increased processing rate may be temporary. For example, the increase in processing rate may be reduced to a normal processing rate (or a reduced processing rate) after a specified period of time or in response to not detecting the presence of a class of objects in an image or at least a threshold number of images.

The camera application <NUM> can adjust the processing rate for coarse classification based on the amount of time since the user has opened the camera application <NUM> or selected a mode of the camera application in which image analysis and content presentation are performed. For example, a user may be more active with capturing images and viewing content related to objects depicted in the image immediately after opening the camera application <NUM> or entering into the analysis and content presentation mode. In this example, the camera application <NUM> may use a faster processing rate when the camera application <NUM> is opened or the analysis and content presentation mode is entered, then use a slower processing rate after a specified amount of time has lapsed or after the coarse classifier <NUM> has evaluated at least a threshold number of images without detecting an object in at least one of the classes of objects.

The camera application <NUM> can adjust the processing rate for coarse classification based on user interactions with the camera application <NUM>. For example, if the user has interacted with a result or other content provided based on an analyzed image, the camera application <NUM> may increase the processing rate for coarse classification or maintain the initial fast processing rate. The camera application <NUM> can adjust the processing rate based on a frequency of user interactions in a current session and/or a frequency of user interactions in multiple user sessions including historical user sessions of the user with the camera application <NUM>. A user session may be defined by the opening and closing of the camera application <NUM> and/or the beginning and ending of the image analysis and content presentation mode.

In some implementations, the camera application <NUM> or the mobile device <NUM> itself includes a power management system that determines and adjusts the processing rate for coarse classification based on whether previous images have been detected as depicting an object of one or more classes. When an image is not being classified by the coarse classifiers, the processor of the mobile device <NUM> (or a controller used to analyze images) can sleep (e.g., go into a sleep mode and not execute instructions) to consume less power.

If the coarse classifier <NUM> detects the presence of a class of objects in an image, the coarse classifier <NUM> can provide the image data for the image to an appropriate object recognizer <NUM>. The object recognizers <NUM> can include a text recognizer <NUM> that recognizes text (e.g., recognizes characters, words, etc.) in images, a barcode recognizer <NUM> that recognizes (e.g., decode) barcodes (e.g., including QR codes) in images, and a landmarks recognizer <NUM> that recognizes landmarks (e.g., identifies actual landmarks) in images. The camera application <NUM> can include analyzers for other types of objects in addition to, or in place of, the analyzers <NUM>, <NUM>, and <NUM>. For example, the camera application <NUM> may include a media cover (e.g., album cover) analyzer, an artwork analyzer, and/or other appropriate analyzers. In some implementations, the camera application <NUM> includes a single object recognizer that recognizes multiple different classes of objects, e.g., text, barcodes, landmarks, media covers, artwork, etc..

If the coarse classifier <NUM> detects the presence of text in an image, the coarse classifier <NUM> can provide the image data for the image to the text recognizer <NUM>. If the coarse classifier <NUM> detects the presence of a barcode in an image, the coarse classifier <NUM> can provide the image data for the image to the barcode recognizer <NUM>. If the coarse classifier <NUM> detects the presence of a landmark in an image, the coarse classifier <NUM> can provide the image data for the image to the landmark recognizer <NUM>.

In some implementations, the coarse classifier <NUM> provides the image data for the image to each object recognizer <NUM> for each class of object for which the image has a confidence value that satisfies (e.g., meets or exceeds) a threshold. For example, the coarse classifier <NUM> can determine a confidence value that the image depicts text and, if the confidence value satisfies the threshold, the coarse classifier <NUM> can provide the image to the text recognizer <NUM>.

Each object recognizer <NUM> can perform more detailed image analysis than the coarse classifier <NUM> to recognize objects depicted in images received from the coarse classifier <NUM>. For example, the object recognizer <NUM> can use edge detection, pattern recognition, and other computer vision techniques to recognize objects depicted in the images and extract information from the images.

The text recognizer <NUM> can recognize text depicted in images using optical character recognition (OCR). The barcode recognizer <NUM> can read/decode barcodes depicted in images and obtain information about an object (e.g., product) represented by the barcode. The landmarks recognizer <NUM> can use image data of landmarks and pattern recognition to recognize landmarks depicted in images. In some implementations, the camera application <NUM> stores an object index <NUM> that includes image data and barcode data for particular objects for use in recognizing objects and reading barcodes depicted in images. The image data for an object can include data specifying visual features of the objects that can be used in computer vision analysis to recognize images of the objects.

The camera application <NUM> can perform inter-frame processing for at least some classes of objects. For example, the camera application <NUM> can perform inter-frame processing <NUM> on images in which the text recognizer <NUM> has recognized text and the camera application <NUM> can perform inter-frame processing <NUM> on images in which the landmark recognizer <NUM> has recognized a landmark. In general, inter-frame processing leverages data regarding objects recognize in previous images in detecting objects in a current image.

The inter-frame processing <NUM> for text can establish correlation between a same line (or same portion) of text in two images to determine whether the same text is being detected. The camera application <NUM> can then keep the best version of the text, e.g., the text for which the text recognizer <NUM> has determined to be of a higher quality, which can improve the accuracy of text recognition. To determine whether two portions of text are the same, the camera application <NUM> can use the inter-frame processing <NUM> to evaluate a distance between the two portions of text in the two (or more) images and an edit distance between text in the two images. The camera application <NUM> can use inter-frame processing <NUM> to determine the distance directly based on the location of the portions of text in two images, by tracking the text using optical tracking, and/or predict where the text will be based on movement of the mobile device <NUM> between the capture of the two images. The edit distance may indicate a number or percentage of characters that is different between the portions of text in the two images. If the distance and the edit distance are both less than a threshold, the two or more images can be correlated and the image having the highest confidence value and/or the highest quality score can be retained for later use.

In some implementations, the portions of text may be retained from multiple images. For example, the camera application <NUM> can use inter-frame processing <NUM> to correlate text on a line-by-line basis, a character-by-character basis, or a word-by-word basis. The camera application <NUM> can identify which of the correlated lines of text has the highest confidence and/or highest quality and retain that the portion of the image having the highest confidence and/or highest quality line of text for the text analyzer <NUM>. Similarly, the camera application <NUM> can determine which of the correlated words (or characters) has the highest confidence and/or highest quality and retain that the portion of the image having the highest confidence and/or highest quality word (or character). Thus, when multiple images have the same text, e.g., due to the camera pointing at the same document or other text source for a period of time, the text recognize in multiple image portions may be provided to the result identifier <NUM> to provide the highest quality version of each portion of the text.

The inter-frame processing <NUM> for landmarks (and other types of objects) can evaluate a confidence value (determined by the landmark recognizer <NUM>) that indicates the confidence that a particular landmark (or other object) has been detected in multiple images in determining whether an image depicts the particular landmark (or object). In some implementations, the inter-frame processing <NUM> uses multiple thresholds for determining how many images in a sequence have to be identified as depicting a particular landmark before classifying an image as depicting the landmark for the result identifier <NUM>. For example, if the confidence values that indicate that the images depict a landmark are greater than a first threshold, the camera application <NUM> can determine that a first number of images must be classified as depicting the particular landmark before classifying the images as depicting the particular landmark. If the confidence values are less than the first threshold but greater than a second threshold, the camera application <NUM> may require a second number of images to have been classified as depicting the particular landmark in order to determine that the images depict the particular landmark, where the second number is higher than the first number.

The result identifier <NUM> can identify content for presentation at the mobile device <NUM> (e.g., within the camera application <NUM>). For example, if an image includes a barcode, the result identifier <NUM> can identify the object represented by the barcode and present content (e.g., images, title, etc.) related to the object. If the image includes a landmark, the result identifier <NUM> can identify and present, in a user interface <NUM>, content related to the landmark, e.g., photos of the landmark, a map to the landmark etc..

The result identifier <NUM> can identity content for a recognized object in a content data store <NUM>. The content data store <NUM> can include content for each of a set of objects. The content in the content data store <NUM> can be loaded onto the mobile device <NUM>, e.g., by the camera application <NUM>. The content can include text, images, videos, and/or other appropriate content that can be presented by a mobile device <NUM>.

The identified content for an object can be presented in the user interface <NUM> with the image. For example, the content may be presented in a viewfinder in which the image is depicted within the camera application <NUM>. In this example, the content may be presented in an overlay over the real-time view of the camera in which the object, text, or barcode was detected.

If the coarse classifier <NUM> detects more than one class of objects depicted in an image, the result identifier <NUM> can identify results for each class and present at least one result for each class. For example, the camera application <NUM> can present a results page that includes a result that corresponds to each class of object detected in the image.

In some implementations, the camera application <NUM> can present icons with which a user can request content related to a detected object. For example, if the image includes a phone number, the camera application <NUM> may present an icon that, if interacted with (e.g., selected) by the user, causes the mobile device <NUM> to initiate a call to the phone number. If a barcode is detected, the camera application <NUM> can present icons to launch a shopping application to purchase the product represented by the barcode, an icon to initiate a search for the product using a search application or a web browser, and/or an icon to share the product, e.g., using a social networking application. If a landmark is detected, the camera application <NUM> can present an icon to launch a map application to present a map to the landmark, an icon to initiate a search for the landmark, and/or an icon to view images of the landmark. The icons can be presented with the image, e.g., within the viewfinder of the camera application <NUM>.

In some implementations, the camera application <NUM> can highlight, in the user interface <NUM> of the camera application <NUM>, an object being detected so that a user can see what object is being detected. For example, while the user is pointing the camera <NUM> at an object, the coarse classifier <NUM> can detect the presence of a class of objects and the image can be sent to one or more of the object recognizers <NUM> for analysis. While the image is being analyzed by the one or more object recognizers <NUM>, the camera application <NUM> can highlight the object being recognized.

In some implementations, the camera application <NUM> customizes the object index <NUM> and or the content in the content data store <NUM> for a user. For example, if the camera application <NUM> receives data indicating that the user is going to travel to a particular location, the camera application <NUM> can update the object index <NUM> to include image data and/or barcode data for objects, landmarks, etc. located at the particular location. The camera application <NUM> can also update the content data store <NUM> to include content related to the objects, landmarks, etc. located at the particular location.

In some implementations, some elements of the image analysis and result identification are located at a visual analysis server <NUM>. For example, the visual analysis server <NUM> may include a visual analyzer <NUM> that recognizes objects depicted in images received from the camera application <NUM> over a data communication network <NUM>, e.g., a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet. The visual analysis server <NUM> can also include a result processor <NUM> that identifies results or other content related to objects recognized by the vision analyzer <NUM> and provides the results or content to the mobile device <NUM>. In this example, the image selector <NUM> and the coarse classifier <NUM> can be implemented on the mobile device <NUM>, e.g., as part of the camera application <NUM>.

The camera application <NUM> can provide portions of an image for which the coarse classifier <NUM> detected the presence of a class of objects or feature data for the image to the visual analysis server <NUM>. For example, if a coarse classifier <NUM> detects the presence of a landmark in an image, the camera application <NUM> can provide the portion of the image that includes the landmark to the visual analysis server <NUM>, without providing the entire image.

Similar to the camera application implementation, if an image is not selected by the image selector <NUM> (e.g., for having a quality score that does not meet a threshold), the camera application <NUM> can determine to not send the image (or a portion of the image) to the visual analysis server <NUM>. Similarly, if the coarse classifier <NUM> does not detect the presence of a class of object in the image, the camera application <NUM> may determine to not send the image (or a portion of the image) to the visual analysis server <NUM>.

Although, the stages of image selection and image analysis are illustrated as being performed in order, some stages can be performed in parallel. For example, multiple images can be classified by coarse classifier(s) at the same time. Similarly, an image can be analyzed by multiple coarse classifiers in parallel.

In some implementations, the image selector <NUM> is implemented as a controller or as part of a controller of the mobile device <NUM>. In this example, the coarse classifier <NUM> and/or the object recognizers <NUM> can be implemented on an image processing apparatus, e.g., of the mobile device <NUM> or the visual analysis server <NUM>.

<FIG> is a flow diagram that illustrates an example process <NUM> for analyzing an image and providing content related to one or more objects depicted in the image. Operations of the process <NUM> can be implemented, for example, by a system that includes one or more data processing apparatus, such as mobile device <NUM> of <FIG> The process <NUM> can also be implemented by instructions stored on a computer storage medium, where execution of the instructions by a system that includes a data processing apparatus cause the data processing apparatus to perform the operations of the process <NUM>.

The system accesses, for images captured by a mobile device camera, data indicative of movement of the camera at a time at which the camera captured the image (<NUM>). For example, the system can receive a stream of images (e.g., a video stream). The system can also obtain data indicative of the movement of the device on which the camera is installed for each image. For example, the movement data for each image can be received from one or more environment sensors such as a gyroscope, an accelerometer, or an IMU. For each image, the movement data can indicate the movement of the device at the time the image was captured. For example, the system can request the movement data from the sensor at the time that the camera is capturing the image.

In another example, the system can provide, to the environment sensor, a time window that includes the time at which the image was captured. In response, the environment sensor can provide movement data detected by the environment sensor during the time window. The system can then determine how the camera was moving at the time the image was captured based on the data.

The system selects, from the images, a particular image for analysis based on the movement data for the images (<NUM>). The system can select the image for which the movement of the device was the least or for which the rotational motion was the least. As described above, the image can also be selected based on a time at which the image was capture relative to the time at which other images were captured and/or based on light features of the images. The selection can be made independent of visual characteristics of the image, or in combination with visual characteristics of the image.

The system analyzes the particular image using one or more coarse classifiers to detect the presence of one or more classes of objects depicted in the particular image (<NUM>). Each coarse classifier can be configured to detect presence of a respective class of object, e.g., text, landmark, artwork, media cover, barcode, etc. Each coarse classifier can output data specifying whether the particular image depicts its class of object and a confidence value indicating the confidence that the coarse classifier has in its determination. Each coarse classifier can also output annotations that include data describing characteristics of the detected object, such as location, orientation, etc..

In response to detecting presence of at least one class of object in the image, the system analyzes the image to recognize one or more objects depicted in the image (<NUM>). For example, the system may use one or more computer vision techniques to recognize objects depicted in the image. The techniques used can be based on the class(es) of objects detected in the image by the coarse classifier. For example, if the image is classified as having a barcode, the system can use a barcode recognizer to read the barcode and identify the product referenced by the barcode.

The system presents content related to the one or more identified objects (<NUM>). For example, the system can present the content in one or more overlays over the image or on a results page. The system can present icons that, when interacted with by a user, cause the system to present content related to the objects recognized in the image. For example, the icons can include an icon to initiate a search for a recognized object, display a map to a recognized landmark, present other images of a recognized object, etc..

<FIG> is a flow diagram that illustrates an example process <NUM> for selecting an image for analysis. Operations of the process <NUM> can be implemented, for example, by a system that includes one or more data processing apparatus, such as mobile device <NUM> of <FIG>. The process <NUM> can also be implemented by instructions stored on a computer storage medium, where execution of the instructions by a system that includes a data processing apparatus cause the data processing apparatus to perform the operations of the process <NUM>.

The system receives movement data for an image captured by a mobile device camera (<NUM>). As described above, the movement data can be indicative of the movement of the mobile device camera at the time at which the image was captured by the mobile device camera. The mobile device camera can capture images in a sequence, e.g., as a video stream. In some implementations, the mobile device camera captures images at a rate in the range of <NUM>-<NUM> frames per second.

The system determines an expected quality of the image based on the movement data (<NUM>). For example, the expected quality of the image can be based on an amount of movement of the mobile device camera at the time at which the image was captured. In this example, more movement can result in a lower quality score as the movement may reduce the quality of the image due to blurring.

The system determines whether the expected quality of the image is greater than the expected quality of a highest quality image for which image data is being stored (<NUM>). For example, the system can store image data for a single image only and this single image may be the highest quality image (based on expected quality) of images captured since a previous image was sent to a coarse classifier. Each time a new image is received, the system can determine whether the newly received image has a higher expected quality than the highest quality image for which image data is currently being stored.

If the newly received image does not have a higher expected quality than the previously identified highest quality image, the system continues storing the previously stored image data for the highest quality image and waits for another image to be received. If the newly received image has a higher expected quality than the highest quality image, the system replaces the image data for the highest quality image with image data for the newly received image (<NUM>).

In some implementations, the system sets a replacement threshold for the expected image quality based on the image data stored for the highest quality image. In this example, the system replaces the image data for the highest quality image with the image data for the newly received data if the expected image quality of the newly received image exceeds the replacement threshold. The system can reduce the replacement threshold over time.

The system determines whether to provide the image data for the highest quality image to a coarse classifier (<NUM>). In some implementations, the system provides the image data to the coarse classifier based on a predetermined time period. In some implementations, the system provides the image data to the coarse classifier in response to receiving a request from the coarse classifier.

If the system determines to not provide the image data to the coarse classifier, the system continues receiving images and determining whether to replace the image data for the highest quality image until determining to provide the image data to the coarse classifier. In this way, the system only stores image data for the highest quality image.

If the system determines to provide the image data to the coarse classifier, the system provides the image data to the coarse classifier for classification (<NUM>).

<FIG> is a flow diagram that illustrates an example process <NUM> for adjusting a processing rate for analyzing images. Operations of the process <NUM> can be implemented, for example, by a system that includes one or more data processing apparatus, such as mobile device <NUM> of <FIG>. The process <NUM> can also be implemented by instructions stored on a computer storage medium, where execution of the instructions by a system that includes a data processing apparatus cause the data processing apparatus to perform the operations of the process <NUM>.

The system receives image data for an image (<NUM>). The image data can include image pixel data for the image, including color values for each pixel of the image.

The system determines whether the image contains an object classified in one or more classes (<NUM>). For example, the system can use a coarse classifier to determine whether the image contains an object classified in the one or more classes, as described above.

If the system determines that the image does not contain an object classified in the one or more classes, the system reduces a processing rate for analyzing the images to determine whether the images contain an object of the one or more classes (<NUM>). For example, the system can use a coarse classifier to analyze the images periodically based on the processing rate. If the system determines that one or more images do not contain an object in one of the classes, the system can decrease the processing rate to reduce the frequency at which the images are analyzed. This allows the system to reduce computer resource usage and power consumption when images that do not contain an object of interest are being captured. The system can wait until a predetermined number of images (e.g., five, ten, or another appropriate number) do not contain an object on one of the classes before reducing the processing rate.

If the system determines that the image does contain an object classified in at least one of the classes, the system increases the processing rate (<NUM>). For example, the system can increase the processing rate to an active processing rate that the system uses when the images contain objects of interest. If the processing rate was already at the active processing rate, the system can leave the processing rate unchanged.

The system sends the image that contain an object of at least one of the classes for further analysis (<NUM>). For example, the system can send the image to an object recognizer that recognizes objects of the at least one class.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

Claim 1:
A system, comprising:
a mobile device camera (<NUM>) configured to capture images;
one or more environment sensors configured to detect movement of the camera (<NUM>);
a data processing apparatus; and
a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
accessing, for each of a plurality of images captured by the mobile device camera (<NUM>), data indicative of movement of the camera (<NUM>) at a time at which the camera (<NUM>) captured the image;
selecting, from the plurality of images, a particular image for analysis based on the data indicative of the movement of the camera (<NUM>) for each image;
analyzing the particular image using a composite coarse classifier (<NUM>) to detect presence of multiple different classes of objects depicted in the particular image, wherein analyzing the particular image using the composite coarse classifier (<NUM>) comprises initiating analysis by the composite coarse classifier (<NUM>) periodically based on a processing rate, the processing rate adjusted based on whether presence of one or more classes of objects has been detected in one or more images analyzed by the coarse classifier over a previous time period;
determining, using the composite coarse classifier (<NUM>), a confidence score for the multiple different classes of objects simultaneously;
analyzing the particular image to recognize one or more objects depicted in the particular image in response to determining the presence of the multiple different classes of objects in the particular image; and
presenting content related to the one or more recognized objects.