Data retrieval system

A system for selecting different aspects of data objects to be matched with similar aspects of other data objects. A user inputs a search data object and a value. A neural network computes features for the search object at multiple layers that correspond to different aspects of the object. A descriptor is generated for the search object from features output at a layer position of the neural network determined from the value. The descriptor is compared to corresponding descriptors for objects in a collection to select objects that include aspects similar to an aspect of the search object. The user can change the value to view different objects that include aspects similar to other aspects of the search object. Thus, the user can explore different aspects of an object to find objects that include aspects similar to the aspect of the object that the user is interested in.

BACKGROUND

Data retrieval systems (e.g., image retrieval systems) use an input data object (e.g., an image) provided as an example as a query to retrieve similar data objects from a large collection of data objects. Data retrieval systems may be trained to find particular aspects of data objects; for example, an image retrieval system may be trained to find images with similar lighting conditions. However, retrieval queries may be ambiguous, since the same data object could be used in queries with different expectations on what should be considered similar. For example, in an image retrieval system, an outdoor image that shows a cyclist on a road could be submitted as a query for images that contain cyclists, but could also be submitted as a query for images that contain other content irrespective of whether the images might contain a cyclist.

SUMMARY

Embodiments of a data retrieval system are described that allow users to select different aspects of search data objects to be matched with similar aspects of other data objects in a collection, thus resolving the ambiguity of retrieval queries in conventional data retrieval systems. The data retrieval system includes a neural network (e.g., a deep neural network) that characterizes data objects and computes features for input data objects that correspond to different aspects of the data objects at multiple layers in the neural network. The data retrieval system also includes an interface that allows a user to input a search data object and to input a value, for example via a slider bar user interface element, corresponding to a layer position in the neural network. A descriptor is generated for the search data object from features output at a layer position of the neural network determined from the input value. In some examples, a layer position refers to a particular layer of the neural network, in which case a descriptor is generated from features output by that layer. In some examples, a layer position refers to a position between two layers, in which case a descriptor is generated from a combination of features output by the two closest layers. The generated descriptor is compared to corresponding neural network layer descriptors for data objects in a collection of data objects to select one or more of the data objects in the collection that include aspects similar to an aspect of the search data object represented in the descriptor. Indications of the selected data object(s) may be displayed to the user via the interface. The user can change the value, for example by moving the slider bar, to view different sets of data objects that include aspects similar to other aspects of the search data object. Embodiments of the data retrieval system thus allow the user to explore different aspects of the search data object (e.g., a query image) to find a set of data objects that include aspects similar to the aspect of the search data object that the user is interested in.

In some embodiments, the data retrieval system is used to index data objects, in addition to uses in retrieval operations in which different sets of data objects (e.g., images or parts of images) are retrieved from a collection of data objects based on features computed for a search data object at different layer positions of a neural network. In an indexing operation, an input data object and descriptors generated for the input data object based on features computed at one or more layer positions of a neural network may be added to a collection of data objects. The collection of data objects can then be used in retrieval operations.

The data objects are, for example, digital images (e.g., still images or video frames captured by conventional digital camera systems, images captured in remote sensing systems including but not limited to Light Detection and Ranging (LIDAR) systems, etc.), or portions of images. In some embodiments, the data retrieval system can index and retrieve other types of data objects, for example audio or sound files or clips, or various types of textual and/or numerical data files.

DETAILED DESCRIPTION

Embodiments of a data retrieval system are described that allow users to select different aspects of search data objects to be matched with similar aspects of other data objects in a collection, thus resolving the ambiguity of retrieval queries in conventional data retrieval systems. The data objects are, for example, digital images (e.g., still images or video frames, LIDAR images, etc.), or portions of images. In some embodiments, the data retrieval system can index and retrieve other types of data objects, for example audio or sound files or clips, or various types of textual and/or numerical data files.

Embodiments of the data retrieval system include a neural network (e.g., a deep neural network) that characterizes data objects and computes features for search data objects that correspond to different aspects of the data objects at multiple layers in the neural network. Note that the neural network does not have to be trained explicitly to compute the features; the neural network computes the features as a side effect of training the neural network with a sufficiently large training set. Embodiments of the data retrieval system also include a retrieval engine that selects data objects from a collection based on descriptors generated from features computed at different layer positions of the neural network.

Embodiments of the data retrieval system also include a user interface that obtains search data objects from users, and that obtains input values, for example via a slider bar user interface element, corresponding to different layer positions in the neural network. Descriptors may be generated for search data objects from features output at layer positions of the neural network determined from the input values. In some examples, a layer position refers to a particular layer of the neural network, in which case a descriptor is generated from features output by that layer. In some examples, a layer position refers to a position between two layers, in which case a descriptor is generated from a combination of features output by the two closest layers. The generated descriptor is then compared to corresponding neural network layer descriptors for data objects in a collection of data objects to select one or more of the data objects in the collection that include aspects similar to the aspect(s) of the search data object represented in the descriptor. Indications of the selected data object(s) may be displayed to the user via the interface. The user may change the input value, for example by moving the slider bar; in response, a new layer position may be determined, a descriptor may be generated for the search data object from features output by one or more layers at the new layer position, and a new set of data objects may be selected from the collection based on the new descriptor and displayed to the user.

Embodiments of the data retrieval system thus allow a user to explore different aspects of the search data object (e.g., a query image) to find a set of data objects that include aspects similar to the aspect of the search data object that the user is interested in. As an example, the user may submit an outdoor image that shows a cyclist on a road as a query image, and by moving the slider bar may view sets of images that contain cyclists, that have similar backgrounds or lighting conditions, or that have other aspects similar to the query image irrespective of whether the images contain a cyclist. Note that the user need not be aware that the position of the slider bar corresponds to layer positions of a neural network; to the user, different positions on the slider bar return different sets of images that correspond to different aspects of the query image as results.

In some embodiments, the data retrieval system is used to index data objects, in addition to uses in retrieval operations in which different sets of data objects (e.g., images or parts of images) are retrieved from a collection of data objects based on features computed for a search data object at different layer positions of a neural network. In an indexing operation, an input data object and descriptors generated for the input data object based on features computed at two or more layer positions of a neural network may be added to a collection of data objects. The collection of data objects can then be used in retrieval operations.

FIG. 1illustrates an example neural network that generates a descriptor for a data object; the descriptor is used to index the data object or to locate similar data objects in a collection. Example neural network100is a deep neural network (DNN). A deep neural network includes at least three layers; an input layer, an output layer, and one or more layers, referred to as middle or hidden layers, between the input layer and output layer.FIG. 1shows a non-limiting example neural network100that includes an input layer110A, an output layer110E, and three hidden layers110B-110D. Each layer110includes one or more nodes112that each perform a computation on input data to generate output data. The input layer110A computes features for an input data object150; each subsequent layer110computes additional features for the input data object150based at least in part on the features output from the previous layer110The features computed at the different layers110characterize different aspects of the input data object150.

A layer of a neural network may be fully-connected or not fully-connected (referred to as convoluted). In a fully-connected layer110, each node112in the layer110is connected to every node112in the previous layer110. In a convoluted layer110, each node112in the layer110is connected to one or more, but not every, node112in the previous layer110. A neural network100may include only fully-connected layers110, only convoluted layers110, or a mix of fully-connected and convoluted layers110. InFIG. 1, layer110E is shown as an example of a fully-connected layer110, as each node112in layer110E is connected to every node112in layer110D, and layer110B is shown as an example of a convoluted layer110, as each node112in layer110B is connected to one or more, but not every, node112in layer110A.

A neural network100as illustrated inFIG. 1may, for example, be used in data retrieval operations. In a conventional retrieval operation, a single descriptor120is generated for an input data object150from features output by one layer110of the neural network100, for example from features output by the output layer110E as shown inFIG. 1. The descriptor120describes some aspect of the input data object150at that layer110. For example, in an image retrieval operation, a descriptor120generated from features output by the output layer110typically describes semantic content of an input image (i.e., objects such as bicycles, people, or faces). The descriptor120is compared to corresponding descriptors for data objects in a collection to select one or more data objects in the collection that include aspects that are similar to an aspect of the input data object150as represented in the descriptor120. In a conventional retrieval operation, a set of data objects is returned that include aspects that best match the aspect described by the single descriptor120for the input data object150. For example, in an image retrieval operation, a set of images that include objects similar to an object or objects in the input image may be returned.

However, a user may be interested in searching for data objects that include aspects similar to aspects of the input data object150other than the aspect described by the single descriptor120for the input data object150. For example, an outdoor image that shows a vehicle on a road could be submitted as a query for images that contain similar vehicles; however, the user may want to locate images with similar backgrounds or lighting conditions to the outdoor image whether or not the images contain similar vehicles. Conventional data retrieval operations do not know the expectations of the user, search for data objects that include aspects similar to the aspect described by the single descriptor120for the input data object150, and so return the same set of data objects (e.g., a set of images that contain vehicles) for the query regardless of the user's expectations in performing the query.

Embodiments of the data retrieval system allow users to select different aspects of search data objects to be matched with similar aspects of other data objects in a collection, thus resolving the ambiguity of retrieval queries in conventional data retrieval operations. The data retrieval system includes a neural network (e.g., a deep neural network) that characterizes data objects and computes features for input data objects that correspond to different aspects of the data objects at multiple layers in the neural network and a user interface that obtains a search data object from a user and obtains a value corresponding to a layer position in the neural network from the user, for example via a slider bar user interface element. A descriptor is generated for the search data object from features output at a layer position of the neural network determined from the input value. The generated descriptor is then used to select one or more of the data objects in a collection that include aspects similar to an aspect of the search data object represented in the descriptor. Indications of the selected data object(s) may be displayed on the user interface. The user can change the input value, for example by moving the slider bar, to view different sets of data objects that include aspects similar to other aspects of the search data object. Thus, embodiments of the data retrieval system allow the user to explore different aspects of the search data object (e.g., a query image) to find a set of data objects that include aspects similar to the aspect of the search data object that the user is interested in.

In some embodiments, the descriptors output by the layer positions are “flat” descriptors (i.e., vectors). However, in some examples, at least one layer position outputs a tensor (i.e., a set of matrices). In some embodiments, the tensor is “flattened” to generate a long vector descriptor. In some embodiments, a pooling operation is performed to reduce the size of the tensor and thus of the descriptor.

FIG. 2illustrates generating descriptors for a search data object that correspond to different aspects of the data object at multiple layer positions in a neural network, according to some embodiments.FIG. 2shows a non-limiting example deep neural network200that includes an input layer210A, an output layer210E, and three hidden layers210B-210D. Each layer210includes one or more nodes212that each perform a computation on input data to generate output data corresponding to features of an input data object250(e.g., an image or portion of an image). A neural network may have more or fewer layers than shown in this example, and the layers may have different numbers of compute nodes. A neural network200may include only fully-connected layers210, only convoluted layers210, or a mix of fully-connected and convoluted layers210. InFIG. 2, layer210E is shown as an example of a fully-connected layer210, and layer210B is shown as an example of a convoluted layer210.

The input layer210A computes features for an input data object250; each subsequent layer210computes additional features for the input data object250based at least in part on the features output from the previous layer210. The features computed at the different layers210characterize different aspects of the input data object250. Features output at shallower layers210are generally structural or statistical features of the input data object250such as background and lighting features of an image, while features output at deeper layers210include semantic features such as edges and details of objects in an image.

In contrast to conventional systems in which a single descriptor is generated for a data object from features computed by a neural network, embodiments of the data retrieval system generate descriptors220from features at multiple layer positions in a neural network200as shown inFIG. 2. In some examples, a layer position refers to a particular layer210of the neural network200, in which case a descriptor220is generated from features output by that layer210. In some examples, a layer position refers to a position between two layers210of the neural network200, in which case a descriptor220is generated from a combination of features output by the two closest layers210. InFIG. 2, descriptors220A,220C,220E,220G, and220I are shown as examples where the layer position is at a particular layer210, and descriptors220B,220D,220F, and220H, are shown as examples where the layer position is between two layers210.

Embodiments of the data retrieval system use a neural network200as illustrated inFIG. 2in performing indexing and retrieval operations. In an indexing operation, an input data object250and descriptors220generated for the input data object250at two or more layer positions are added to a collection of data objects. In a retrieval operation, a descriptor220generated for an input data object250generated from features at a layer position determined from a current value of a user control260(e.g., a slider bar) is compared to corresponding descriptors220for data objects in a collection to select one or more data objects in the collection that include aspects that are similar to an aspect of the input data object250as represented in the descriptor220.

As noted above, in some examples a layer position is between two layers210of the neural network200, in which case a descriptor220is generated from a combination of features output by the two closest layers210. In some embodiments, the two layers210are given equal weight when comparing the descriptors to descriptors of data objects in the collection. However, in some embodiments, the two layers210are given different weights, for example based on the location of the layer position relative to the two layers210. For example, if the layer position is halfway between the two layers210, the two layers210are given equal weight, but if the layer position is closer to one layer210than the other layer210, the two layers210are given different weights, for example 0.25 for the layer farther from the layer position and 0.75 for the layer closer to the layer position.

FIGS. 3A through 3Cillustrate a data retrieval system300that allows a user to select different aspects of a search data object to be matched with similar aspects of other data objects in a collection, according to some embodiments. Data retrieval system300may be used to perform indexing and retrieval operations on a collection310that includes data objects350associated or tagged with descriptors320. Data retrieval system300includes a neural network200(e.g., a deep neural network) that characterizes data objects250and computes features for input data objects250that correspond to different aspects of the data objects at multiple layers in the neural network200. Data retrieval system300also includes a user interface330that obtains search data objects250from the user, and that obtains input values corresponding to different aspects of the search data objects from the user via a user control360such as a slider. The input values map to layer positions in the neural network200. The data retrieval system300also includes a retrieval engine370that selects data objects350from the collection310based on descriptors250generated from features for an input data object250computed at different layer positions of the neural network200corresponding to the input values, and provides indications352of the selected data objects350to the user interface330for display to the user.

FIG. 3Aillustrates an indexing operation in which one or more data objects250are added to a collection310of data objects350and input to a neural network200to generate descriptors220for the input data objects250, for example as illustrated inFIG. 2. Descriptors220generated for the input data object(s)250at two or more layer positions of neural network200are stored as descriptors320that are associated with respective ones of the input data objects250added to the collection310of data objects320. In some examples, a layer position is at a particular layer210of the neural network200, in which case a descriptor220is generated from features output by that layer210. In some examples, a layer position is between two layers210of the neural network200, in which case a descriptor220is generated from a combination of features output by the two closest layers210.

The data objects250and350are, for example, digital images (e.g., still images or video frames, LIDAR images, etc.), or portions of images. In some embodiments, the data retrieval system can index other types of data objects, for example audio or sound files or clips, or various types of textual and/or numerical data files.

Neural network200is implemented on one or more computing devices that include memory and one or more processors (e.g., CPUs, GPUs, etc.) that execute program instructions to perform operations of the neural network200. An example computing device is illustrated inFIG. 10. Collection310is a database or other data structure stored on one or more devices in which descriptors320are associated with data objects350. Collection310can be on the same device that implements neural network200, or can be on one or more separate devices, such as storage devices in a network-based storage system.

FIGS. 3B and 3Cillustrate a retrieval operation via which a user selects different aspects of a search data object250to be matched with similar aspects of other data objects350in a collection310, according to some embodiments. As shown inFIG. 3B, a user specifies or selects a search data object250, for example an image or a portion of an image, via user interface330. The user also specifies a value362A corresponding to an aspect of the search data object, for example via a user control360such as a slider; the value362A maps to a layer position in the neural network200. The search data object250and value362A are submitted to data retrieval system300as a query. The search data object250is input to neural network200(e.g., a deep neural network) for characterizing data objects250that computes features for search data object250that correspond to different aspects of the data object250at multiple layers in the neural network200. The value362A is mapped to a layer position in neural network200, for example by the retrieval engine300. In this example, referring toFIG. 2, the value362A maps to output layer210E. Retrieval engine370obtains a descriptor220I generated from features output by layer210E, or alternatively generates a descriptor220I from features output by layer210E. The retrieval engine370then compares the descriptor220I to corresponding neural network layer descriptors320for data objects350in collection310to select one or more data objects350in the collection310that include aspects similar to the aspect of the search data object250that corresponds to value362A. Indications (352A,352B,352C . . . ) of the selected data object(s)350are provided to the interface330for display to the user. In some embodiments, indications352may include graphical representations of the selected data objects350, such as image thumbnails. In some embodiments, indications352may instead or also include textual representations of the selected data objects350, such as file names and/or descriptions.

The user can review the currently selected data objects350via the user interface310to determine if the selected data objects350are or include what the user is looking for; that is, to determine if the selected data objects350exhibit aspects or characteristics similar to aspects or characteristics of the search data object250that the user is interested in. For example, for images, the user can view the selected images to determine if the images include background aspects, lighting aspects, color aspects, or semantic content (e.g., foreground objects) that the user is interested in. In some embodiments, the user can select particular indications352via user interface310to further review the respective data objects350. For example, the user can select an image thumbnail or image file name to view a higher- or full-resolution version of the image, or select an audio file name to play at least a portion of the audio file. In some embodiments, the search data object250(e.g., an image or a portion of an image) can be displayed so that the user can compare the selected data objects350to the search data object250.

The currently selected data objects350may not include the aspect of the search data object250that the user is interested in, or the user may simply want to explore other aspects of the search data object250to find other data objects350in collection310that exhibit similar aspects. As shown inFIG. 3D, the user can manipulate control360to a different position corresponding to a different aspect of the search data object250, for example using a cursor control device or through touch screen technology; a value362B corresponding to the new position maps to a different layer position in the neural network200. The search data object250and new value362B are submitted to data retrieval system300as a query. In some embodiments, the search data object250is input to neural network200to compute features for search data object250that correspond to different aspects of the data object at multiple layers in the neural network200. In some embodiments, the features and/or descriptors220generated in response to the first query as shown inFIG. 3Aare persisted in a memory, and so neural network220processing of search data object250is not performed in response to input of the new value362B. The value362B is mapped to a layer position in neural network200; in this example, referring toFIG. 2, the value362B maps to a position between layers210A and210B. Retrieval engine370obtains a descriptor220B generated from a combination of features output by layers210A and210B, or alternatively generates a descriptor220B from a combination of features output by layers210A and210B. The retrieval engine370then compares the descriptor220B to corresponding neural network layer descriptors320for data objects350in collection310to select one or more data objects350in the collection310that include aspects similar to the aspect of the search data object250that corresponds to value362B. Indications (352D,352E,352F . . . ) of the selected data object(s)350are provided to the interface330for display to the user.

The user can continue to manipulate control360to different positions corresponding to a different aspects of the search data object250and to review the results of the respective queries at the different positions until the user finds a set (or sets) of data objects350that the user is interested in. In some embodiments, copies of one or more of the selected data objects350can be obtained from the collection310by the user. For example, interface330may include a “get copy” user interface element via which a user can request a copy of a specified data object350. Thus, embodiments allow the user to explore different aspects of the search data object250(e.g., a query image) to find data objects350that include aspects similar to the aspect of the search data object250that the user is interested in. Note that the user need not be aware that the position of control360corresponds to layer positions of a neural network200; to the user, different positions of the control360return different sets of data objects350(e.g., images) that correspond to different aspects of the search data object250as results.

FIG. 4is a flowchart of a method of operation for a data retrieval system as illustrated inFIGS. 3A and 3B, according to some embodiments. As indicated at400, input specifying a search data object (e.g., an image or a portion of an image) is obtained. In some embodiments, a user interface to the data retrieval system includes one or more user interface elements via which the user can specify or select a data object, for example an image or a portion of an image, for which the user is interested in locating data objects with similar aspects.

As indicated at410, a value corresponding to an aspect of the search data object to be matched with similar aspects of other data objects is obtained. In some embodiments, the user interface to the data retrieval system includes a user interface element, for example a slider bar or dial, that the user can manipulate, for example using a cursor control device or through touch screen technology, to select different positions on the user interface element that correspond to different values. The different values map to different layer positions of a neural network that characterizes data objects, for example a deep neural network200as illustrated inFIG. 2.

As indicated at420, the system may determine, based on the value, a layer position in the neural network. The layer position can be at a particular layer of the neural network, or between two layers.

As indicated at430, the search data object is input to the neural network to generate a descriptor for the search data object from one or more features output by the determined layer position of the neural network. If the layer position is at a particular layer of the neural network, a descriptor is generated from features output by that layer. If the layer position is between two layers, a descriptor is generated from a combination of features output by the two closest layers.

As indicated at440, the generated descriptor is compared to corresponding neural network layer descriptors for data objects in a collection of data objects to select one or more data objects in the collection that include aspects similar to the aspect of the search data object. As indicated at450, an indication of the one or more selected data objects (e.g., names and/or descriptions of the data objects, image thumbnails, lower-resolution versions of full-sized images, etc.) is provided for display on the user interface. In some embodiments, the user can select particular data objects via the user interface to further review the selected data objects; for example, the user can select an image thumbnail or image file name to view the image, or select an audio file name to play at least a portion of the audio file. In some embodiments, the search data object (e.g., an image or a portion of an image) can be displayed so that the user can compare the selected data objects to the search data object.

The user can review the currently selected data objects via the user interface to determine if the data objects are or include what the user is looking for; that is, to determine if the data objects exhibit aspects or characteristics similar to aspects or characteristics of the search data object that the user is interested in. For example, for images, the user can view the selected images to determine if the images include background aspects, lighting aspects, color aspects, or semantic content (e.g., foreground objects) that the user is interested in.

At460, if the currently selected data objects do not exhibit the aspect of the search data object that the user is interested in, or if the user wants to view data objects that match other aspects of the search data object, then the user can manipulate the user interface element (e.g., slider bar) to change the value that is mapped to the layer positions of the neural network; the method returns to element410. A new layer position is determined, a descriptor is generated for the search data object from features output by one or more neural network layers at the new layer position, and a new set of data objects is selected from the collection based on the new descriptor and displayed to the user via the user interface. Thus, embodiments allow the user to explore different aspects of the search data object (e.g., a query image) to find a set of data objects that include aspects similar to the aspect of the search data object that the user is interested in. Note that the user need not be aware that the position of the slider bar corresponds to layer positions of a neural network; to the user, different positions on the slider bar return different sets of data objects (e.g., images) that correspond to different aspects of the search data object as results.

FIG. 5provides another flowchart of a method of operation for a data retrieval system as illustrated inFIGS. 3A and 3B, according to some embodiments.

As indicated at500, input specifying a search data object (e.g., an image or a portion of an image) is obtained. In some embodiments, a user interface to the data retrieval system includes one or more user interface elements via which the user can specify or select a data object, for example an image or a portion of an image, for which the user is interested in locating data objects with similar aspects.

As indicated at510, the search data object may be input to a neural network for characterizing data objects, for example a deep neural network200as illustrated inFIG. 2, to compute features for the search data object at respective layers of the neural network; the features computed at the different layers characterize different aspects of the input data object. In some embodiments, descriptors for different layer positions of the neural network, including layer positions at and between the layers, can be generated from the features output at the layer positions at this time. The features and/or descriptors may be persisted to a memory.

As indicated at520, a value corresponding to an aspect of the search data object to be matched with similar aspects of other data objects is obtained. In some embodiments, the user interface to the data retrieval system includes a user interface element, for example a slider bar or dial, that the user can manipulate, for example using a cursor control device or through touch screen technology, to select different positions on the user interface element that correspond to different values. The different values map to different layer positions of the neural network.

As indicated at530, the system may determine, based on the value, a layer position in the neural network. The layer position can be at a particular layer of the neural network, or between two layers. At540, if the layer position is at a layer, then as indicated at542a descriptor is generated from the features computed at that layer. Alternatively, in some embodiments, a descriptor generated at element510may be obtained. At540, if the layer position is between two layers, then as indicated at544a descriptor is generated from a combination of the features computed at the two closest layers. Alternatively, in some embodiments, a descriptor generated at element510may be obtained.

At550, the generated descriptor is compared to corresponding neural network layer descriptors for data objects in a collection of data objects to select one or more data objects in the collection that include aspects similar to the aspect of the search data object. As indicated at560, an indication of the one or more selected data objects (e.g., names and/or descriptions of the data objects, image thumbnails, lower-resolution versions of full-sized images, etc.) is provided for display on the user interface. In some embodiments, the user can select particular data objects to further review the selected data objects; for example, the user can select an image thumbnail or image file name to view the image, or select an audio file name to play at least a portion of the audio file. In some embodiments, the search data object (e.g., an image or a portion of an image) can be displayed so that the user can compare the selected data objects to the search data object.

The user can review the currently selected data objects via the user interface to determine if the data objects are or include what the user is looking for; that is, to determine if the data objects exhibit aspects or characteristics similar to aspects or characteristics of the search data object that the user is interested in. For example, for images, the user can view the selected images to determine if the images include background aspects, lighting aspects, color aspects, or semantic content (e.g., foreground objects) that the user is interested in.

At570, if the currently selected data objects do not exhibit the aspect of the search data object that the user is interested in, or if the user wants to view data objects that match other aspects of the search data object, then the user can manipulate the user interface element (e.g., slider bar) to change the value that is mapped to the layer positions of the neural network; the method returns to element520. A new layer position is determined, a new set of data objects is selected from the collection based on a descriptor corresponding to the new layer position, and the new set of data objects is displayed to the user via the user interface. Thus, embodiments allow the user to explore different aspects of the search data object (e.g., a query image) to find a set of data objects that include aspects similar to the aspect of the search data object that the user is interested in. Note that the user need not be aware that the position of the slider bar corresponds to layer positions of a neural network; to the user, different positions on the slider bar return different sets of data objects (e.g., images) that correspond to different aspects of the search data object as results.

FIG. 6is a flowchart of a method for performing operations based on features of the data objects output by one or more layers of a neural network as illustrated inFIG. 2, according to some embodiments. As indicated at600, a data object is input to a neural network for characterizing data objects, for example a deep neural network200as illustrated inFIG. 2. As an example, the data object is a digital image or a portion of an image.

As indicated at610, the layers of the neural network compute features for the input data object; the features computed at the different layers characterize different aspects of the input data object. Features output at shallower layers of the neural network generally include structural or statistical features of the input data object such as background and lighting features of an image, while features output at deeper layers include semantic features such as edges and details of objects in an image (e.g., a bicyclist or person). Features output at middle layers generally represent the overall structure of the input data object, such as the overall setting of a scene in an image (e.g., a street scene with trees or buildings).

As indicated at620, one or more descriptors are generated for the input data object from the features output by one or more of the plurality of layers. A descriptor can be generated from features output by one layer, or alternatively can be generated from a combination of features output by two neighboring layers.

As indicated at630, an operation is performed for the input data object with respect to a collection of data objects using the one or more descriptors for the input data object. For example, an indexing operation can be performed in which the input data object and the descriptors generated for the input data object are added to the collection of data objects, for example as illustrated inFIG. 3A. As another example, a retrieval operation can be performed in which at least one of the descriptors generated for the input data object is compared to corresponding neural network layer descriptors for data objects in the collection to select one or more data objects in the collection that include aspects that are similar to an aspect of the input data object, for example as illustrated inFIGS. 3B and 3C.

In some embodiments, to perform a retrieval operation, a value is obtained corresponding to an aspect of the input data object to be matched with similar aspects of other data objects from a user interface element of a user interface; the user interface element can, for example, be a slider bar that is adjustable to select from a range of aspects. A layer position in the neural network is determined based on the value. A descriptor for the input data object generated from the features output by the determined layer position is compared to corresponding neural network layer descriptors for the data objects in the collection to select one or more data objects in the collection that include aspects similar to the aspect of the input data object. An indication of the one or more selected data objects is displayed on the user interface. The user can adjust the user interface element (e.g., slider bar) to change the value that is mapped to the layer positions of the neural network. A new layer position is determined, a new set of data objects is selected from the collection based on a descriptor corresponding to the new layer position, and the new set of data objects is displayed to the user via the user interface.

At640, if there are more data objects to be processed, then the method returns to element600to obtain and process a next data object. Otherwise, the method is done.

FIGS. 7A through 7Dillustrate example implementations of a data retrieval system, according to some embodiments. Note that these examples are not intended to be limiting; other implementations are possible. An example computing device that can be used as any of the devices shown inFIGS. 7A through 7Dis illustrated inFIG. 10.

FIG. 7Ashows an example implementation in which the interface330is implemented on a device700, and the neural network200, retrieval engine370, and collection310are implemented on one or more other devices720. Device700can, for example, be a mobile multipurpose device such as a smartphone or tablet/pad device, or a personal computer such as a desktop, laptop, or notebook computer. Device(s)720can, for example, include one or more application or web server systems. In some embodiments, device700communicates with device(s)720via a wired and/or wireless connection over an intermediate network710such as the Internet. Interface330can, for example, be implemented as a web browser interface, mobile application or applet, or other type of application on device700.

FIG. 7Bshows an example implementation in which the interface330and retrieval engine370are implemented on a device700, and the neural network200and collection310are implemented on one or more other devices720. Device700can, for example, be a mobile multipurpose device such as a smartphone or tablet/pad device, or a personal computer such as a desktop, laptop, or notebook computer. Device(s)720can, for example, include one or more application or web server systems. In some embodiments, device700communicates with device(s)720via a wired and/or wireless connection over an intermediate network710such as the Internet. Interface330and retrieval engine370can, for example, be implemented as a web browser interface, mobile application or applet, or other type of application on device700.

FIG. 7Cshows an example implementation in which the interface330and neural network200are implemented on a device700, and the retrieval engine370and collection310are implemented on one or more other devices720. Device700can, for example, be a mobile multipurpose device such as a smartphone or tablet/pad device, or a personal computer such as a desktop, laptop, or notebook computer. Device(s)720can, for example, include one or more application or web server systems. In some embodiments, device700communicates with device(s)720via a wired and/or wireless connection over an intermediate network710such as the Internet to send data objects and descriptors generated for the data objects by the neural network200to the retrieval370and to receive results from the retrieval engine370. Interface330and neural network200can, for example, be implemented as a web browser interface, mobile application or applet, or other type of application on device700. In some embodiments, a neural network200can also be implemented on device(s)720to index data objects in the collection310.

FIG. 7Dshows an example implementation in which the interface330, retrieval engine370, neural network200and collection310are implemented on a device750. Device750can, for example, be a mobile multipurpose device such as a smartphone or tablet/pad device, a personal computer such as a desktop, laptop, or notebook computer, or other types of computing devices. Interface330and retrieval engine370can, for example, be implemented as a mobile application or applet, or other type of application, on device750.

Embodiments of the data retrieval system have been described that allow users to select different aspects of a search data object to be matched with similar aspects of other data objects in a collection. In some embodiments, the data retrieval system allows users to select multiple aspects of a search data object to be used in a search of the collection, and/or to select aspects of two or more data objects to be used in a search of the collection. In these embodiments, an interface330to the system300as shown inFIGS. 3A-3Cincludes a “save aspect” or similar user interface element via which the user can save particular aspects of the search data object250that the user has identified by inputting different values via user control360. In some embodiments, selecting the “save aspect” user interface element causes the system300to store or mark the descriptor220corresponding to the respective aspect. The user can then use the user control360to search for, identify, and save one or more other aspects of the search data object250. The interface330also includes a “search using saved aspects” or similar user interface element that, when selected by the user, causes the retrieval engine370to search for matching data objects350in collection310using a combination of the stored or marked descriptors220for the search data object250, and return one or more data object352that best match the combination of aspects.

In some embodiments, the system300allows the user to search for, identify, and save one or more aspects of two or more search data objects250. The system300stores or marks descriptors220corresponding to aspects of the two or more search data objects250that are saved by the user via the “save aspect” user interface element. The user can select the “search using saved aspects” user interface element, which causes the retrieval engine370to search for matching data objects350in collection310using a combination of the stored or marked descriptors220for the two or more search data objects250, and return one or more data object352that best match the combination of aspects from the two or more search data objects250.

FIG. 8is a high-level flowchart of a method for selecting multiple aspects of a search data object for searching a collection of data objects, according to some embodiments. As indicated at800, the user inputs a data object to the system via a user interface. As indicated at810, the user explores different aspects of the data object using the user interface and selects two or more aspects of interest corresponding to two or more layer positions via the user interface. The system stores or marks descriptors corresponding to the selected aspects. As indicated at820, the descriptors for the two or more layer positions are combined and used to search the collection of data objects to select data objects in the collection that include aspects similar to the two or more aspects of interest selected by the user. As indicated at830, an indication of the one or more selected data objects (e.g., names and/or descriptions of the data objects, image thumbnails, lower-resolution versions of full-sized images, etc.) are provided for display on the user interface.

FIG. 9is a high-level flowchart of a method for selecting aspects in multiple data objects for searching a collection of data objects, according to some embodiments. As indicated at900, the user inputs multiple data objects to the system via a user interface. As indicated at910, the user explores different aspects of the data objects using the user interface and selects one or more aspects of interest from at least two of the data objects. The system stores or marks descriptors corresponding to the selected aspects. As indicated at920, descriptors corresponding to the selected aspects from the at least two data objects are combined and used to search the collection of data objects to select data objects in the collection that include aspects similar to the aspects of interest selected by the user. As indicated at930, an indication of the one or more selected data objects (e.g., names and/or descriptions of the data objects, image thumbnails, lower-resolution versions of full-sized images, etc.) are provided for display on the user interface.

Example Use Cases

Embodiments of a data retrieval system as illustrated inFIGS. 2 through 9can be used in any of a variety of applications, including but not limited to user experience applications, for example on mobile multipurpose devices, and machine learning applications. The following provides some example use cases for embodiments of a data retrieval system as illustrated inFIGS. 2 through 9.

An example use case is in generic image searching and retrieval for image curation applications. Embodiments enable searches for objects in or other aspects of images for which there is not an ad-hoc trained classification algorithm; the located images can then be sent for annotation in the curation application.

Another example use case is bootstrap training for object classification or object detection algorithms. Embodiments allow images with a specific feature to be quickly identified and retrieved in order to train a classification algorithm to detect the feature, or to train a detection algorithm to detect desired objects as well as their positions in images.

Another example use case is in training neural networks. Embodiments allow images with a specific feature or features to be quickly identified and retrieved using one neural network in order to train another neural network for those feature(s).

Another example use case is in diagnosing neural networks. For example, a user may have two images that should be considered similar, but for some reason the neural network classifies the images as different. The user can use the system to compare the two images using the slider to select different aspects at different levels of the neural network to determine at which layer position the images diverge. This information can be used to determine that from the input layer to the diverging point the network is well-trained, but the neural network needs to be refined from the diverging point to the output layer. The first part of the network can then be frozen, and the remainder of the network can be further trained.

Example Computing Device

FIG. 10illustrates an example computing device, referred to as computer system2000, that may be used in embodiments of a data retrieval system as illustrated inFIGS. 2 through 9. In different embodiments, computer system2000may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, server, laptop, notebook, tablet or pad device, slate, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, a wireless phone, a smartphone, a consumer device, video game console, handheld video game device, application server, storage device, a television, a video recording device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device.

In the illustrated embodiment, computer system2000includes one or more processors2010coupled to a system memory2020via an input/output (I/O) interface2030. Computer system2000further includes a network interface2040coupled to I/O interface2030, and one or more input/output devices2050, such as cursor control device2060, keyboard2070, and display(s)2080. Computer system2000may also include one or more cameras2090, which may also be coupled to I/O interface2030.

In various embodiments, computer system2000may be a uniprocessor system including one processor2010, or a multiprocessor system including several processors2010(e.g., two, four, eight, or another suitable number). Processors2010may be any suitable processor capable of executing instructions. For example, in various embodiments processors2010may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors2010may commonly, but not necessarily, implement the same ISA.

System memory2020may store program instructions2022and data2032accessible by processor2010. In various embodiments, system memory2020may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions2022and data2032may include, but are not limited to, program instructions for implementing and data used in embodiments of the data retrieval system as described in reference toFIGS. 2 through 9. In some embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory2020or computer system2000.

In one embodiment, I/O interface2030may coordinate I/O traffic between processor2010, system memory2020, and any peripheral devices in the device, including network interface2040or other peripheral interfaces, such as input/output devices2050. In some embodiments, I/O interface2030may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory2020) into a format suitable for use by another component (e.g., processor2010). In some embodiments, I/O interface2030may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface2030may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface2030, such as an interface to system memory2020, may be incorporated directly into processor2010.

Network interface2040may allow data to be exchanged between computer system2000and other devices attached to a network2085(e.g., carrier or agent devices) or between nodes of computer system2000. Network2085may in various embodiments include one or more networks including but not limited to Local Area Networks (LANs) (e.g., an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof. In various embodiments, network interface2040may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.

Input/output devices2050may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or accessing data by computer system2000. Multiple input/output devices2050may be present in computer system2000or may be distributed on various nodes of computer system2000. In some embodiments, similar input/output devices may be separate from computer system2000and may interact with one or more nodes of computer system2000through a wired or wireless connection, such as over network interface2040.