Patent Description:
This document describes techniques and systems that enable privacy controls for sharing embeddings for searching and indexing media content. The techniques described herein allow a user to give access to searching and indexing features based on an embedding of the user's face to a limited, whitelisted set of users and subsequently revoke the access from one or more of those users.

The techniques and systems include a method that controls privacy for sharing embeddings for searching and indexing media content. The method includes obtaining a set of images of a first user's face and applying a machine-learned model to the set of images to generate a user-specific dataset of face embeddings for the first user. Then, media content stored in a media storage is indexed based on the user-specific dataset to provide indexed information identifying one or more faces shown in the media content. Access to the indexed information by a second user querying the media content for images or videos depicting the first user is controlled based on a digital key shared by the first user with the second user, where the digital key is associated with the user-specific dataset and the user-specific dataset is usable to identify the images or videos depicting the first user.

The digital keys may be stored only "in the cloud," such as at a service provider system or other secure online storage associated with the service provider system, and not stored at the user's devices. The service provider system may provide online storage of personalized media collections for users. These personalized media collections may be account-based and securely encrypted. In such an implementation, the user can provide user credentials to log in to their account and then initiate the search query through the service provider system based on their account. In this way, shared digital keys are not shared with any actual user devices and may therefore be kept more secure from counterfeits and copies. Further, a personalized media collection that is searchable for a particular user may be tied to the account of that user, which protects other media collections from unauthorized searches.

Users' personalized media collections may be indexed, and the service provider system may store indexed information for each media collection separately. Keeping different users' indexed information separately stored can provide an additional level of security against unauthorized access (e.g., permitted access to a first user's indexed information gaining unauthorized access to a second user's indexed information).

This summary is provided to introduce simplified concepts concerning privacy controls for sharing embeddings for searching and indexing media content, which is further described below in the Detailed Description and Drawings. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter. "<NPL>) describes a face search system which combines a fast search procedure, coupled with a state-of-the-art commercial of the shelf (COTS) matcher, in a cascaded framework. "<NPL>) describes a scheme to achieve privacy-preserving face recognition in the cloud. "<NPL>) describes a framework (HideMe) to preserve the associated users' privacy for online photo sharing.

Patent application <CIT> discloses teachings relevant to the technical field of the application.

The details of one or more aspects of privacy controls for sharing embeddings for searching and indexing media content are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:.

This document describes techniques and systems that enable privacy controls for sharing embeddings for searching and indexing media content. Sharing embeddings of a user, such as face embeddings, can allow another user to detect the face of the user in arbitrary data, which may not be desirable. For example, a leaked embedding may allow anyone using it to detect the user's face in public video streams. The techniques described in this document provide a software-based solution to control access to searching for a particular user depicted in media content. In particular, the techniques allow a user to give access to searching and indexing features based on an embedding of the user's face to a limited, whitelisted set of users and subsequently revoke access to one or more of the users.

In aspects, a method for privacy controls for face-embedding sharing for searching and indexing media content is described. The method is performed by a service provider system obtaining a set of images of a face of a first user and applying a machine-learned model to the set of images to generate a user-specific dataset of face embeddings for the first user. The method also includes the service provider system indexing media content stored in a media storage by applying the machine-learned model to the media content to provide indexed information identifying one or more faces shown in the media content. Additionally, the method includes the service provider system controlling, by an application programming interface, access to the indexed information by a second user querying the media content for images or videos depicting the first user. The access is controlled based on a digital key shared by the first user with the second user. The digital key is associated with the user-specific dataset. The user-specific dataset is usable with the indexed information to identify the images or videos in the media content that depict the first user.

These are but a few examples of how the described techniques and devices may be used to enable privacy controls for face-embedding sharing or searching and indexing media content. Other examples and implementations are described throughout this document. The document now turns to an example operating environment, after which example devices, methods, and systems are described.

<FIG> illustrates an example environment <NUM> in which techniques enabling privacy controls for face-embedding sharing for searching and indexing media content can be implemented. The example environment <NUM> includes an electronic device <NUM> (e.g., a user device) configured to communicate with a service provider system <NUM> via a network <NUM>. The electronic device <NUM> includes a privacy-control module <NUM>, an embedding module <NUM>, one or more digital keys <NUM>, and media content <NUM>.

The service provider system <NUM> is illustrated as including media storage <NUM>, indexed media information <NUM>, a storage service module <NUM>, a machine-learned model <NUM>, one or more user-specific datasets <NUM>, and a search manager module <NUM>. The storage service module <NUM> can apply the machine-learned model <NUM> to media content <NUM> provided by the electronic device <NUM> (e.g., image data, video data, audio data, etc.). The machine-learned model <NUM> (e.g., a convolutional neural network) is trained using machine-learning techniques to generate embeddings from the media content <NUM> to create a user-specific dataset <NUM> for a user depicted in the media content <NUM>. For example, the media content <NUM> may include a set of images of a user, such as a face of the user of the electronic device <NUM>. The storage service module <NUM> can run a general purpose face detect algorithm to detect a face boundary of the face in the images. Then, using the machine-learned model <NUM>, the storage service module <NUM> computes embeddings (also referred to as embedding vectors) for substantially all the faces found in the images to create the user-specific dataset <NUM> for the user. The user-specific dataset <NUM> is one or more n-dimensional vectors representing the user's face. The user-specific dataset <NUM> is stored "in the cloud" securely encrypted for the user and inaccessible by other devices.

The storage service module <NUM> can index the media storage <NUM> by identifying users who appear in it, such as in images or videos in the media storage <NUM>, to generate the indexed media information <NUM>. For security, the storage service module <NUM> only identifies those users who have consented and have opted to store their user-specific dataset <NUM> in the cloud (e.g., at the service provider system <NUM>). It is noted that access to the user-specific dataset <NUM> is not allowed to any other user or device without the user's consent.

The user of electronic device <NUM> can initiate a search query to the service provider system <NUM> to search for images of the user in the media storage <NUM>. The electronic device <NUM> calls an application programming interface (API) <NUM> at the search manager module <NUM> to access the indexed media information <NUM> and search for embedding information that substantially matches the user-specific dataset <NUM> of the user. Based on matching embedding information, the search manager module <NUM> can locate corresponding images in the media storage <NUM> that depict the user. Resulting images are then presented at the electronic device <NUM> in a display application via the display device <NUM>.

The user of the electronic device <NUM> may share access to the indexed media information <NUM> corresponding to the user to one or more other users (e.g., family, friends, etc.). This allows the other users to query the media storage <NUM> with user-specific queries. In an example, user-B initiates a user-specific query, at their user device <NUM>, to find images of user-A (user of electronic device <NUM>) in the media storage <NUM> of the service provider system <NUM>. However, neither the machine-learned model <NUM> nor the user-specific dataset <NUM> is shared with the user-B. Instead, the user-B has a digital key, previously provided by the user-A, corresponding to the user-specific dataset <NUM> of the user-A. The user-B provides the digital key to the search manager module <NUM>. The API <NUM> of the search manager module <NUM> uses the digital key to identify the user-specific dataset <NUM> of the user-A. The API <NUM> compares the embeddings in the user-specific dataset <NUM> of the user to the indexed media information <NUM> to identify matching embeddings and index information corresponding to the matching embeddings. The API <NUM> uses the identified index information to locate corresponding images or videos, from the media storage <NUM>, which depict user-A. The API <NUM> can then return the corresponding images to the user device <NUM> of the user-B for display.

In some aspects, the user-B may wish to query a different image or video corpus with user-specific queries of the user-A. For example, the user-B may query a media storage service <NUM> for images of the user-A in the media storage <NUM>. Alternatively, the user-B may query local storage (not shown) at the user device <NUM>. In either case, the user-B initiates a query to the API <NUM>, with the query including the digital key shared by the user-A and identification of the location of the particular image or video corpus to search (e.g., the media storage <NUM> or local storage of the user device <NUM>). If these corpuses have not yet been indexed by the storage service module <NUM>, the API <NUM> applies (or calls the storage service module <NUM> to apply) the machine-learned model <NUM> to the particular image or video corpus to index the content. The storage service module <NUM> of the service provider system <NUM> retains the resulting indexed information in the indexed media information <NUM>. The search manager module <NUM> can then compare the resulting indexed information to the user-specific dataset <NUM> to identify which images in the particular image or video corpus (e.g., the media storage <NUM> or local storage of the user device <NUM>) include the user-A and provide results to the user device <NUM> for the user-B.

In some aspects, the electronic device <NUM> may include the machine-learned model <NUM>, or an instance of the machine-learned model <NUM>. The embedding module <NUM> can apply the machine-learned model <NUM> to a set of images captured and/or stored by the electronic device <NUM> in the media content <NUM> to generate the user-specific dataset <NUM> (e.g., set of embeddings). In some aspects, the set of images may be images captured using a near-infrared camera, such as by a face authentication system (e.g., face unlock application) that captures near-infrared image data of the user's face and generates embeddings (n-dimensional numerical vectors) representing the user's unique facial features in a numerical fashion without including personally-identifiable information. Then, the electronic device <NUM> can transmit (e.g., upload) the embeddings (e.g., the user-specific dataset <NUM>) to the service provider system <NUM> to store it "in the cloud. " In one example, the electronic device <NUM> or the service provider system <NUM> can create a three-dimensional (3D) representation of a face to then use as input data for the face embeddings.

The privacy-control module <NUM> is configured to generate the digital keys <NUM> for sharing with authorized users. Only the authorized users who have a shared key (e.g., digital key <NUM>) can call the API <NUM> to access the indexed media information <NUM> to identify particular images or videos, from an image or video corpus, depicting the user.

Because the user-specific dataset <NUM> is not exposed directly to other users (the only access granted to other users is to the indexed content for search queries), the granted access can be revoked. The user-A of the electronic device <NUM> can select to revoke the shared access by deleting the digital key previously shared with a particular user (e.g., the user-B). Then, the next time that the user-B initiates a search for images or videos depicting the user-A, no results are provided.

Throughout this disclosure examples are described where a computing system (e.g., the electronic device <NUM>, a client device, a server device, the service provider system <NUM>, a computer, or other type of computing system) may analyze information (e.g., radar, inertial, and facial-recognition sensor data, images) associated with a user, such as the just-mentioned face images. The computing system, however, can be configured to only use the information after the computing system receives explicit permission from the user of the computing system to use the data. The individual users may have constant control over what programs can or cannot do with sensor data. For example, before the electronic device <NUM> shares sensor data with another device (e.g., to train a model executing at another device), the electronic device <NUM> may pre-treat the sensor data to ensure that any user-identifying information or device-identifying information embedded in the data is removed. Thus, the user may have control over whether information is collected about the user and the user's device, and how such information, if collected, may be used by the computing device and/or a remote computing system.

Although the examples described herein are directed to image data depicting the user's face, the techniques described herein can also be implemented on other types of data to generate a corresponding embedding, which can be used to search media content for data corresponding to the user. Some example other types of input may include voice data, substantial full-body images, fingerprint data, iris-scan data, video data, and so forth. Embeddings generated from voice data can be used to find videos or audio files, or audio segments within the videos or audio files, of the user speaking. In one example, these techniques can be used to identify when, during a recorded meeting, a particular person speaks or who is speaking at different times during the meeting. Embeddings generated from full-body images of a user can be used to identify a user based on their gait or how they move. These techniques can also be used for face authentication. For example, if a company uses face authentication and has the user-specific dataset for the user, then a digital key can be given to the user to access the API with face scanning to enter a secure building. In yet another example, using these techniques, a user may not be required to train a new electronic device for a face unlock application. Rather, a digital key can be provided to the new electronic device, which allows the new electronic device to call the API to access to the user-specific dataset for authenticating the user for face unlock.

In more detail, consider <FIG>, which illustrates an example implementation <NUM> of the electronic device <NUM> that can implement privacy controls for face-embedding sharing for searching and indexing media content. The electronic device <NUM> of <FIG> is illustrated with a variety of example devices, including a smartphone <NUM>-<NUM>, a tablet <NUM>-<NUM>, a laptop <NUM>-<NUM>, a desktop computer <NUM>-<NUM>, a computing watch <NUM>-<NUM>, computing spectacles <NUM>-<NUM>, a gaming system <NUM>-<NUM>, a home-automation and control system <NUM>-<NUM>, and a microwave <NUM>-<NUM>. The electronic device <NUM> can also include other devices, such as televisions, entertainment systems, audio systems, automobiles, drones, track pads, drawing pads, netbooks, e-readers, home security systems, and other home appliances. Note that the electronic device <NUM> can be wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances).

The electronic device <NUM> also includes one or more computer processors <NUM> and one or more computer-readable media <NUM>, which includes memory media and storage media. Applications and/or an operating system <NUM> implemented as computer-readable instructions on the computer-readable media <NUM> can be executed by the computer processors <NUM> to provide some or all of the functionalities described herein. For example, the computer-readable media <NUM> can include the privacy-control module <NUM>, the embedding module <NUM>, the media content <NUM>, the machine-learned model <NUM>, and a secure storage unit <NUM>. The embedding module <NUM> can call the machine-learned model <NUM>. The privacy-control module <NUM> can control (e.g., grant, revoke) access to the digital keys <NUM>, which authorize user-specific searches in a media storage.

The secure storage unit <NUM> is configured to store security data (e.g., user credentials) used for privacy controls, such as controls to unlock the electronic device <NUM> (including face authentication data, password/passcode information, fingerprint data, and so on). Although this security data can be used to authenticate the user to unlock the electronic device <NUM> using face authentication, password/passcode authentication, fingerprint authentication, voice authentication, and so on, personal information about the user cannot be obtained by the security data. Specifically, the user cannot be identified by the security data. Rather, the security data is used to simply determine whether data received from a user attempting to unlock the phone matches stored profile data (e.g., the user-specific dataset <NUM>) representing a user that set up the security on the electronic device <NUM>. In an example, the embeddings generated from captured images of the user's face are numerical vector representations of facial features of the user. These embeddings are simply used for comparison to new embeddings, generated from images captured during a face-authentication attempt, to locate a match. In other implementations, these embeddings are used for comparison to new embeddings generated from images depicting the user's face, captured by a camera of the electronic device <NUM> or stored in the media content <NUM> (obtained from another device).

The electronic device <NUM> may also include a network interface <NUM>. The electronic device <NUM> can use the network interface <NUM> for communicating data over wired, wireless, or optical networks. By way of example and not limitation, the network interface <NUM> may communicate data over a local-area-network (LAN), a wireless local-area-network (WLAN), a personal-area-network (PAN), a wide-area-network (WAN), an intranet, the Internet, a peer-to-peer network, point-to-point network, or a mesh network.

Various implementations of the authentication system <NUM> can include a System-on-Chip (SoC), one or more Integrated Circuits (ICs), a processor with embedded processor instructions or configured to access processor instructions stored in memory, hardware with embedded firmware, a printed circuit board with various hardware components, or any combination thereof. In an example, the authentication system <NUM> can, in a secure mode, compare authentication data received from the user to security data stored in the secure storage unit <NUM> for authenticating the user to unlock the electronic device <NUM>. In some aspects, the authentication system <NUM> generates the authentication data using image data obtained from the camera system and provides the authentication data to the secure storage unit <NUM> to enable the secure storage unit <NUM> to compare the authentication data to the stored security data and determine if there is a match.

The electronic device <NUM> also includes a camera system <NUM> implemented to capture image data. The image data may be usable to generate a three-dimensional depth map of an object, such as a user's face. Any suitable camera system may be used, including a color camera (e.g., Red-Green-Blue (RGB) camera or a near-infrared (NIR) camera). The camera system <NUM> may be integrated into, or otherwise associated with, the electronic device <NUM>. In aspects, the camera system <NUM> may be wirelessly connected to the electronic device <NUM>.

The electronic device <NUM> can also include one or more sensors <NUM>. The one or more sensors <NUM> can include any of a variety of sensors, such as an audio sensor (e.g., a microphone), a touch-input sensor (e.g., a touchscreen), an image-capture device (e.g., a camera or video-camera), proximity sensors (e.g., capacitive sensors), or ambient light sensors (e.g., a photodetector).

The electronic device <NUM> can also include the display device <NUM>. The display device <NUM> can include any suitable display device, such as a touchscreen, a liquid crystal display (LCD), thin film transistor (TFT) LCD, an in-place switching (IPS) LCD, a capacitive touchscreen display, an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode (AMOLED) display, super AMOLED display, and so forth.

The electronic device <NUM> can also include a face detector <NUM>. The face detector <NUM> can detect a face boundary of each of one or more faces in an image. By detecting the face boundary, the face detector <NUM> limits the amount of image data that is processed by the embedding module <NUM>. In implementations, the face detector <NUM> can be implemented as computer-readable instructions on the computer-readable media <NUM> and executed by the computer processors <NUM> to detect a face boundary of respective faces in the image.

These and other capabilities and configurations, as well as ways in which entities of <FIG> act and interact, are set forth in greater detail below. These entities may be further divided, combined, and so on. The environment <NUM> of <FIG>, the implementations <NUM>, <NUM>, and <NUM> of <FIG>, and the detailed methods of <FIG> illustrate some of many possible environments and devices capable of employing the described techniques.

<FIG> illustrates an example implementation <NUM> of privacy controls for sharing embeddings for searching and indexing media content. <FIG> illustrates the service provider system <NUM> and multiple electronic devices <NUM>, e.g., device-A <NUM>, device-B <NUM>, and device-C <NUM>. In the illustrated example, each of the devices corresponds to a different user, e.g., the device-A <NUM> corresponds to user-A <NUM>, the device-B corresponds to user-B <NUM>, and the device-C corresponds to user-C <NUM>.

Each of the devices may be implemented with an instance of the privacy-control module <NUM> to control access to image or video searches of the corresponding user. In this example implementation <NUM>, the user-A <NUM> of device-A <NUM> has selected to store a user-specific dataset <NUM> (e.g., dataset-A <NUM>) at the service provider system <NUM>. The dataset-A <NUM> is generated based on the machine-learned model <NUM> being applied to a set of images of the user-A <NUM>, either provided by the device-A <NUM> or stored in the media storage <NUM> at the service provider system <NUM>, or both.

The service provider system <NUM> provides online storage of personalized media collections for users. These personalized media collections may be account-based and securely encrypted in the media storage <NUM>. Each user may have one or more personalized media collections, such as images and videos organized in different folders or subfolders. Because the personalized media collections are account-based, a user may log in to their account using any suitable electronic device and upload images and/or videos to create the personalized media collection. For simplicity of the illustrated example, each user is illustrated with one electronic device and one personalized media collection. For instance, the media storage <NUM> includes a media collection-A <NUM> uploaded by the user-A <NUM>, a media collection-B <NUM> uploaded by the user-B <NUM>, and a media collection-C <NUM> uploaded by the user-C <NUM>.

To share access to a user-specific dataset <NUM> for searching for media content depicting the user, the corresponding privacy-control module <NUM> can provide a corresponding digital key <NUM>. A new key is created for each user with whom access is shared (e.g., ten different keys generated for ten users, respectively). To revoke access for a particular user, privacy-control module <NUM> deletes their corresponding key. In the illustrated example, the device-A <NUM> and/or the service provider system <NUM> stores set of digital keys <NUM> (e.g., owned keys <NUM>, which include keys A1, A2,. , An) for the user-A <NUM>. Also, a set of digital keys <NUM> (e.g., owned keys <NUM>, which include keys B1, B2,. , Bn) are stored by the device-B <NUM> and the service provider system <NUM>. In some implementations, only the service provider system <NUM> stores the digital keys <NUM> for each user and management of the digital keys <NUM> is account-based, such that the user may log into their account at the service provider system <NUM> to manage shared keys.

As further illustrated, the user-A <NUM> has shared a first key A1 with the device-B <NUM> (illustrated by shared keys <NUM>) and a second key A2 with the device-C <NUM> (illustrated by shared keys <NUM>). Similarly, the device-B <NUM> has shared a key B <NUM> with the device-C <NUM> but has not provided any keys to the device-A <NUM>. In the illustrated example, the device-C <NUM> does not own any keys because the user-C <NUM> has not chosen to store a user-specific dataset at the service provider system <NUM> and, thus, no digital keys have been generated for the device-C <NUM>. Alternatively, the digital keys <NUM> may be stored only "in the cloud," such as at the service provider system <NUM> or other secure online storage associated with the service provider system <NUM>, and not stored at the user's devices. In such an implementation, the user can provide user credentials to log in to their account and then initiate the search query through the service provider system <NUM> based on their account. In this way, the shared digital keys <NUM> are not shared with any user devices and may therefore be kept more secure from counterfeits and copies. Further, the media collection that is searchable for a particular user may be tied to the account of that user, which protects other media collections from unauthorized searches.

Assume the user-B <NUM> wishes to search their personalized media collection-B <NUM> for images and/or videos of the user-A <NUM>. The user-B <NUM> inputs a search query to the device-B <NUM>. The device-B <NUM> then calls the API <NUM> at the service provider system <NUM> and provides the shared key A1 and an indication of the media collection-B <NUM> to the API <NUM>. If the media collection-B <NUM> has not yet been indexed, the API <NUM> can apply the machine-learned model <NUM> to the media collection-B <NUM> to generate corresponding indexed information. The service provider system <NUM> can store the corresponding indexed information in the indexed media information <NUM>, separate from other indexed information corresponding to other media collections. Keeping different users' indexed information separately stored can provide an additional level of security against unauthorized access (e.g., permitted access to a first user's indexed information gaining unauthorized access to a second user's indexed information).

The API <NUM> uses the shared key A1 to access the dataset-A <NUM> of the user-A <NUM> and compare the embeddings in the dataset-A <NUM> against the indexed media information <NUM> corresponding to the media collection-B <NUM>. If the API locates matching embeddings in the indexed media information <NUM> corresponding to the media collection-B <NUM>, the API <NUM> identifies corresponding images and/or videos in the media collection-B <NUM> and returns those images and/or videos to the device-B <NUM> as results to the search query. The results do not identify which face or person in the images and/or videos is the user-A <NUM>. Without such identifying information, the user-B <NUM> cannot build a copy of the dataset-A <NUM>. The results only indicate images and/or videos that depict the user-A <NUM>.

In some implementations, the service provider system <NUM> may use the results of the search query by the user-B <NUM> to update the dataset-A <NUM> for the user-A <NUM>, even though the user-A <NUM> may not have access to the media collection-B <NUM> of the user-B <NUM>. The larger the user-specific dataset <NUM>, the more accurate the results of the user-specific search query may be.

The user-C <NUM> may initiate a search query for images with one or both of the user-A <NUM> and the user-B <NUM> because the shared keys <NUM> A2 and B1 have been shared with the device-C <NUM>. Assume the user-C <NUM> initiates a search query for images of either the user-A <NUM> or the user-B <NUM>. The privacy-control module <NUM> provides both keys A2 and B <NUM> to the API <NUM>, and the API <NUM> identifies the corresponding datasets (e.g., the dataset-A <NUM> and the dataset-B <NUM>). The API <NUM> compares these datasets to the indexed media information <NUM> corresponding to the media collection-C <NUM> of the user-C <NUM>. Based on matching embeddings, the service provider system <NUM> returns search results identifying images, from the media collection-C <NUM>, that show one or both of the user-A <NUM> and the user-B <NUM>.

Notice that the user-B <NUM> has not shared a digital key (e.g., one of the owned keys <NUM>) with the user-A <NUM>. As a result, the user-A <NUM> is not permitted to use the dataset-B <NUM> for search queries. More specifically, if the user-A <NUM> initiates a search query for images of the user-B <NUM> in the media collection-A <NUM>, the service provider system <NUM> does not return any results. Because the device-A <NUM> did not provide the appropriate digital key to the API <NUM>, the API <NUM> is not able to identify which user-specific dataset <NUM> to use for comparison with the indexed media information <NUM>.

Similarly, if the user-A <NUM> chooses to revoke access to the dataset-A <NUM> from the user-C <NUM>, the user-A <NUM> can enter a command to cause the privacy-control module <NUM> to delete the previously-shared key A2 at the service provider system <NUM> (and at the device-A <NUM>). Then, when the device-C <NUM> performs a subsequent query search, using the key A2, of the media collection-C <NUM> for images or videos showing the user-A <NUM>, the service provider system <NUM> returns no results, regardless of whether similar search queries previously provided results. Accordingly, the device-C is essentially blocked from performing a user-specific search for images or videos depicting the user-A <NUM>.

Further in the illustrated example, the user-C <NUM> has not chosen to store a user-specific dataset <NUM> at the service provider system <NUM>. Therefore, neither the device-A <NUM> nor the device-B <NUM> can obtain results of a search query for images and/or videos of the user-C <NUM>.

Continuing with the example illustrated in <FIG>, <FIG> depicts example search results of user-specific search queries, according to the techniques described herein. In <FIG>, an index <NUM> includes example indexed information (e.g., indexed media information <NUM>), which indicates that an image A <NUM> contains the user-A <NUM> and the user-B <NUM>, an image B contains the user-B <NUM>, and a video C <NUM> includes the user-A <NUM> in frames X-Y and the user-B <NUM> in frames Q-Z.

The user-C <NUM> of the device-C <NUM> enters a search query in a search bar <NUM> for "images and/or videos having user-A. " The device-C <NUM> transmits the search query to the API <NUM> along with the appropriate shared keys (e.g., the key A2 shown in the shared keys <NUM> of <FIG>). The API <NUM> uses the shared key A2 to identify the corresponding user-specific dataset <NUM> (e.g., the dataset-A <NUM>) to compare with the index <NUM>. Based on this comparison, the API <NUM> returns search results <NUM> including the image A <NUM> and the video C <NUM>.

In another example, the user-A <NUM> of the device-A <NUM> enters a search query in a search bar <NUM> for "videos having user-B. " Here, no results are provided in search pane <NUM> because the device-A <NUM> cannot provide, with the search query, a shared key corresponding to the dataset-B <NUM> of the user-B <NUM>, such as a copy of one of the owned keys <NUM> of the user-B <NUM>.

<FIG> depict example methods <NUM>, <NUM>, <NUM>, and <NUM> for controlling privacy for sharing embeddings for searching and indexing media content. These methods can be performed by the service provider system <NUM>, which uses the search manager module <NUM> to control access to searching an image or video corpus using a user-specific query. <FIG> depicts a method of indexing media content based on a user-specific dataset. <FIG> describes additional details of the method <NUM> in <FIG>, which include a method <NUM> for controlling access to the media content for user-specific search queries. <FIG> describes additional details of the method <NUM> in <FIG>, which include a method <NUM> of updating the user-specific dataset based on additional data. <FIG> describes additional details of the method <NUM> in <FIG>, which include method <NUM> of updating the user-specific dataset based on data removal.

The methods <NUM>, <NUM>, <NUM>, and <NUM> are shown as sets of blocks that specify operations performed but are not necessarily limited to the order or combinations shown for performing the operations by the respective blocks. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and/or alternate methods. In portions of the following discussion, reference may be made to the example operating environment <NUM> of <FIG> or to entities or processes as detailed in <FIG>, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.

At <NUM>, a service provider system obtains a set of images of a first user's face. For example, the service provider system <NUM> can obtain the set of images from the electronic device <NUM>.

At <NUM>, the service provider system applies a machine-learned model to the set of images to generate a user-specific dataset of face embeddings for the first user. For example, the service provider system <NUM> can apply the machine-learned model <NUM> to the set of images to generate the user-specific dataset <NUM> for the user of the electronic device <NUM>.

At <NUM>, the service provider system indexes media content stored in a media storage by applying the machine-learned model to the media content to provide indexed information identifying one or more faces shown in the media content. For example, the service provider system <NUM> can index the media storage <NUM> stored at the service provider system <NUM> by applying the user-specific dataset <NUM> to media content in the media storage <NUM> and identifying images or videos that depict the user of the electronic device <NUM>, which provides the indexed media information <NUM>.

At <NUM>, the service provider system controls access to the indexed information by a second user querying the media content for images or videos depicting the first user. For example, the service provider system <NUM> can implement the search manager module <NUM> for controlling access to the indexed media information <NUM>. The search manager module includes the API <NUM>, which acts as an intermediary between a user device (e.g., the electronic device <NUM>, the user device <NUM>) and the secure data stored at the service provider system <NUM>. If the second user provides an authorized digital key, the API <NUM> accesses the indexed media information <NUM> and, using the user-specific dataset <NUM> corresponding to the digital key, identifies matching embeddings in the indexed media information <NUM>, which indicate particular images or videos in the media storage <NUM> that depict the first user. The method <NUM> may optionally proceed to any one of the methods <NUM>, <NUM>, or <NUM> described with respect to <FIG>, respectively.

<FIG> describes additional details of the method <NUM> depicted in <FIG>, which include a method <NUM> of controlling access to the media content for user-specific search queries. At <NUM>, a service provider system receives a search query for images or videos from media content depicting a first user. For example, the service provider system <NUM> receives a search query from the requesting user device <NUM> for images or videos depicting the first user of the electronic device <NUM>. The search query includes a digital key that was previously shared with a second user of the requesting user device <NUM> by the first user of the electronic device <NUM>. The search query also includes an indication of a particular media storage (e.g., personalized media collection in the media storage <NUM>, the media storage <NUM> at the media storage service <NUM>, local storage at the user device <NUM>).

At <NUM>, the service provider system determines, using a search manager module having an application programming interface (API), whether the search query has an authorized digital key. For example, the API <NUM> determines whether the digital key provided in the search query matches one of the digital keys <NUM> associated with the first user.

If the API determines that the digital key is not authorized ("NO" at <NUM>), then at <NUM> the service provider system returns no results to the search query. For example, the digital key <NUM> may have been deleted based on input from the first user, which results in the digital key <NUM> no longer being authorized for the search. In some cases, the search query may not include a digital key.

If the API determines that the digital key is authorized ("YES" at <NUM>), then at <NUM>, the API identifies the user-specific dataset based on the digital key. For example, the API <NUM> uses the digital key <NUM> to locate the user-specific dataset <NUM> for the first user, which provides an indication as to which face to search for in the media storage <NUM>.

At <NUM>, the API accesses the indexed information to identify which of the images or videos from the media content depict the first user based on the user-specific dataset. For example, the API <NUM> uses the face embeddings in the user-specific dataset <NUM> for comparison to the indexed media information <NUM>. The indexed media information <NUM> includes embeddings associated with faces in the media storage <NUM>. Accordingly, embeddings in the indexed media information <NUM> that match the face embeddings in the user-specific dataset <NUM> direct the API <NUM> to images or videos in the media storage that depict the first user.

At <NUM>, the service provider system provides search results including the identified images or videos from the media content that depict the first user. For example, the service provider system <NUM> transmits, to the requesting user device <NUM>, the identified images or videos depicting the first user as search results for the search query.

<FIG> describes additional details of the method <NUM> in <FIG>, which include a method <NUM> of updating the user-specific dataset based on additional data. At <NUM>, the service provider system receives one or more additional images from an electronic device of the first user. For example, the electronic device <NUM> may upload one or more new images that depict the user's face, such as a close-up photo or a full-body photo.

At <NUM>, the service provider system applies the machine-learned model to the one or more additional images to generate one or more new face embeddings. At <NUM>, the service provider system adds the one or more new face embeddings to the user-specific dataset of face embeddings for the first user to update the user-specific dataset. Additional embeddings can improve the user-specific dataset to enable more accurate results (e.g., more accurate identification of the user in the images or videos) than with less embeddings. The method <NUM> then returns to <NUM> in <FIG> to control access to the indexed information.

<FIG> describes additional details of the method <NUM> in <FIG>, which include a method <NUM> of updating the user-specific dataset based on data removal. At <NUM>, the service provider system receives input from the first user to delete one or more images from the set of images previously used to generate the user-specific dataset. For example, the first user may select to remove or delete one or more of the images that were used to generate the user-specific dataset <NUM>. The user may dislike a specific image or the image may be an old image that is no longer a good representation of the user's current facial features.

At <NUM>, the service provider system deletes, based on the input from the first user, the one or more images from the set of images to provide a subset of images. For example, the storage service module <NUM> of the service provider system <NUM> deletes images selected by the user for removal.

At <NUM>, the service provider system updates the user-specific dataset of face embeddings for the first user by applying the machine-learned model to the subset of images to provide an updated user-specific dataset. To update the user-specific dataset <NUM> when the set of images is reduced, the storage service module <NUM> can recreate the user-specific dataset <NUM>. Alternatively, the storage service module <NUM> can delete an embedding, from the user-specific dataset <NUM>, that corresponds to the deleted image. For multiple deleted images, multiple corresponding embeddings can be deleted from the user-specific dataset <NUM>. The method <NUM> then returns to <NUM> in <FIG> to control access to the indexed information. In at least some aspects, the methods <NUM> and <NUM> can be combined such that the user-specific dataset can be updated based on both the addition and removal of images used to generate the user-specific dataset.

<FIG> depicts an example method <NUM> for controlling privacy for sharing embeddings for searching and indexing media content. The method <NUM> can be performed by the electronic device <NUM>, which uses the privacy-control module <NUM> to control access to searching an image or video corpus using a user-specific query.

The method <NUM> is shown as a set of blocks that specify operations performed but are not necessarily limited to the order or combinations shown for performing the operations by the respective blocks. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and/or alternate methods. In portions of the following discussion, reference may be made to the example operating environment <NUM> of <FIG> or to entities or processes as detailed in <FIG>, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.

At <NUM>, an electronic device captures a set of images of a first user of the electronic device. For example, the first user of the electronic device <NUM> may capture images (e.g., "selfies") using the camera system <NUM>. The images may be captured using a color camera of the camera system <NUM>. Alternatively, the images can be captured using a near-infrared camera of the camera system <NUM>, such as during a face authentication process of the authentication system <NUM>.

According to one option at <NUM>, the electronic device transmits the set of images to a service provider system for cloud storage and generation of a user-specific dataset of face embeddings for the first user. For example, the electronic device <NUM> uploads the set of images to the service provider system <NUM> for secure storage in the media storage <NUM>.

As an alternative to transmitting the set of images to the service provider system at <NUM>, at <NUM> the electronic device may optionally apply a machine-learned model to the set of images to generate a user-specific dataset of face embeddings for the first user. For example, the electronic device <NUM> may apply the machine-learned model <NUM>, which is stored at the electronic device <NUM>, to the set of images. The machine-learned model <NUM> may be part of a face authentication system (e.g., authentication system <NUM>) used to create face embeddings from captured images of the user to authenticate the user against enrolled embeddings to unlock the electronic device <NUM>. The machine-learned model <NUM> can generate the user-specific dataset <NUM> of face embeddings based on the set of images.

At <NUM>, the electronic device transmits the user-specific dataset to a service provider system for cloud storage. For example, the electronic device <NUM> may transmit the user-specific dataset <NUM>, rather than the set of images, to the service provider system <NUM>.

At <NUM>, whether proceeding from <NUM> or <NUM>, the electronic device shares a digital key associated with the user-specific dataset with a second user to enable the second user to call a user-specific image search at the service provider system for images or videos depicting the first user. The user of the electronic device <NUM> can share a digital key <NUM> with a friend to enable the friend to search the media storage <NUM> for images or videos depicting the user. For each friend of the user, a different digital key is provided.

At <NUM>, the electronic device revokes access to the user-specific image search by requesting that the service provider system delete the digital key previously shared with the second user. For example, the user of the electronic device <NUM> can revoke the access previously provided to the friend by deleting the digital key that was shared with the friend. Consequently, the friend is no longer authorized to conduct a search using the user-specific dataset <NUM> of the user to find images or videos depicting the user.

Generally, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods may be described in the general context of executable instructions stored on computer-readable storage memory that is local and/or remote to a computer processing system, and implementations can include software applications, programs, functions, and the like. Alternatively or in addition, any of the functionality described herein can be performed, at least in part, by one or more hardware logic components, such as, and without limitation, Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SoCs), Complex Programmable Logic Devices (CPLDs), and the like.

<FIG> illustrates various components of an example computing system <NUM> that can be implemented as any type of client, server, and/or electronic device as described with reference to the previous <FIG> to implement privacy controls for sharing embeddings for searching and indexing media content.

The computing system <NUM> includes communication devices <NUM> that enable wired and/or wireless communication of device data <NUM> (e.g., radar data, authentication data, reference data, received data, data that is being received, data scheduled for broadcast, and data packets of the data). The device data <NUM> or other device content can include configuration settings of the device, media content stored on the device, and/or information associated with a user of the device (e.g., an identity of a person within a radar field or customized air gesture data). Media content stored on the computing system <NUM> can include any type of radar, biometric, audio, video, and/or image data. The computing system <NUM> includes one or more data inputs <NUM> via which any type of data, media content, and/or inputs can be received, such as human utterances, interactions with a radar field, touch inputs, user-selectable inputs or interactions (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and/or image data received from any content and/or data source.

The computing system <NUM> also includes communication interfaces <NUM>, which can be implemented as any one or more of a serial and/or a parallel interface, a wireless interface, any type of network interface, a modem, and as any other type of communication interface. The communication interfaces <NUM> provide a connection and/or communication links between the computing system <NUM> and a communication network by which other electronic, computing, and communication devices communicate data with the computing system <NUM>.

The computing system <NUM> includes one or more processors <NUM> (e.g., any of microprocessors, controllers, or other controllers) that can process various computer-executable instructions to control the operation of the computing system <NUM> and to enable techniques for, or in which can be implemented, privacy controls for sharing embeddings for searching and indexing media content. Alternatively or additionally, the computing system <NUM> can be implemented with any one or combination of hardware, firmware, or fixed logic circuitry that is implemented in connection with processing and control circuits, which are generally identified at <NUM>. Although not shown, the computing system <NUM> can include a system bus or data transfer system that couples the various components within the device. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures.

The computing system <NUM> also includes computer-readable media <NUM>, such as one or more memory devices that enable persistent and/or non-transitory data storage (in contrast to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of a read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and a disk storage device. A disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and/or rewriteable compact disc (CD), any type of a digital versatile disc (DVD), and the like. The computing system <NUM> can also include a mass storage media device (storage media) <NUM>.

The computer-readable media <NUM> provides data storage mechanisms to store the device data <NUM>, as well as various device applications <NUM> and any other types of information and/or data related to operational aspects of the computing system <NUM>. For example, an operating system <NUM> can be maintained as a computer application with the computer-readable media <NUM> and executed on the processors <NUM>. The device applications <NUM> may include a device manager, such as any form of a control application, software application, signal-processing and control modules, code that is native to a particular device, an abstraction module, an air gesture recognition module, and other modules. The device applications <NUM> may also include system components, engines, modules, or managers to implement privacy controls for sharing embeddings for searching and indexing media content, such as the storage service module <NUM> or the search manager module <NUM>. The computing system <NUM> may also include, or have access to, one or more machine-learning systems.

Claim 1:
A method of controlling privacy for sharing embeddings for searching and indexing media content, the method performed by a service provider system:
obtaining (<NUM>) a set of images of a face of a first user;
applying (<NUM>) a machine-learned model to the set of images to generate a user-specific dataset of face embeddings for the first user;
indexing (<NUM>) media content stored in a media storage by applying the machine-learned model to the media content to provide indexed information identifying one or more faces shown in the media content; and
controlling (<NUM>), by an application programming interface, access to the indexed information by a second user querying the media content for images or videos depicting the first user, the access controlled based on a digital key shared by the first user with the second user, the digital key associated with the user-specific dataset, the user-specific dataset usable for comparison with the indexed information to identify the images or videos in the media content that depict the first user.