Directional assistance for centering a face in a camera field of view

Methods and systems are provided for providing directional assistance to guide a user to position a camera for centering a person's face within the camera's field of view. A neural network system is trained to determine the position of the user's face relative to the center of the field of view as captured by an input image. The neural network system is trained using training input images that are generated by cropping different regions of initial training images. Each initial image is used to create a plurality of different training input images, and directional assistance labels used to train the network may be assigned to each training input image based on how the image is cropped. Once trained, the neural network system determines a position of the user's face, and automatically provides a non-visual prompt indicating how to center the face within the field of view.

BACKGROUND

Many users look at a display screen showing a feed from a digital camera to determine whether the subject of a desired image or video is centered or even within the camera's field of view. Mobile devices now have front-facing cameras to allow a user to direct the camera at himself or herself while using the display screen to ensure the camera is capturing the user as desired. Users with visual impairments such as blurred vision or partial or complete blindness, however, cannot use the display screen to see what is being captured by the camera or determine how to adjust the camera's position to correctly capture the desired subject. Solutions exist that analyze facial features to determine whether a face is centered and what percentage of the face is being captured. But these solutions do not provide feedback to guide the user to the appropriate camera position. Additionally, these existing solutions fail to detect a subject when the subject's face is not within the field of view, and thus, do not work when the subject's face is out of view.

SUMMARY

Embodiments of the present invention are directed towards a system trained to provide directional assistance to camera users to guide the user to position the camera for centering a face within the camera's field of view. In accordance with embodiments of the present invention, such a system can be created using one or more neural networks. The one or more neural networks are trained to assist in providing directional assistance by determining a current position of a person's face within the field of view. For instance, a neural network may be trained to assign a frame to a particular category corresponding to a directional assistance prompt based on the current position of the person's face within the camera's field of view. In exemplary embodiments, the network is trained to classify image data into one of five directional assistance categories where four categories correspond to prompts directing the user to adjust the camera or the subject and one category corresponds to a prompt indicating a face is centered within the field of view.

Training of the neural network system is accomplished using a training data set of images. Each image in the training dataset is labeled with one of the categories corresponding to a directional assistance. The training dataset is used to train the neural network to assign a new input image to one of the categories. In exemplary embodiments, the training dataset comprises cropped images that are created from a smaller set of images, and the labels are assigned based on how the original image is cropped to form the particular training image. For instance, where there are five categories for providing directional assistance, an original image within the smaller set of images may be copied to create five training images, with a different portion of each copy being cropped to create each training image. The training images are input into the neural network system to generate directional assistance output, and the labels for each training image are compared to the directional assistance output for that image to determine errors, which are used to adjust the system to avoid similar errors in future iterations. Upon completion of training, the system may be used to determine a position of a subject's face within an unknown input image and automatically generate a non-visual directional assistance prompt to guide a user to center a face within a camera's field of view.

DETAILED DESCRIPTION

Various terms are used throughout this description. Definitions of some terms are included below to provide a clearer understanding of the ideas disclosed herein:

The term “image data” is used herein to refer to data captured from a camera. Image data may comprise video files or one or more photographs, including LDR and HDR image files. Image data may also be used to refer to a frame extracted from a video, including a recorded video or a video feed. Image data may comprise data captured by a camera and displayed on a user device in real time even if the data is not stored.

The term “training image” herein refers image data that is used for training a neural network in accordance with embodiments of the disclosure. The training image depicts at least part of a person and may depict all or part of a person's face. The training image may be a “cropped training image” or an “uncropped training image”.

The term “uncropped training image” herein refers to an image that is cropped to create one or more training images. An uncropped training image is also referred to herein as an “original image” and an “initial image”.

The term “cropped training image” herein refers to a training image that has been created by cropping another image and is used to train a neural network.

The term “field of view” herein refers to the entire view of an environment that is visible through a camera at a particular position and orientation is space. The field of view comprises what is captured in any given image. The field of view may also be referred to as “angle of view”.

The term “directional assistance category” herein refers to a category of image data that indicates the position of a person's face within the field of view captured in the image data. The directional assistance category may indicate whether the face, or at least a center portion of the face, is centered within the field of view, to the right of the center of the field of view, to the left of the center of the field of view, above the center of the field of view, or below the center of the field of view. Image data may be assigned to a directional assistance category even if all or substantially all of the face is not within the field of view.

The term “directional assistance label” herein refers to a reference label associated with a training image when training the neural network. Each directional assistance label may correspond to a directional assistance category and may be automatically assigned to a training image based on how the region of the initial image that is cropped to form the training image.

Embodiments of the present disclosure are directed towards providing directional assistance to camera users to guide the user to center a face within the camera's field of view. Typically, users look at a display screen of a camera or a device with a built-in camera to see whether the subject of an image or video is centered or even within the field of view of the camera. Front-facing cameras are now provided on many mobile devices so that the camera and display screen are facing the same direction, enabling self-portrait photographs and videos, which are referred to as “selfies”. Using the front-facing camera and display screen together allows a user to take an image or video of himself or herself while ensuring the user is centered or at least within the field of view of the camera. In this way, users can take selfies for personal use or to use as a live photograph when preparing an electronic form and can participate in video conferencing, for example.

However, users with visual impairments, such as blurred vision or partial or complete blindness, cannot use the display screen to see what is being captured by the camera or to determine how to adjust the position of the camera to correctly capture the desired subject. This limitation is present when taking selfies or video conferencing with a front-facing camera as well as when taking images with a rear-facing camera (i.e., a camera facing away from a display screen). Current accessibility features on mobile devices with cameras can determine the number of people within a frame, if the faces are centered, and what percentage of the screen is occupied by the face. These existing solutions recognize particular facial features, such as a person's eyes to determine whether the face is entered. However, by relying on select facial features to determine whether a face is centered, these programs cannot provide assistance when a face is completely or almost completely out of the camera's field of view. Additionally, the outputs of these solutions do not direct the user to the appropriate camera position. For instance, while an output indicating a face is “not centered” tells the user that the camera position needs to be adjusted, the user does not know how to adjust the camera to center the face.

Accordingly, embodiments of the present invention are directed towards a system trained to provide directional assistance to camera users to guide the user to position a camera for centering a face within the camera's field of view. The system uses one or more neural networks trained to provide directional assistance by determining a current position of a user within the field of view. For instance, a neural network is trained to assign a directional assistance category to a frame extracted from image data received by a camera. The directional assistance category corresponds to a directional assistance prompt that is output to a user based on the current position of a person, such as the user, within the field of view of the camera. In an example embodiment, the network is trained to assign a frame within received image data to one of five categories where four categories correspond to prompts with directions to adjust the camera position or the person and one category corresponds to a prompt indicating the face is centered within the field of view of the camera.

Training the neural network system is accomplished using a training dataset. Each image in the training dataset is associated with a directional assistance label representing a directional assistance category. The training dataset is used to train the neural network to determine a position of a face relative to the center of an input image by assigning the image to one of the categories. The training dataset comprises images with faces of individuals in different positions within the camera's field of view, such as being centered, to the right side, to the left side, to the top side, and to the bottom side. Further, some training images depict an individual in which the individuals' face is completely or almost completely cut off from the camera's field of view. By training the neural network with images in which the face is completely or almost completely cut off from view, directional assistance may be provided to guide a user even when the user's face is completely or almost completely out of the frame.

In some aspects, the training dataset comprises cropped images that are created from a smaller set of images and that are associated with directional assistance labels based on how the original image is cropped to generate the particular training image. For instance, where there are five categories for providing directional assistance, an original image from the smaller set of images is duplicated to create five copies of the images, and a different portion of each copy is cropped to create five different training images. The training images are input into the neural network system to output directional assistance categories, and the label associated with each training image is compared to the output for that image to determine any errors, which are used to adjust the system to avoid similar errors in future iterations.

Upon completion of training, the system may be used to determine a position of a person's face within a new input image, such as a frame extracted from a camera's video feed, and automatically generate a directional assistance prompt to guide a user to center a person's face within the camera's field of view. In exemplary embodiments, the directional assistance prompt comprises an audio output either with a direction for moving the camera or subject of the image or indicating a face is centered. For example, the trained system may provide an audio output of “Move Right” when the trained neural network detects that person's face is offset to the right of the center of the frame. When the image data is being continually received, such as in a video feed, the system may continue to provide directional assistance prompts based on additional frames until a frame is detected as showing the face is centered and a corresponding directional assistance prompt is given.

Example Computing Environment

It should be understood that environment100shown inFIG. 1is an example of one suitable operating environment. Among other components not shown, environment100includes a number of user devices, such as user devices102aand102bthrough102n, network104, and server(s)108. Each of the components shown inFIG. 1may be implemented via any type of computing device, such as one or more of computing device1100described in connection toFIG. 11, for example. These components may communicate with each other via network104, which may be wired, wireless, or both. Network104may include multiple networks, or a network of networks, but is shown in simple form so as not to obscure aspects of the present disclosure. By way of example, network104can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet, and/or one or more private networks. Where network104includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) may provide wireless connectivity. Networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. Accordingly, network104is not described in significant detail.

It should be understood that any number of user devices, servers, and other components may be employed within environment100within the scope of the present disclosure. Each may comprise a single device or multiple devices cooperating in a distributed environment.

User devices102athrough102nmay be any type of computing device capable of being operated by a user. For example, in some implementations, user devices102athrough102nare the type of computing device described in relation toFIG. 11. By way of example and not limitation, a user device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a camera, a wearable computer, a personal digital assistant (PDA), an MP3 player, a global positioning system (GPS) or device, a video player, a handheld communications device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, any combination of these delineated devices, or any other suitable device with a built-in camera or capable of being connected to a camera.

User devices102athrough102nmay include one or more processors and one or more computer-storage media. The computer-storage media may include computer-readable instructions executable by the one or more processors. The instructions may be embodied by one or more applications, such as application110shown inFIG. 1. Application110is referred to as a single application for simplicity, but its functionality can be embodied by one or more applications in practice. As indicated above, the other user devices can include one or more applications similar to application110.

Application110may generally be any application capable of facilitating the exchange of information between user devices102athrough102nand the server(s)108in carrying out steps for providing directional camera assistance based on image data from the user devices102through102n. In some implementations, application110comprises a web application that can run in a web browser and could be hosted at least partially on the server-side of environment100. In addition, or instead, application110may comprise a dedicated application, such as an application having image processing functionality. In some instance, application110may be an application that provides for image editing, such as the Adobe Capture application, or video conferences, such as the Adobe Connect application. In another instance, application110may be an application dedicated to creating, managing, publishing, and/or updating electronic forms, such as the Adobe Experience Manager application or the Adobe Fill & Sign application. In some cases, application110is integrated into the operating system (e.g., as a service). It is, therefore, contemplated herein that “application” be interpreted broadly.

In accordance with embodiments herein, application110can facilitate providing directional assistance, using an input image, to guide a user in centering a face within a camera's field of view. To provide directional camera assistance, image data, such as an image captured by a camera, is received, and directional assistance is provided to the user through a prompt, such as an audio instruction, corresponding to a directional assistance category assigned to the image data. The image data may be received directly from a camera112integrated into or connected to user device102a. Once a directional assistance category is assigned to the image data, a directional assistance prompt may be automatically generated and communicated through user device102a, for example. In some instances, camera112captures and provides a live video feed such that the directional assistance category is assigned and output is provided in real time. Additionally, when the image data is a live video feed, this process may be repeated on later received frames until the input frame is assigned a category corresponding to a centered face and a directional assistance prompt is given acknowledging the centered face.

As described herein, server108can facilitate providing directional assistance via directional assistance manager106. Server108includes one or more processors, and one or more computer-storage media. The computer-storage media includes computer-readable instructions executable by the one or more processors. The instructions may optionally implement one or more components of directional assistance manager106, described in additional detail below.

Directional assistance manager106trains and operates a neural network system to provide directional assistance for centering a face within a camera's field of view. The neural network system trained and operated by the directional assistance manager106may be comprised of one or more neural networks trained to generate designated output. For example, the neural network system may comprise a neural network trained to assign a directional assistance category to an input image.

At a high level, directional assistance manager106trains a neural network system to determine a position of a face within the field of view depicted in input image data and initiate a prompt based on the position. The input image data contains at least a portion of an individual who is an intended subject of an image or video. Analyzing the pixels of the input frame, the neural network system determines a probability of the image data belonging to one or more of the directional assistance categories, and based on an assigned directional assistance category, at least one non-visual directional assistance prompt is generated.

Prior to providing directional assistance based on new image data, the neural network system is trained using input images, referred to herein as training input images. Each training input image is labeled with one of the directional assistance categories. The training dataset comprises images with individuals in various different positions relative to the camera's field of view, such as being centered, to the right side, to the left side, to the top side, and to the bottom side. Further, some training images depict an individual in which the individuals' face is completely or substantially completely cut off from the camera's field of view. As described further with respect toFIGS. 4A-4E, in some embodiments, the training dataset comprise images cropped from a smaller set of images, and directional assistance category labels are assigned based on how the original image is cropped to from the particular training image. The training images are input into the neural network system to output directional assistance categories, which are compared to the directional assistance category labels associated with the input training images. The neural network system may be modified or adjusted based on the comparisons between the training output and the references such that the quality of subsequently generated output increases.

For cloud-based implementations, the instructions on server108may implement one or more components of directional assistance manager106, and application110may be utilized by a user to interface with the functionality implemented on server(s)108. In some cases, application110comprises a web browser. In other cases, server108may not be required. For example, the components of directional assistance manager106may be implemented completely on a user device, such as user device102a. In this case, directional assistance manager106may be embodied at least partially by the instructions corresponding to application110and may be provided as an add-on or plug-in to application110. Thus, it should be appreciated that directional assistance manager106may be provided via multiple devices arranged in a distributed environment that collectively provide the functionality described herein. Additionally, other components not shown may also be included within the distributed environment. In addition, or alternatively, directional assistance manager106may be integrated, at least partially, into a user device, such as user device102a. Furthermore, directional assistance manager106may at least partially be embodied as a cloud computing service.

Referring toFIG. 2, aspects of an illustrative directional assistance system200are shown, in accordance with various embodiments of the present disclosure. Directional assistance manager202may include training engine204, directional assistance engine212, and data store218. The foregoing components of directional assistance manager202may be implemented, for example, in operating environment100ofFIG. 1. In particular, those components may be integrated into any suitable combination of user devices102aand102bthrough102n, and server(s)108, including directional assistance manager106.

Data store218may store computer instructions (e.g., software program instructions, routines, or services), data, and/or models used in embodiments described herein. In some implementations, data store218stores information or data received via the various components of directional assistance manager202and provides the various components with access to that information or data as needed. Although depicted as a single component, data store218may be embodied as one or more data stores. Further, the information in data store218may be distributed in any suitable manner across one or more data stores for storage (which may be hosted externally or internally).

In embodiments, data stored in data store218includes training data216. Training data generally refers to data used to train a neural network, or portion thereof. As such, training data216can include references (such as directional assistance category labels), training images, cropped training input images, uncropped training images, any transformed form of training images created in training the neural network (such as gray scale versions of training images), and output category labels. In some cases, directional assistance manager202receives data from user devices (e.g., an input image received by user device102aor another device associated with a user, via, for example, application110). In other cases, data is received from one or more data stores in the cloud. Data store218may also be used to store neural network system214comprising one or more neural networks.

Training engine204may be used to train neural network system214to determine a position of a user's face within the frame of view of an input image such as by assigning each input image to a directional assistance category. As depicted inFIG. 2, training engine204includes a training image component206and a training component208. Although these components are illustrated separately, it can be appreciated that the functionality described in association therewith can be performed by any number of components.

Training image component206generates training input images that are to be fed into the neural network system214for training purposes. The training input images comprise images of individuals. In exemplary aspects, each training image depicts one person. The training image captures at least a portion of the individual, including the individual's face, a portion of the individual's face, or none of the individual's face. In some embodiments, each training image depicts multiple people with each person having all or part of the face within the training image. Further, the training input images may comprise LDR image types, such as a JPEG, or HDR images. The input images may be frames extracted from a video, including a recorded video stored in a database or a live video feed.

In implementations, the training image component206extracts the training input images from uncropped training images that each depict an individual's entire face. In this way, the training input images may be a portion of and cropped from an uncropped training image. Accordingly, “uncropped training image”, as used herein, refers to an image prior to cropping for purposes of generating training input images. However, it is contemplated that the “uncropped training image” may have been cropped prior to further cropping for creating the training input images. Uncropped training image may also be referred to herein as an original training image or an initial training image.

In aspects, each uncropped training image is used to generate a plurality of training input images wherein the number of training input images generated is based on the number of directional assistance categories. For instance, in exemplary aspects, five training input images are created from one uncropped training image. Accordingly, an uncropped training image may be copied four times to create five copies, and each copy of the uncropped training image is cropped in a different region to create five different training input images. Each training image created from the same uncropped training image corresponds to one of the directional assistance categories.

FIG. 3depicts an example uncropped training image300, andFIGS. 4A-4Edepict training input images that are created from different crops of the uncropped training image300. Each of the training input images ofFIGS. 4A-4Eare created from a different region of the uncropped training image300, and a directional assistance label, which is also referred to herein as a reference label, is assigned to each training input image based on the particular region cropped. For example, training input image410ofFIG. 4Ais a crop of the upper portion of uncropped training image300. The lower portion of uncropped training image300is cropped out of training input image410, resulting in the center of the face in the image to be lower than the center of the training input image410. Training input image410is labeled “Go Down,” indicating that the camera should be moved or tilted downwards to center the face within the image. Training input image420ofFIG. 4Bis a crop of the lower portion of uncropped training image300and is labeled “Go Up,” indicating that the camera should be moved or tilted upwards to center the face within the image. Similarly, training input images430and440ofFIGS. 4C and 4D, respectively, are crops of the left portion and the right portion, respectively, of the uncropped training image300. Training input image430is labeled “Go Right,” and training input image440is labeled “Go Left.”FIG. 4Eillustrates training input image450that is formed from a center portion of the uncropped training image300, and the face captured in training input image450is substantially centered within training input image450such that the center of the person's face is substantially aligned with the center of the training input image450. As used herein, a face is “substantially” centered when the center of the face unaligned with the center of the image by not more than 10%. Alternatively, in some embodiments, a face is “substantially” centered when a bounding box defining the face is entirely within the field of view of the image. Training input image450is labeled “Perfect”, indicating that face is centered and the camera does not need to be moved. In the training input images inFIGS. 4A-4D, a portion of the face is cropped out; however, it is contemplated that training images corresponding to “Go Right”, “Go Left”, “Go Up”, and/or “Go Down” labels may capture a person's entire face but the face is not centered. Similarly, the training images may not comprise any of the person's face but, instead, depict non-facial features of the person, such as the person's hair, ears, shoulder, neck, and the like.

The directional assistance categories illustrated inFIGS. 4A-4Dcorrespond to how a camera should be moved so that the face is within the center of the camera's field of view. It is also contemplated that the directional assistance categories (and, consequently, the output prompts) are based on how a subject of the image should be moved. For instance, when a subject's face is to the right of the center of the field of view, a “Move left” category may be assigned and corresponding prompt may be initiated to indicate that the user should move left to center the subject's face. This prompt may be appropriate for indicating movement of the subject when the subject is the user using a front-facing camera or when a rear-facing camera is being used to capture another person's face. Further, as previously mentioned, at least some of the training images within the training data set depict an individual's face being completely or almost completely out of the camera's field of view.

FIGS. 5A-5Dillustrate a training input image that is created in accordance with an embodiment.FIG. 5Aillustrates an uncropped training image500similar to the uncropped training image300ofFIG. 3. Cropping the uncropped training image500utilizes a crop ratio (a), which is the ratio of length to be cropped out to the total side length, and the length of a resulting cropped image is equal to 1−α of the length of the uncropped training image500. As such, each resulting training image may have a width and a height equal to 1−α; however, the region of the uncropped training image500to which the crop ratio is applied differs for each resulting training image.FIGS. 5B-5Dillustrate a cropped training image corresponding to a “Go Right” prompt such that the cropped training image is a crop of a left-side portion of the uncropped training image500. Accordingly, the 1−α width is measured from the left side of the uncropped training image500, resulting in the length of a being cropped off from the right side. The 1−α height of the resulting training image is centered along a vertical axis of the uncropped training images500such that a bottom α/2 length and a top α/2 length are cropped, as shown inFIG. 5C. By centering 1−α for the height of a “Go Right” picture, the system may be indifferent to height. Lastly, the image is resized to the original size of the uncropped training image500to create the “Go Right” training input image510, similar to training input image430ofFIG. 4C. In some embodiments, the image is resized to 150×150 pixels.

Similarly, for a “Go Left” training input image, the length 1−α extends from a right side of the image so that a left-side portion of the image is cropped off. For “Go Up” and “Go Down” images, the 1−α length extends from one side of the vertical axis. For instance, for “Go Up”, the 1−α length extends from the bottom of the image, thereby cropping off a top portion, and for “Go Down”, the 1−α extends from the top of the image, cropping off a bottom portion. For “Go Up” and “Go Down” images, the 1−α length for the width is measured in the center of image such that a right α/2 portion and a left α/2 portion are cropped off.

Generally, the greater amount of training data available, the more accurate the neural network can become. Accordingly, some aspects of the training image component206perform data augmentation, including modifying existing training images to create additional training images. Data augmentation techniques may include rotating images, mirroring images, changing color contrast levels, adjusting the overall brightness, and the like. In embodiments in which training input images are cropped images created from an uncropped training image, as described with respect toFIGS. 3-5E, data augmentation may occur prior to or after creating the cropped training input images.

In some embodiments, training input images are created in accordance with this method, and the images are divided into a training set and a validation set. In one embodiment reduced to practice, approximately 2000 images were generated for each directional assistance category, totally 10,000 images. A validation set included approximately 500 images (100 images per category), and the training data set comprised the remaining 9,500 images. After running the model for 15 epochs with a batch size of 10 images (for a total of 935 batches of training data), an accuracy of approximately 98% was achieved on the training data set and approximately 95% was achieved on the validation set.

As previously mentioned, when training the one or more neural networks in the neural network system, the training output that is generated is compared to references. As used herein, a reference refers to a standard, or ground truth, for evaluating the quality of the output generated by a neural network during training. In exemplary implementations, the training output includes a directional assistance category indicating a position of a person's face within the captured field of view. Accordingly, the references for each training input image is a directional assistance label identifying the directional assistance category to which the training input image properly belongs. In exemplary aspects, the directional assistance label (also referred to herein as a reference label) for a training input image is automatically assigned when the training input image is created. The reference label is based on the portion of the original uncropped training image that is used to generate the training input image. For instance, because training image510ofFIG. 5Dis a crop of the left portion of the uncropped training image500such that the right side is cropped out, a “Go right” reference label is assigned to the training image510. In exemplary aspects, the training images are generated through an automatic cropping process for creating different crops each associated with a directional assistance category, and the references labels are automatically assigned to the resulting training images based on the directional assistance category associated with the particular crop. In alternative implementations, reference labels are manually assigned. A reference label may be stored as metadata for a particular training input image, and a referential table may be updated to reflect the reference label that is associated with the training input image.

Training component208uses training input images generated by the training image component206for training a neural network system. From a training input image, the neural network system generates a directional assistance output indicating a position of a person's face within the field of view, and the output is compared to a corresponding reference label. Based on this comparison, the training component208may adjust or modify the neural network system so that the neural network system becomes more accurate, and this process may be repeated for each training input image. The process of training the neural network system is discussed further with respect toFIG. 9.

The neural network system trained according to the present disclosure may be used to determine a position of a user's face within the field of view of new images input into the system. Providing directional assistance may be performed by the directional assistance engine212using the neural network system214. Image data is provided to the directional assistance engine212to generate one or more directional assistance outputs corresponding to directional assistance categories assigned to image data. For example, select input frames from a live video feed from a camera may be automatically fed into the neural network system214to assign a directional assistance category to each frame. As used herein, an input frame generally refers to one or more frames extracted from a video, including a live video feed, or portion thereof. The input frame may comprise a LDR image, such as a JPEG image file, or may include a HDR image. The image data, such as the image frame, may depict at least a portion of an individual who is an intended subject of an image or video. The image data may include all of the individual's face, a portion of the individual's face, or none of the individual's face.

An audio or other non-visual output may be automatically generated for each determined directional assistance category, such as “Go Right” or “Perfect”. The output indicating the directional assistance category may be provided in real time to allow a user to make the appropriate adjustments to the camera position and/or the position of the intended subject of the image, which may be the user's self.FIGS. 6A-6Cdepict directional assistance outputs being generated in accordance with embodiments of the present invention. InFIGS. 6A-6C, an individual is taking a selfie using a mobile device602with a front-facing camera604designed to capture images from the same side of the display screen606of the mobile device602.FIG. 6Adepicts the individual's face offset to the right side of the camera's field of view. A frame from the image data received from the camera is assigned, using a trained neural network system, a “Move Right” directional assistance category, which is associated with images in which the subject's face is off-centered towards the right side. A first audio prompt608of the phrase “Move Right” is automatically generated to instruct the user to move the device602to the right or to turn the device to the right. After the user makes an adjustment, the user's face may still not be centered, and additional prompts may be given.FIG. 6B, for instance, shows the user moved the camera too much to the right in response to the first prompt608, resulting in the user's face to be off-centered to the left side of the camera's field of view. Accordingly, a new directional assistance category (i.e., “Move Left”), and a corresponding second audio prompt610is automatically generated instructing the user to move the camera to the left or turn the camera towards the left side. Once the user's face is determined to be centered, a third audio prompt612may be given (e.g., “Perfect”), indicating that the user's face is centered, as illustrated inFIG. 6C.

Turning toFIG. 7, an illustrative architecture for a neural network system700used for providing directional camera assistance is provided. Neural network system700comprises a convolutional neural network720comprising three convolution blocks722,724, and726and a fully-connected classifier728. Each convolution block comprises one or more convolution layers and one or more max pooling layers, and the fully-connected classifier728comprises two fully-connected layers. Each convolution layer in convolution blocks722,724, and726may be followed by an activation layer with an exponential linear unit, such as a sigmoid activation layer.

As one of ordinary skill in the art may appreciate, each layer within the network720comprises a plurality of neurons, and each neuron can be tuned to increase the overall accuracy of the system700when training data is provided. In this way, the training data provides a correct answer to the convolutional neural network720and its neurons such that over time, the convolutional neural network720can begin tuning the computations computed within each neuron to eventually find a correct answer on its own. As such, any one or more neural network neurons of the convolutional neural network720can be modified based, at least in part, on received training image data.

An input image710is fed to the first convolution block722, and the outputs of each convolution block are fed to the next architectural layer. The fully-connected classifier728outputs a directional assistance determination, such as a probability of the image being a particular directional assistance category. The output directional assistance category730may comprise the category in which the training input image has the highest probability of belonging. During training, input image710comprises a training input image, such as one of training input images410,420,430,440, or450ofFIGS. 4A-4E. The output directional assistance category730is compared to a reference directional assistance category previously associated the input training image710to determine errors between the output of the neural network720and the references. Such errors are then fed back through the neural network system700to appropriately train the neural network720, for example, by adjusting the weight of the network connections to reduce the value of the errors. This process can be repeated for a sufficiently large number of training cycles until the neural network system700converges to a state where the error of the calculations is small enough that the output reaches a desired threshold level of similarity to the reference labels.

Once the convolutional neural network720is trained, input image710may comprise an image that does not have an associated directional assistance label, such as a frame extracted from a live video feed, and the convolutional neural network720outputs a directional assistance category730that is the category with the highest probability for the input image710.FIG. 8depicts an example embodiment of a neural network, such as convolutional neural network720, that was reduced to practice.

Example Flow Diagrams

With reference toFIG. 9, a flow diagram is provided to show an embodiment of a method900for training a neural network, such as convolutional neural network720ofFIG. 7, for example, to provide directional camera assistance based on an input image, in accordance with embodiments of the present invention. Method900may be performed, for example, by training engine204of directional assistance manager202, as illustrated inFIG. 2.

At block902, a plurality of training images are received. These images may be received from a data store, such as data store218ofFIG. 2. In exemplary aspects, each training images, which may be referred to as an original or initial training image, depicts one person and depicts the person's face. In some aspects, each initial training image depicts an entire face of a person. At block904, a plurality of cropped training images are created from each training image. Each cropped training image is associated with a cropped region of a training image. Accordingly, the cropped training images may be generated by creating copies of an initial training image, cropping different regions of the initial training image for each copy, and resizing each resulting cropped training image to the size of the initial training image as described with respect toFIGS. 5A-5D. For example, the cropped training images created from each initial image may comprise an image with a left portion of the initial training image cropped; (b) an image of a right portion of the initial training image cropped, (c) an image of a top portion of the initial training image cropped, (d) an image of a bottom portion of the initial training image cropped; and (e) an image of a central region of the initial training image. One or more of these cropped training images may depict a portion of a person's face. Further, at least some of the cropped training images may be a crop in which the person's face is completely or almost completely cropped out. The cropped training images may comprise non-facial features of the person, such as the person's hair, ears, neck, shoulders, and the like.

At block906, a directional assistance reference label is associated with each cropped training image. The reference label associated with a particular cropped training image is determined based on the particular cropped region of the initial training image that was used to create the cropped training image. All cropped training images created from the same training image are associated with different reference labels. In exemplary aspects, the reference labels include “Go right”, “Go left”, “Go up”, “Go down”, and “Perfect” as discussed with respect toFIGS. 3 and 4A-4E. The cropped training images may be automatically associated with directional assistance reference labels as the cropped training images are being created.

At block908, a neural network system is trained to determine the position of a person's face (also referred to herein as the subject's face) within a field of view captured in the image data. The neural network system is trained using the cropped training images and the associated directional assistance reference labels. The neural network may comprise a convolution neural network with three convolution blocks and a two fully-connected classifier layers as depicted inFIG. 7and trained as discussed with respect to training component208ofFIG. 2. Accordingly, each cropped training image is input into the neural network system to determine the position of the person's face within the field of view of that particular training image. In exemplary aspects, this determination is made by classifying the cropped training image into a directional assistance category that indicates the position of the person's face relative to the center of the field of view. The training directional assistance output of the neural network, such as the directional assistance category, is compared to the directional assistance reference label associated with the cropped training image. Based on errors determined from these comparisons, the neural network is adjusted to increase accuracy for future iterations.

These comparisons of the generated output determinations and the reference labels may include determining one or more loss function types. For example, the comparison between a training directional assistance output and an associated reference label may include determining cross-entropy as a loss function. It is contemplated that other types of loss functions may be used including, for instance, an adversarial term, an L2/L3/L4/Ln loss, a masked loss, a render loss, and the like. In some embodiments, Adam optimizer is used for the loss function minimization over an entire training set.

In some embodiments, the neural network is trained to provide directional assistance for centering a face when there are multiple faces or at least portions of multiple faces within an image. In this case, the neural network may be trained with training images having a least portion of multiple faces. The position of each face within the image relative to a center may be determined in a similar manner as described above except the threshold for a substantially centered face may be different for images with multiple faces. For example, in some aspects, the faces are considered centered when a bounding box for each detected face is within the field of view. Alternatively, the neural network may be trained to identify whether a center of a group of faces, which may not necessarily be a center of an individual face, is aligned with the center of the field of view. For instance, if two faces are detected within an input image, a midpoint between the two faces may be determined to be either centered or off-centered in a specific direction.

FIG. 10depicts a block diagram illustrating an example method1000for providing directional camera assistance. Method1000may be performed, for example, by directional assistance engine212of the directional assistance manager202ofFIG. 2. Additionally, method1000may be performed utilizing neural network system700ofFIG. 7after the system has been trained.

At block1002, image data is received from a user device. The image data may comprise a frame extracted from a video feed of a camera and depict at least part of a person. The image data may be automatically received in real time, and the image data may be displayed on a display screen of the user device. In other embodiments, the image data is received from a data store, such as data store218, or may be received directly from a user's input such as through uploading an image from a camera or inputting a like or URL to an image.

At block1004, a trained neural network system is used to determine the position of the person's face within the field of view captured in the image data. The position of the person's face within the field of view may be determined by classifying a frame within the image data to a directional assistance category that indicates a directional relationship between a user's face and a center region of a frame within the image data, which represents the center of a camera's field of view. The directional assistance category may be selected from a plurality of categories, including move right, indicating a camera needs to be moved a distance to the right to capture the subject's face in the center region of the frame; move left, indicating the camera needs to be moved a distance to the left to capture the subject's face in the center region of the frame; move up, indicating the camera needs to be moved a distance upward to capture the subject's face in the center region of the frame; move down, indicating the camera needs to be moved a distance to the right to capture the subject's face in the center region of the frame; and centered, indicating the subject's face is centered or substantially centered within the frame. The neural network determines a probability of the image data belonging to each directional assistance category, and the assigned category is the one with the highest probability for that image data.

The neural network system is trained to determine the position of the person's face within the field of view using features identified from pixels within the image data. The features may include facial features of a person, but the features also include non-facial facials, allowing the system to assign a category even if the person's face is not captured within the image data.

At block1006, a non-visual prompt corresponding to the position of the person's face is automatically initiated to guide a user to center the person's face within the field of view of the camera. As such, the non-visual prompt may be an instruction for repositioning the camera. The non-visual output may comprise an audio message and/or tactile feedback, such as vibrations. For instance, if the image data captures a person's face that is off-centered to the right side of the camera's field of view, and audio prompt of “Move right” may be automatically generated to instruct the user to move the camera to the right. One or more visual outputs may also be provided with the non-visual output.

As mentioned, the image data may comprise a video feed, and every n number of frames from the video feed may be automatically assigned a directional assistance category, allowing the user to receive directional assistance feedback in real time. In some embodiments, every 20thframe of video captured is input in the neural network system to determine a directional assistance category. Frames from the video data may continue to be assigned directional assistance category until a face is determined to be centered. In some embodiments, directional assistance categories are assigned to frames at an initial frame rate until a centered category is assigned to a frame, and after the centered category is assigned, a new frame rate that is lower than the initial frame rate may be used while image data is continuing to be received. In some embodiments, directional assistance continues to be performed at the initial or reduced rate but output prompts are only initiated when a user is no longer centered. In other embodiments, directional assistance is turned off after a centered (e.g., “Perfect”) category is determined. Additionally, directional assistance may be initiated upon a user command. For example, when setting up a camera for a video conference, a user may achieve a centered category assignment, which may turn off the directional assistance or reduce the rate of the directional assistance so prompts are not continued to be given to the user. If the user later moves, the user may input an indication to receive directional assistance so the user's face can be re-centered within the camera's field of view.

Example Operating Environment

With reference toFIG. 11, computing device1100includes a bus1110that directly or indirectly couples the following devices: memory1112, one or more processors1114, one or more presentation components1116, input/output (I/O) ports1118, input/output components1120, and an illustrative power supply1122. Bus1110represents what may be one or more busses (such as an address bus, data bus, or combination thereof). Although the various blocks ofFIG. 11are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be grey and fuzzy. For example, one may consider a presentation component such as a display device to be an I/O component. Also, processors have memory. The inventor recognizes that such is the nature of the art, and reiterates that the diagram ofFIG. 11is merely illustrative of an exemplary computing device that can be used in connection with one or more embodiments of the present invention. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope ofFIG. 11and reference to “computing device.”

Memory1112includes computer-storage media in the form of volatile and/or nonvolatile memory. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical-disc drives, etc. Computing device1100includes one or more processors that read data from various entities such as memory1112or I/O components1120. Presentation component(s)1116present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.

From the foregoing, it will be seen that this invention is one well adapted to attain all the ends and objects set forth above, together with other advantages which are obvious and inherent to the system and method. It will be understood that certain features and subcombinations are of utility and may be employed without reference to other features and subcombinations. This is contemplated by and is within the scope of the claims. For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to one skilled in the art that alternate embodiments may be practiced without the specific details. In other instances, well-known features have been omitted or simplified in order not to obscure the illustrative embodiments.

Embodiments presented herein have been described in relation to particular embodiments which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those of ordinary skill in the art to which the present disclosure pertains without departing from its scope.