Patent Description:
In the field of personal biometric identification, one of the most effective known methods is to use the naturally occurring patterns in the human eye, predominantly the iris or the retina. In both the iris and the retina, patterns of color, either from the fibers of the stroma in the case of the iris or from the patterns of blood vessels in the case of the retina, are used for personal biometric identification. In either case, these patterns are generated epigenetically by random events in the morphogenesis of this tissue; this means that they will be distinct for even genetically identical (monozygotic) twins.

A conventional iris code is a bit string extracted from an image of the iris. To compute the iris code, an eye image is segmented to separate the iris form the pupil and sclera, the segmented eye image is mapped into polar or pseudo-polar coordinates, and phase information is extracted using complex-valued two-dimensional wavelets (e.g., Gabor or Haar). A typical iris code is a bit string based on the signs of the wavelet convolutions and has <NUM> bits. The iris code may be accompanied by a mask with an equal number of bits that signify whether an analyzed region was occluded by eyelids, eyelashes, specular reflections, or corrupted by noise. Use of such an iris code is the standard for many common iris-based biometric tasks such as identification of passengers from passport data.

<NPL>), discloses a system for multitask visual recognition comprising hardware programmed to process an image using a convolution neural network comprising several towers with shared layers and cross-residual dependencies. Each tower is designed for achieving a specific task. An output of the shared layers is connected to a first input layer and to as second input layer of one of the specific task layers. Jifeng Dai ET AL: "Instance-aware Semantic Segmentation via Multi-task Network Cascades", <NUM> December <NUM>, DOI: <NUM>/CVPR. <NUM> discloses a multi-task network cascade consisting of three networks, respectively differentiating instances, estimating masks, and categorizing objects. <NPL>, DOI: <NUM>/ICIP. <NUM> discloses a multi-task Convolutional Neural Network (CNN) for estimating image quality and identifying distortions. <NPL> discloses a multi-task learning framework to train a network to perform gland segmentation, wherein contour detection is used as the second task. The aforementioned prior art documents do not relate to eye images and more importantly to combining the tasks of image segmentation and image quality assessment in a multi-task network.

<NPL>, discloses a computer algorithm for iris recognition, involving a segmentation of the eye image followed by a quality assessment.

The prior art fails to provide an efficient iris recognition optimizing the computer and energy resources available.

The process of segmenting an eye image to separate the iris from the pupil and sclera has many challenges.

The invention is directed to a system according to claim <NUM>.

Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter.

Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure.

A conventional wavelet-based iris code with <NUM> bits can be used for iris identification. However, the iris code can be sensitive to variations including image cropping, image blurring, lighting conditions while capturing images, occlusion by eyelids and eyelashes, and image angle of view. Additionally, prior to computing the iris code, an eye image needs to be segmented to separate the iris region from the pupil region and the surrounding sclera region.

A convolutional neural network (CNN) may be used for segmenting eye images. Eye images can include the periocular region of the eye, which includes the eye and portions around the eye such as eyelids, eyebrows, eyelashes, and skin surrounding the eye. An eye image can be segmented to generate the pupil region, iris region, or sclera region of an eye in the eye image. An eye image can also be segmented to generate the background of the eye image, including skin such as an eyelid around an eye in the eye image. The segmented eye image can be used to compute an iris code, which can in turn be used for iris identification. To generate an eye image segmentation useful or suitable for iris identification, quality of the eye image or segmented eye image may be determined or estimated. With the quality of the eye image or segmented eye image determined, eye images that may not be useful or suitable for iris identification can be determined and filtered out from subsequent iris identification. For example, eye images which capture blinking eyes, blurred eye images, or improperly segmented eye images may not be useful or suitable for iris identification. By filtering out poor quality eye images or segmented eye images, iris identification can be improved. One possible cause of generating improperly segmented eye images is having an insufficient number of eye images that are similar to the improperly segmented eye images when training the convolutional neural network to segment eye images.

Systems and methods disclosed herein address various challenges related to eye image segmentation and image quality estimation. A convolutional neural network such as a deep neural network (DNN) is used to perform both eye image segmentation and image quality estimation. The CNN for performing both eye image segmentation and image quality estimation has a merged architecture. A CNN with a merged architecture includes a segmentation tower, which segments eye images, and a quality estimation tower, which determines quality estimations of eye images so poor quality eye images can be filtered out. The segmentation tower includes segmentation layers connected to shared layers. The segmentation layers are CNN layers unique to the segmentation tower and not shared with the quality estimation tower. The quality estimation tower includes quality estimation layers connected to the shared layers. The quality estimation layers are CNN layers unique to the quality estimation tower and not shared with the segmentation tower. The shared layers are CNN layers that are shared by the segmentation tower and the quality estimation tower.

The segmentation tower segments eye images to generate segmentations of the eye images. The shared layers of the segmentation tower (or the quality estimation tower) receive as its input an eye image, for example a <NUM> x <NUM> grayscale image. The segmentation tower generates segmentation tower output. The segmentation tower output can include multiple images, e.g., four images, one for each of the pupil region, iris region, sclera region, or background region of the eye image. The quality estimation tower generates quality estimations of the eye images or segmented eye images.

When training the convolutional neural network with the merged architecture, many kernels can be learned. A kernel, when applied to its input, produces a resulting feature map showing the response to that particular learned kernel. The resulting feature map can then be processed by a kernel of another layer of the CNN which down samples the resulting feature map through a pooling operation to generate a smaller feature map. The process can then be repeated to learn new kernels for computing their resulting feature maps.

The segmentation tower (or the quality estimation tower) in the merged CNN architecture can implement an encoding-decoding architecture. The early layers of the segmentation tower (or the quality estimation tower) such as the shared layers can encode the eye image by gradually decreasing spatial dimension of feature maps and increasing the number of feature maps computed by the layers. Some layers of the segmentation tower (or the quality estimation tower) such as the last layers of the segmentation layers (or the quality estimation layers) can decode the encoded eye image by gradually increasing spatial dimension of feature maps back to the original eye image size and decreasing the number of feature maps computed by the layers.

A possible advantage of the merged CNN architecture including both a segmentation tower and a quality estimation tower is that during training, the shared layers of the CNN find feature maps that are useful for both segmentation and image quality. Accordingly, such a CNN can be beneficial compared to use of separate CNNs, one for segmentation and another one for quality estimation, in which the feature maps for each separate CNN may have little or no relationship.

<FIG> is a block diagram of an example convolutional neural network <NUM> with a merged architecture that includes a segmentation tower <NUM> and a quality estimation tower <NUM> sharing shared layers <NUM>. The convolutional neural network <NUM> such as a deep neural network (DNN) is used to perform both eye image segmentation and image quality estimation. A CNN <NUM> with a merged architecture includes a segmentation tower <NUM> and a quality estimation tower <NUM>. The segmentation tower <NUM> includes segmentation layers <NUM> connected to the shared layers <NUM>. The shared layers <NUM> are CNN layers that are shared by the segmentation tower <NUM> and the quality estimation tower <NUM>. An output layer of the shared layers <NUM> is connected to an input layer of the segmentation layers <NUM>. One or more output layers of the shared layers <NUM> are connected to one or more input layers of the segmentation layers <NUM>. The segmentation layers <NUM> are CNN layers unique to the segmentation tower <NUM> and not shared with the quality estimation tower <NUM>.

The quality estimation tower <NUM> includes quality estimation layers <NUM> and the shared layers <NUM>. The quality estimation layers <NUM> are CNN layers unique to the quality estimation tower <NUM> and not shared with the segmentation tower <NUM>. An output layer of the shared layers <NUM> is a shared layer <NUM> that is connected to an input layer of the quality estimation layers <NUM>. An input layer of the quality estimation layers <NUM> is connected to an output layer of the shared layers <NUM>. One or more output layers of the shared layers <NUM> can be connected to one or more input layers of the quality estimation layers <NUM>.

The segmentation tower <NUM> processes an eye image <NUM> to generate segmentations of the eye image. <FIG> schematically illustrates an example eye <NUM> in an eye image <NUM>. The eye <NUM> includes eyelids <NUM>, a sclera <NUM>, an iris <NUM>, and a pupil <NUM>. A curve 216a shows the pupillary boundary between the pupil <NUM> and the iris <NUM>, and a curve 212a shows the limbic boundary between the iris <NUM> and the sclera <NUM> (the "white" of the eye). The eyelids <NUM> include an upper eyelid 204a and a lower eyelid 204b.

With reference to <FIG>, an input layer of the shared layers <NUM> of the segmentation tower <NUM> (or the quality estimation tower <NUM>) receives as its input an eye image <NUM>, for example a <NUM> x <NUM> grayscale image. The segmentation tower <NUM> generates segmentation tower output <NUM>. The segmentation tower output <NUM> can include multiple images, e.g., four images, one for each region corresponding to the pupil <NUM>, the iris <NUM>, the sclera <NUM>, or the background in the eye image <NUM>. The background of the eye image can include regions that correspond to eyelids, eyebrows, eyelashes, or skin surrounding an eye in the eye image <NUM>. The segmentation tower output <NUM> includes a segmented eye image. A segmented eye image can include segmented pupil, iris, sclera, or background.

The quality estimation tower <NUM> processes an eye image <NUM> to generate quality estimation tower output such as a quality estimation of the eye image <NUM>. A quality estimation of the eye image <NUM> can be a binary classification: a good quality estimation classification or a bad quality estimation classification. A quality estimation of the eye image <NUM> can comprise a probability of the eye image <NUM> having a good quality estimation classification. If the probability of the eye image <NUM> being good exceeds a high quality threshold (such as <NUM>%, <NUM>%, <NUM>%), the image can be classified as being good. Conversely, in some embodiments, if the probability is below a low quality threshold (such as <NUM>%, <NUM>%, <NUM>%), then the eye image <NUM> can be classified as being poor.

When training the convolutional neural network <NUM>, many kernels are learned. A kernel, when applied to the input eye image <NUM> or a feature map computed by a previous CNN layer, produces a resulting feature map showing the response of its input to that particular kernel. The resulting feature map can then be processed by a kernel of another layer of the convolutional neural network <NUM> which down samples the resulting feature map through a pooling operation to generate a smaller feature map. The process can then be repeated to learn new kernels for computing their resulting feature maps. Accordingly, the shared layers can be advantageously trained simultaneously when training the segmentation tower <NUM> and the quality estimation tower <NUM>.

The segmentation tower <NUM> (or the quality estimation tower <NUM>) can implement an encoding-decoding architecture. The early layers of the segmentation tower <NUM> (or the quality estimation tower <NUM>) such as the shared layers <NUM> can encode an eye image <NUM> by gradually decreasing spatial dimension of feature maps and increasing the number of feature maps computed by the layers. Decreasing spatial dimension may advantageously result in the feature maps of middle layers of the segmentation tower <NUM> (or the quality estimation tower <NUM>) global context aware.

However decreasing spatial dimension may result in accuracy degradation, for example, at segmentation boundaries such as the pupillary boundary or the limbic boundary. A layer of the segmentation tower <NUM> (or the quality estimation tower <NUM>) concatenates feature maps from different layers such as output layers of the shared layers <NUM>. The resulting concatenated feature maps are advantageously multi-scale because features extracted at multiple scales are used to provide both local and global context and the feature maps of the earlier layers retain more high frequency details leading to sharper segmentation boundaries.

In some implementations, a convolution layer with a kernel size greater than <NUM> pixels x <NUM> pixels can be replaced with consecutive <NUM> pixels x <NUM> pixels convolution layers. With consecutive <NUM> pixels x <NUM> pixels convolution layer, the convolutional neural network <NUM> can advantageously be smaller or faster.

Some layers of the segmentation tower <NUM> (or the quality estimation tower <NUM>) such as the last layers of the segmentation layers <NUM> (or the quality estimation layers <NUM>) can decode the encoded eye image by gradually increasing spatial dimension of feature maps back to the original eye image size and decreasing the number of feature maps. Some layers of the convolutional neural network <NUM>, for example the last two layers of the quality estimation layers <NUM>, can be fully connected.

The convolutional neural network <NUM> can include one or more neural network layers. A neural network layer can apply linear or non-linear transformations to its input to generate its output. A neural network layer can be a convolution layer, a normalization layer (e.g., a brightness normalization layer, a batch normalization (BN) layer, a local contrast normalization (LCN) layer, or a local response normalization (LRN) layer), a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, a softsign layer, a recurrent layer, or any combination thereof.

A convolution layer can apply a set of kernels that convolve or apply convolutions to its input to generate its output. The normalization layer can be a brightness normalization layer that normalizes the brightness of its input to generate its output with, for example, L2 normalization. A normalization layer can be a batch normalization (BN) layer that can normalize the brightness of a plurality of images with respect to one another at once to generate a plurality of normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization can normalize the contrast of an image non-linearly by normalizing local regions of the image on a per pixel basis to have mean of zero and variance of one. Local response normalization can normalize an image over local input regions to have mean of zero and variance of one. The normalization layer may speed up the computation of the eye segmentations and quality estimations.

A rectified linear layer can be a rectified linear layer unit (ReLU) layer or a parameterized rectified linear layer unit (PReLU) layer. The ReLU layer can apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) can be, for example, max(<NUM>, x). The PReLU layer can apply a PReLU function to its input to generate its output. The PReLU function PReLU(x) can be, for example, x if x ≥ <NUM> and ax if x < <NUM>, where a is a positive number.

An upsampling layer can upsample its input to generate its output. For example, the upsampling layer can upsample a <NUM> pixels x <NUM> pixels input to generate a <NUM> pixels x <NUM> pixels output using upsampling methods such as the nearest neighbor method or the bicubic interpolation method. The concatenation layer can concatenate its input to generate its output. For example, the concatenation layer can concatenate four <NUM> pixels x <NUM> pixels feature maps to generate one <NUM> pixels x <NUM> pixels feature map. As another example, the concatenation layer can concatenate four <NUM> pixels x <NUM> pixels feature maps and four <NUM> pixels x <NUM> pixels feature maps to generate eight <NUM> pixels x <NUM> pixels feature maps. The pooling layer can apply a pooling function which down samples its input to generate its output. For example, the pooling layer can down sample a <NUM> pixels x <NUM> pixels image into a <NUM> pixels x <NUM> pixels image. Non-limiting examples of the pooling function include maximum pooling, average pooling, or minimum pooling.

A node in a fully connected layer is connected to all nodes in the previous layer. A linear fully connected layer, similar to a linear classifier, can be a fully connected layer with two output values such as good quality or bad quality. The softsign layer can apply a softsign function to its input. The softsign function (softsign(x)) can be, for example, (x / (<NUM> + |x|)). The softsign layer may neglect impact of per-element outliers. A per-element outlier may occur because of eyelid occlusion or accidental bright spot in the eye images.

At a time point t, the recurrent layer can compute a hidden state s(t), and a recurrent connection can provide the hidden state s(t) at time t to the recurrent layer as an input at a subsequent time point t+<NUM>. The recurrent layer can compute its output at time t+<NUM> based on the hidden state s(t) at time t. For example, the recurrent layer can apply the softsign function to the hidden state s(t) at time t to compute its output at time t+<NUM>. The hidden state of the recurrent layer at time t+<NUM> has as an input the hidden state s(t) of the recurrent layer at time t. The recurrent layer can compute the hidden state s(t+<NUM>) by applying, for example, a ReLU function to its input.

The number of the neural network layers in the convolutional neural network <NUM> can be different in different implementations. For example, the number of the neural network layers in the convolutional neural network <NUM> can be <NUM>. The input type of a neural network layer can be different in different implementations. For example, a neural network layer can receive the output of a neural network layer as its input. The input of a neural network layer can be different in different implementations. For example, the input of a neural network layer can include the output of a neural network layer.

The input size or the output size of a neural network layer can be quite large. The input size or the output size of a neural network layer can be n × m, where n denotes the height in pixels and m denotes the width in pixels of the input or the output. For example, n × m can be <NUM> pixels x <NUM> pixels. The channel size of the input or the output of a neural network layer can be different in different implementations. For example, the channel size of the input or the output of a neural network layer can be eight. Thus, the a neural network layer can receive eight channels or feature maps as its input or generate eight channels or feature maps as its output. The kernel size of a neural network layer can be different in different implementations. The kernel size can be n × m, where n denotes the height in pixels and m denotes the width in pixels of the kernel. For example, n or m can be <NUM> pixels. The stride size of a neural network layer can be different in different implementations. For example, the stride size of a neural network layer can be three. A neural network layer can apply a padding to its input, for example a n × m padding, where n denotes the height and m denotes the width of the padding. For example, n or m can be one pixel.

<FIG> depict an example convolutional neural network <NUM> with a merged architecture. <FIG> depicts an example architecture of the shared layers <NUM> of the segmentation tower <NUM> of the convolutional neural network <NUM>. An input layer of the shared layers <NUM> can be a convolution layer 302a that convolves an input eye image <NUM> (a <NUM> x <NUM> grayscale image) with <NUM> x <NUM> kernels (<NUM> pixels x <NUM> pixels) after adding a <NUM> x <NUM> padding (<NUM> pixel x <NUM> pixel). After adding a padding and convolving its input, the convolution layer 302a generates <NUM> channels of output with each channel being a <NUM> x <NUM> feature map, denoted as <NUM> x <NUM> x <NUM> in the block representing the convolution layer 302a. The <NUM> channels of output can be processed by a local response normalization (LRN) layer 302b, a batch normalization (BN) layer 302c, and a rectified linear layer unit (ReLU) layer 302d.

The ReLU layer 302d can be connected to a convolution layer 304a that convolves the output of the ReLU layer 302d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate eight channels of output (<NUM> x <NUM> feature maps). The eight channels of output can be processed by a batch normalization layer 304c and a ReLU layer 304d. The ReLU layer 304d can be connected to a maximum pooling (MAX POOLING) layer 306a that pools the output of the ReLU layer 304d with <NUM> x <NUM> kernels using <NUM> x <NUM> stride (<NUM> pixels x <NUM> pixels) to generate <NUM> channels of output (<NUM> x <NUM> feature maps).

The maximum pooling layer 306a can be connected to a convolution layer 308a that convolves the output of the maximum pooling layer 306a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 308c and a ReLU layer 308d.

The ReLU layer 308d can be connected to a convolution layer 310a that convolves the output of the ReLU layer 308d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 310c and a ReLU layer 310d. The ReLU layer 310d can be connected to a maximum pooling layer 312a that pools the output of the ReLU layer 310d with <NUM> x <NUM> kernels using <NUM> x <NUM> stride to generate <NUM> channels of output (<NUM> x <NUM> feature maps).

The maximum pooling layer 312a can be connected to a convolution layer 314a that convolves the output of the maximum pooling layer 312a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the convolution layer 314a can be randomly set to values of zero, for a dropout ratio of <NUM>. The <NUM> channels of output can be processed by a batch normalization layer 314c and a ReLU layer 314d.

The ReLU layer 314d can be connected to a convolution layer 316a that convolves the output of the ReLU layer 314d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 316c and a ReLU layer 316d. The ReLU layer 316d can be connected to a maximum pooling layer 318a that pools the output of the ReLU layer 316d with <NUM> x <NUM> kernels using <NUM> x <NUM> stride to generate <NUM> channels of output (<NUM> x <NUM> feature maps).

The maximum pooling layer 318a can be connected to a convolution layer 320a that convolves the output of the maximum pooling layer 318a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the convolution layer 320a can be randomly set to values of zero, for a dropout ratio of <NUM>. The <NUM> channels of output can be processed by a batch normalization layer 320c and a ReLU layer 320d.

The ReLU layer 320d can be connected to a convolution layer 322a that convolves the output of the ReLU layer 320d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 322c and a ReLU layer 322d. The ReLU layer 322d can be connected to a maximum pooling layer 324a that pools the output of the ReLU layer 322d with <NUM> x <NUM> kernels using <NUM> x <NUM> stride after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The maximum pooling layer 324a can be connected to an input layer of the segmentation layers <NUM>.

The maximum pooling layer 324a can be connected to a convolution layer 326a that convolves the output of the maximum pooling layer 324a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the convolution layer 326a can be randomly set to values of zero, for a dropout ratio of <NUM>. The <NUM> channels of output can be processed by a batch normalization layer 326c and a ReLU layer 326d. The maximum pooling layer 324a can be connected to the segmentation layers <NUM>.

The ReLU layer 326d can be connected to a convolution layer 328a that convolves the output of the ReLU layer 326d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 328c and a ReLU layer 328d. The ReLU layer 328d can be connected to a maximum pooling layer 330a that pools the output of the ReLU layer 328d with <NUM> x <NUM> kernels using <NUM> x <NUM> stride to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The maximum pooling layer 330a can be connected to the segmentation layers <NUM> and the quality estimation layers <NUM>.

The example shared layers <NUM> in <FIG> implements an encoding architecture. The example shared layers <NUM> encodes an eye image <NUM> by gradually decreasing spatial dimension of feature maps and increasing the number of feature maps computed by the layers. For example, the convolution layer 302a generates <NUM> channels of output with each channel being a <NUM> x <NUM> feature map while the convolution layer 326a generates <NUM> channels of output with each channel being a <NUM> x <NUM> feature map.

<FIG> depicts an example architecture of the segmentation layers <NUM> of the segmentation tower <NUM> of the convolutional neural network <NUM>. An input layer of the segmentation layers <NUM> can be an average pooling layer 332a that is connected to the maximum pooling layer 330a of the shared layers <NUM>. The average pooling layer 332a can pool the output of the maximum pooling layer 330a with <NUM> x <NUM> kernels (<NUM> pixels x <NUM> pixels) to generate <NUM> channels of output (<NUM> x <NUM> feature maps, i.e. feature maps each with a dimension of <NUM> pixel x <NUM> pixel). The average pooling layer 332a can be connected to an upsampling layer 334a that uses the nearest neighbor method with a -<NUM> x <NUM> padding (-<NUM> pixel x <NUM> pixel) to generate <NUM> channels of output (<NUM> x <NUM> feature maps).

A concatenation layer 336a is an input layer of the segmentation layers <NUM> that is connected to the maximum pooling layer 330a of the shared layers <NUM>. The concatenation layer 336a is also connected to the upsampling layer 334a. After concatenating its input received from the maximum pooling layer 330a and the upsampling layer 334a, the concatenation layer 336a can generate <NUM> channels of output (<NUM> x <NUM> feature maps). By concatenating the outputs from two layers, features extracted at multiple scales are used to provide both local and global context and the feature maps of the earlier layers retain more high frequency details leading to sharper segmentation boundaries. Thus, the resulting concatenated feature maps generated by the concatenation layer 336a are advantageously multi-scale. The concatenation layer 336a can be connected to an upsampling layer 338a that uses the nearest neighbor method to generate <NUM> channels of output (<NUM> x <NUM> feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the upsampling layer 338a can be randomly set to values of zero, for a dropout ratio of <NUM>.

The upsampling layer 338a can be connected to a convolution layer 340a that convolves the output of the upsampling layer 338a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 340c and a ReLU layer 340d. The ReLU layer 340d can be connected to a convolution layer 342a that convolves the output of the ReLU layer 340d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 342c and a ReLU layer 342d.

A concatenation layer 344a is an input layer of the segmentation layers <NUM> that is connected to the maximum pooling layer 324a of the shared layers <NUM>. The concatenation layer 344a can also be connected to the ReLU layer 342a. After concatenating its input received from the ReLU layer 342a and the maximum pooling layer 324a, the concatenation layer 344a generates <NUM> channels of output (<NUM><NUM> x <NUM> feature maps). The concatenation layer 344a can be connected to an upsampling layer 346a that uses the nearest neighbor method to generate <NUM> channels of output (<NUM> x <NUM> feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the upsampling layer 346a can be randomly set to values of zero, for a dropout ratio of <NUM>.

The upsampling layer 346a can be connected to a convolution layer 348a that convolves the output of the upsampling layer 346a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 348c and a ReLU layer 348d. The ReLU layer 348d can be connected to a convolution layer 350a that convolves the output of the ReLU layer 348d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 350c and a ReLU layer 350d.

The ReLU layer 350d can be connected to an upsampling layer 352a that uses the nearest neighbor method to generate <NUM> channels of output (<NUM> x <NUM> feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the upsampling layer 352a can be randomly set to values of zero, for a dropout ratio of <NUM>.

The upsampling layer 352a can be connected to a convolution layer 354a that convolves the output of the upsampling layer 352a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 354c and a ReLU layer 354d. The ReLU layer 354d can be connected to a convolution layer 356a that convolves the output of the ReLU layer 354d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 356c and a ReLU layer 356d.

The ReLU layer 356d can be connected to an upsampling layer 358a that uses the nearest neighbor method to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The upsampling layer 358a can be connected to a convolution layer 360a that convolves the output of the upsampling layer 358a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 360c and a ReLU layer 360d. The ReLU layer 360d can be connected to a convolution layer 362a that convolves the output of the ReLU layer 360d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 362c and a ReLU layer 362d.

The ReLU layer 362d can be connected to an upsampling layer 364a that uses the nearest neighbor method to generate <NUM> channels of output (<NUM> by <NUM> feature maps). The upsampling layer 364a can be connected to a convolution layer 366a that convolves the output of the upsampling layer 364a with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> output images). The convolution layer 366a can be an output layer of the segmentation layers <NUM>. The <NUM> output images can be the segmentation tower output <NUM>, one for reach region corresponding to the pupil <NUM>, the iris <NUM>, the sclera <NUM>, or the background of the eye image <NUM>. In some implementations, the segmentation tower output <NUM> can be an image with four color values, one for each region corresponding to the pupil <NUM>, the iris <NUM>, the sclera <NUM>, or the background of the eye image <NUM>.

The example segmentation layers <NUM> in <FIG> implements a decoding architecture. The example segmentation layers <NUM> decodes the encoded eye image by gradually increasing spatial dimension of feature maps back to the original eye image size and decreasing the number of feature maps. For example, the average pooling layer 332a generates <NUM> channels of output with each channel being a <NUM> x <NUM> feature map, while the convolution layer 366a generates <NUM> channels of output with each channel being a <NUM> x <NUM> feature map.

<FIG> depicts an example architecture of the quality estimation layers <NUM> of the quality estimation tower <NUM> of the convolutional neural network <NUM>. An input layer of the quality estimation layers <NUM> is a convolution layer 368a. The convolution layer 368a convolves the output of the maximum pooling layer 330a of the shared layers <NUM> with <NUM> x <NUM> kernels (<NUM> pixels x <NUM> pixels) after adding a <NUM> x <NUM> padding (<NUM> pixel x <NUM> pixel) to generate <NUM> channels of output (<NUM> x <NUM> feature maps, i.e. feature maps with a dimension of <NUM> pixels x <NUM> pixels). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the convolution layer 368a can be randomly set to values of zero, for a dropout ratio of <NUM>. The <NUM> channels of output can be processed by a batch normalization layer 368c and a ReLU layer 368d.

The ReLU layer 368d can be connected to a convolution layer 370a that convolves the output of the ReLU layer 368d with <NUM> x <NUM> kernels after adding a <NUM> x <NUM> padding to generate <NUM> channels of output (<NUM> x <NUM> feature maps). The <NUM> channels of output can be processed by a batch normalization layer 370c and a ReLU layer 370d. The ReLU layer 370d can be connected to an average pooling layer 372a that can pool the output of the ReLU layer 370d with <NUM> x <NUM> kernels to generate <NUM> channels of output (<NUM> x <NUM> feature maps).

The average pooling layer 370d can be connected to linear, fully connected layer 374a that generates <NUM> channels of output (<NUM> pixel x <NUM> pixel feature maps). During a training cycle when training the convolutional neural network <NUM>, <NUM> % of weight values of the linear, fully connected layer 374a can be randomly set to values of zero, for a dropout ratio of <NUM>. The <NUM> channels of output can be processed by a batch normalization layer 374c and a ReLU layer 374d. The ReLU layer 374d can be connected to a linear, fully connected layer 376a that generates at least two channels of output (<NUM> x <NUM> feature maps). The linear, fully connected layer 376a can be an output layer of the quality estimation layers <NUM>. The at least two channels of output can be the quality estimation tower output <NUM> with one channel corresponding to the good quality estimation and one channel corresponding to the bad quality estimation.

Different convolutional neural networks (CNNs) can be different from one another in two ways. The architecture of the CNNs, for example the number of layers and how the layers are interconnected, can be different. The weights which can affect the strength of effect propagated from one layer to another can be different. The output of a layer can be some nonlinear function of the weighted sum of its inputs. The weights of a CNN can be the weights that appear in these summations, and can be approximately analogous to the synaptic strength of a neural connection in a biological system.

The process of training a CNN <NUM> is the process of presenting the CNN <NUM> with a training set of eye images <NUM>. The training set can include both input data and corresponding reference output data. This training set can include both example inputs and corresponding reference outputs. Through the process of training, the weights of the CNN <NUM> can be incrementally learned such that the output of the network, given a particular input data from the training set, comes to match (as closely as possible) the reference output corresponding to that input data.

Thus, in some implementations, a CNN <NUM> having a merged architecture is trained, using a training set of eye images <NUM>, to learn segmentations and quality estimations of the eye images <NUM>. During a training cycle, the segmentation tower <NUM> being trained can process an eye image <NUM> of the training set to generate a segmentation tower output <NUM> which can include <NUM> output images, one for reach region corresponding to the pupil <NUM>, the iris <NUM>, the sclera <NUM>, or the background of the eye image <NUM>. The quality estimation tower <NUM> being trained can process an eye image <NUM> of the training set to generate a quality estimation tower output <NUM> of the eye image <NUM>. A difference between the segmentation tower output <NUM> of the eye image <NUM> and a reference segmentation tower output of the eye image <NUM> can be computed. The reference segmentation tower output of the eye image <NUM> can include four reference output images, one for reach region corresponding to the pupil <NUM>, the iris <NUM>, the sclera <NUM>, or the background of the eye image <NUM>. A difference between the quality estimation tower output <NUM> of the eye image <NUM> and a reference quality estimation tower output of the eye image <NUM> can be computed.

Parameters of the CNN <NUM> can be updated based on one or both of the differences. For example, parameters of the segmentation layers <NUM> of the CNN <NUM> can be updated based on the difference between the segmentation tower output <NUM> of the eye image <NUM> and the reference segmentation tower output of the eye image <NUM>. As another example, parameters of the quality estimation layers <NUM> of the CNN <NUM> can be updated based on the difference between the quality estimation tower output <NUM> of the eye image <NUM> and the reference quality estimation tower output of the eye image <NUM>. As yet another example, parameters of the shared layers <NUM> can be updated based on both differences. As a further example, parameters of the segmentation layers <NUM> of the CNN <NUM> or parameters of the quality estimation layers <NUM> of the CNN <NUM> can be updated based on both differences. The two differences can affect the parameters of the shared layers <NUM>, the segmentation layers <NUM>, or the quality estimation layers <NUM> differently in different implementations. For example, the difference between the segmentation tower output <NUM> and the reference segmentation tower output can affect the parameters of the shared layers <NUM> or the segmentation layers <NUM> to a greater extent compared to the effect of the difference between the quality estimation tower output <NUM> and the reference quality estimation tower output.

During a training cycle, a percentage of the parameters of the convolutional neural network <NUM> can be set to values of zero. The percentage can be, for example, <NUM> % - <NUM>%, for a dropout ratio of <NUM> - <NUM>. The parameters of the CNN <NUM> set to values of zero during a training cycle can be different in different implementations. For example, parameters of the CNN <NUM> set to values of zero can be randomly selected. As another example, if <NUM>% of the parameters of the CNN <NUM> are set to values of zero, then approximately <NUM>% of parameters of each layer of the CNN <NUM> can be randomly set to values of zero.

When training the convolutional neural network <NUM> with the merged architecture, many kernels are learned. A kernel, when applied to its input, produces a resulting feature map showing the response to that particular learned kernel. The resulting feature map can then be processed by a kernel of another layer of the CNN which samples the resulting feature map through a pooling operation to generate a smaller feature map. The process can then be repeated to learn new kernels for computing their resulting feature maps.

<FIG> shows example results of segmenting eye images <NUM> using a convolutional neural network <NUM> with the merged convolutional network architecture illustrated in <FIG>. <FIG>, panel a shows a segmentation of the eye image shown in <FIG>, panel b. The segmentation of the eye image included a background region 404a, a sclera region 408a, an iris region 412a, or a pupil region 416a of the eye image. The quality estimation of the eye image shown in <FIG>, panel b was a good quality estimation of <NUM>. Accordingly, the quality estimation of the eye image was a good quality estimation.

<FIG>, panel c shows a segmentation of the eye image shown in <FIG>, panel d. The segmentation of the eye image included a background region 404c, a sclera region 408c, an iris region 412c, or a pupil region 416c of the eye image. The quality estimation of the eye image shown in <FIG>, panel d was a good quality estimation of <NUM>. Accordingly, the quality estimation of the eye image was a good quality estimation.

<FIG>, panel e shows a segmentation of the eye image shown in <FIG>, panel f. A sclera, an iris, and a pupil of an eye in the eye image shown in <FIG>, panel f were occluded by eyelids of the eye. The segmentation of the eye image included a background region 404e, a sclera region 408e, an iris region 412e, or a pupil region 416e of the eye image. The quality estimation of the eye image shown in <FIG>, panel f was a good quality estimation of <NUM>. Accordingly, the quality estimation of the eye image was a bad quality estimation.

<FIG>, panel g shows a segmentation of the eye image shown in <FIG>, panel h. A sclera, an iris, and a pupil of an eye in the eye image shown in <FIG>, panel h were occluded by eyelids of the eye. Furthermore, the eye image is blurry. The segmentation of the eye image included a background region <NUM>, a sclera region <NUM>, an iris region <NUM>, or a pupil region <NUM> of the eye image. The quality of the eye image shown in <FIG>, panel h was a good quality estimation of <NUM>. Accordingly, the quality estimation of the eye image was a bad quality estimation.

<FIG> is a flow diagram of an example process <NUM> of creating a convolutional neural network <NUM> with a merged architecture. The process <NUM> starts at block <NUM>. At block <NUM>, shared layers <NUM> of a convolutional neural network (CNN) <NUM> are created. The shared layers <NUM> can include a plurality of layers and a plurality of kernels. Creating the shared layers <NUM> can include creating the plurality of layers, creating the plurality of kernels with appropriate kernel sizes, strides, or paddings, or connecting the successive layers of the plurality of layers.

At block <NUM>, segmentation layers <NUM> of the CNN <NUM> are created. The segmentation layers <NUM> can include a plurality of layers and a plurality of kernels. Creating the segmentation layers <NUM> can include creating the plurality of layers, creating the plurality of kernels with appropriate kernel sizes, strides, or paddings, or connecting the successive layers of the plurality of layers. At block <NUM>, an output layer of the shared layers <NUM> can be connected to an input layer of the segmentation layers <NUM> to generate a segmentation tower <NUM> of the CNN <NUM>.

At block <NUM>, quality estimation layers <NUM> of the CNN <NUM> are created. The quality estimation layers <NUM> can include a plurality of layers and a plurality of kernels. Creating the quality estimation layers <NUM> can include creating the plurality of layers, creating the plurality of kernels with appropriate kernel sizes, strides, or paddings, or connecting the successive layers of the plurality of layers. At block <NUM>, an output layer of the shared layers <NUM> can be connected to an input layer of the quality estimation layers <NUM> to generate a quality estimation tower <NUM> of the CNN <NUM>. The process <NUM> ends at block <NUM>.

<FIG> is a flow diagram of an example process <NUM> of segmenting an eye image <NUM> using a convolutional neural network <NUM> with a merged architecture. The process <NUM> starts at block <NUM>. At block <NUM>, a neural network receives an eye image <NUM>. For example, an input layer of shared layers <NUM> of a CNN <NUM> can receive the eye image <NUM>. An image sensor (e.g., a digital camera) of a user device can capture the eye image <NUM> of a user, and the neural network can receive the eye image <NUM> from the image sensor.

After receiving the eye image <NUM> at block <NUM>, the neural network segments the eye image <NUM> at block <NUM>. For example, a segmentation tower <NUM> of the CNN <NUM> can generate a segmentation of the eye image <NUM>. An output layer of the segmentation tower <NUM> can, together with other layers of the segmentation tower <NUM>, compute the segmentation of the eye image <NUM>, including a pupil region, an iris region, a sclera region, or a background region of an eye in the eye image <NUM>.

At block <NUM>, the neural network computes a quality estimation of the eye image <NUM>. For example, a quality estimation tower <NUM> of the CNN <NUM> can generate the quality estimation of the eye image <NUM>. An output layer of the quality estimation tower <NUM> can, together with other layers of the quality estimation tower <NUM>, compute the quality estimation of the eye image <NUM>, such as a good quality estimation or a bad quality estimation.

A conventional iris code is a bit string extracted from an image of the iris. To compute the iris code, an eye image is segmented to separate the iris form the pupil and sclera, for example, using the convolutional neural network <NUM> with the merged architecture illustrated in <FIG>. The segmented eye image can then be mapped into polar or pseudo-polar coordinates before phase information can be extracted using complex-valued two-dimensional wavelets (e.g., Gabor or Haar). One method of creating a polar (or pseudo-polar) image of the iris can include determining a pupil contour, determining an iris contour, and using the determined pupil contour and the determined iris contour to create the polar image.

<FIG> is a flow diagram of an example process <NUM> of determining a pupil contour, an iris contour, and a mask for irrelevant image area in a segmented eye image. The process <NUM> starts at block <NUM>. At block <NUM>, a segmented eye image is received. The segmented eye image can include segmented pupil, iris, sclera, or background. A user device can capture an eye image <NUM> of a user and compute the segmented eye image. A user device can implement the example convolutional neural network (CNN) <NUM> with the merged architecture illustrated in <FIG> or the example process <NUM> illustrated in <FIG> to compute the segmented eye image.

The segmented eye image can be a semantically segmented eye image. <FIG> schematically illustrates an example semantically segmented eye image <NUM>. The semantically segmented eye image <NUM> can be computed from an image of the eye <NUM> illustrated in <FIG>. The semantically segmented eye image <NUM> can have a dimension of n pixels x m pixels, where n denotes the height in pixels and m denotes the width in pixels of the semantically segmented eye image <NUM>.

A pixel of the semantically segmented eye image <NUM> can have one of four color values. For example, a pixel <NUM> of the semantically segmented eye image <NUM> can have a color value that corresponds to a background <NUM> of the eye image (denoted as "first color value" in <FIG>). The color value that corresponds to the background <NUM> of the eye image can have a numeric value such as one. The background <NUM> of the eye image can include regions that correspond to eyelids, eyebrows, eyelashes, or skin surrounding the eye <NUM>. As another example, a pixel of the semantically segmented eye image <NUM> can have a color value that corresponds to a sclera <NUM> of the eye <NUM> in the eye image (denoted as "second color value" in <FIG>). The color value that corresponds to the sclera <NUM> of the eye <NUM> in the eye image can have a numeric value such as two. As yet example, a pixel of the semantically segmented eye image <NUM> can have a color value that corresponds to an iris <NUM> of the eye <NUM> in the eye image (denoted as "third color value" in <FIG>). The color value that corresponds to the iris <NUM> of the eye <NUM> in the eye image can have a numeric value such as three. As another example, a pixel <NUM> of the semantically segmented eye image <NUM> can have a color value that corresponds to a pupil <NUM> of the eye <NUM> in the eye image (denoted as "fourth color value" in <FIG>). The color value that corresponds to the pupil <NUM> of the eye <NUM> in the eye image can have a numeric value such as four. In <FIG>, curve 216a shows the pupillary boundary between the pupil <NUM> and the iris <NUM>, and curve 212a shows the limbic boundary between the iris <NUM> and the sclera <NUM> (the "white" of the eye).

With reference to <FIG>, at block <NUM>, a pupil contour of the eye <NUM> in the eye image can be determined. The pupil contour can be the curve 216a that shows the pupillary boundary between the pupil <NUM> and the iris <NUM>. The pupil contour can be determined using an example process <NUM> illustrated in <FIG> (described in greater detail below). At block <NUM>, an iris contour of the eye <NUM> in the eye image can be determined. The iris contour can be the curve 212a that shows the limbic boundary between the iris <NUM> and the sclera <NUM>. The iris contour can be determined using the example process <NUM> illustrated in <FIG> (described in greater detail below). The processes used for determining the pupil contour and the iris contour can be the same or can be optimized for each determination because, for example, the pupil size and the iris size can be different.

At block <NUM>, a mask image for an irrelevant area in the eye image can be determined. The mask image can have a dimension of n pixels x m pixels, where n denotes the height in pixels and m denotes the width in pixels of the mask image. A dimension of the semantically segmented eye image <NUM> and a dimension of the mask image can be the same or can be different. The mask can be a binary mask image. A pixel of the binary mask image can have a value of zero or a value of one. The pixel of the binary mask image can have a value of zero if a corresponding pixel in the semantically segmented eye image <NUM> has a value greater than or equal to, for example, the third color value such as the numeric value of three. The pixel of the binary mask image can have a value of one if a corresponding pixel in the semantically segmented eye image <NUM> does not have a value greater than or equal to, for example, the third color value such as the numeric value of three. In some implementations, the process <NUM> can optionally create a polar image of the iris <NUM> of the eye <NUM> in the eye image using the pupil contour, the iris contour, and the mask for the irrelevant area in the semantically segmented eye image. The process <NUM> ends at block <NUM>.

<FIG> is a flow diagram of an example process <NUM> of determining a pupil contour or an iris contour in a segmented eye image. The process <NUM> starts at block <NUM>. At block <NUM>, a binary image can be created from a segmented eye image, such as the semantically segmented eye image <NUM>. <FIG> schematically illustrates an example binary image 1000A created at block <NUM>. The binary image 1000A can have a dimension of n pixels x m pixels, where n denotes the height in pixels and m denotes the width in pixels of the binary image 1000A. The dimension of the segmented eye image or the semantically segmented eye image <NUM> and the dimension of the binary image 1000A can be the same or can be different.

A pixel 1004a of the binary image 1000A can have a color value of zero if a corresponding pixel in the semantically segmented eye image <NUM> has a value not greater than or equal to a threshold color value, for example the "fourth color value. " A pixel 1012a of the binary image 1000A can have a color value of one if a corresponding pixel in the semantically segmented eye image <NUM> has a value greater than or equal to a threshold color value, for example the "fourth color value. " In some implementations, pixels of the binary image 1000A can have values other than zero or one. For example, the pixel 1004a of the binary image 1000A can have a color value of "third color value" such as the numeric value three. The pixel 1012a of the binary image 1000A can have a color value of "fourth color value," such as the numeric value fourth, where the "fourth color value" is greater than the "third color value".

With reference to <FIG>, at block <NUM>, contours in the binary image 1000A are determined. For example, contours in the binary image 1000A can be determined using, for example, the OpenCV findContours function (available from opencv. <FIG> schematically illustrates an example contour <NUM> in the binary image 1000A. Referring to <FIG>, at block <NUM>, a contour border can be determined. The contour border can be a longest contour in the binary image 1000A. The contour <NUM> in the binary image 1000A can be the longest contour in the binary image 1000A. The contour <NUM> can include a plurality of pixels of the binary image 1000A, such as the pixel 1024a.

At block <NUM>, a contour points bounding box (e.g., a contour points bounding box <NUM> in <FIG>) is determined. The contour points bounding box <NUM> can be a smallest rectangle enclosing the longest contour border such as the contour border <NUM>. At block <NUM>, a points area size can be determined. The points area size can be a diagonal <NUM> of the contour points bounding box <NUM> in the binary image 1000A in <FIG>.

At block <NUM>, a second binary image can be created from a segmented eye image, such as the semantically segmented eye image <NUM>. <FIG> schematically illustrates an example second binary image 1000C. The second binary image 1000C can have a dimension of n pixels x m pixels, where n denotes the height in pixels and m denotes the width in pixels of the second binary image 1000C. The dimension of the binary image 1000A and the dimension of the binary image 1000A can the same or can be different.

A pixel 1004c of the second binary image 1000C can have a color value of zero if a corresponding pixel in the semantically segmented eye image <NUM> has a value not greater than or equal to a threshold color value, for example the "third color value. " A pixel 1012c of the second binary image 1000C can have a color value of one if a corresponding pixel in the semantically segmented eye image <NUM> has a value greater than or equal to a threshold color value, for example the "third color value. " In some implementations, pixels of the second binary image 1000C can have values other than zero or one. For example, the pixel 1004c of the second binary image 1000C can have a color value of "second color value" such as the numeric value two. The pixel 1012c of the second binary image 1000B can have a color value of "third color value," such as the numeric value three, where the "third color value" is greater than the "second color value".

With reference to <FIG>, at block <NUM>, a pixel (e.g. a pixel 1024c in <FIG>) in the second binary image 1000C that corresponds to the pixel 1024a in the binary image 1000A is determined. If a dimension of the second binary image 1000C and a dimension of the binary image 1000A are the same, then the pixel 1024c can have a coordinate of (m<NUM>; n<NUM>) in the second binary image 1000C and the pixel 1024a can have a coordinate of (m<NUM>; n<NUM>) in the binary image 1000A, wherein m<NUM> denotes the coordinate in the width direction and n<NUM> denotes the coordinate in the height direction. A distance between the pixel 1024c and a pixel in the second binary image 1000C that has a color value of <NUM> and is closest to the pixel 1024c is determined. For example, the distance can be a distance <NUM> in <FIG> between the pixel 1024c and the pixel <NUM> in the second binary image 1000C that has a color value of <NUM> and is closest to the pixel 1024c. The distance <NUM> can be determined using, for example, the OpenCV distanceTransform function.

At block <NUM>, the pixel 1024a can be removed from the pixels of the contour <NUM> if it is inappropriate for determining a pupil contour. The pixel 1024a can be inappropriate for determining a pupil contour if the distance <NUM> is smaller than a predetermined threshold. The predetermined threshold can be a fraction multiplied by a size of the contour points bounding box <NUM>, such as the points area size or a size of a diagonal <NUM> of the contour points bounding box <NUM> in <FIG>. The fraction can be in the range from <NUM> to <NUM>. For example, the fraction can be <NUM>.

At block <NUM>, a pupil contour can be determined from the remaining pixels of the contour border <NUM> by fitting a curve (such as an ellipse) to the remaining pixels. The ellipse can be determined using, for example, the OpenCV fitEllipse function. The process <NUM> ends at block <NUM>. Although <FIG> has been used to illustrates using the process <NUM> to determine a pupil contour, the process <NUM> can also be used to determine an iris contour.

<FIG> show example results of determining iris contours, pupil contours, and masks for irrelevant image areas using the example processes <NUM> and <NUM> illustrated in <FIG> and <FIG>. <FIG>, panels a-f show example results of determining an iris contour, a pupil contour, and a mask for irrelevant image area of an eye image. <FIG>, panel a shows an eye image. <FIG>, panel b shows a semantically segmented eye image of the eye image in <FIG>, panel a using a convolutional neural network <NUM> with the merged convolutional network architecture illustrated in <FIG>. The semantically segmented eye images included a background region 1104a with a numeric color value of one, a sclera region 1108a with a numeric color value of two, an iris region 1112a with a numeric color value of three, or a pupil region 1116a of the eye image with a numeric color value of four.

<FIG>, panels c shows the remaining pixels 1120a of a contour border of the pupil and the remaining pixels 1124a of a contour border of the iris overlaid on the eye image shown in <FIG>, panel a determined using the process <NUM> at block <NUM>. <FIG>, panels d shows the remaining pixels 1120a of the contour border of the pupil and the remaining pixels 1124a of the contour border of the iris overlaid on the semantically segmented eye image shown in <FIG>, panel b. <FIG>, panel e shows an ellipse of the pupil 1128a and an ellipse of the iris 1132a determined by fitting the remaining pixels of the contour border of the pupil 1120a and the contour border of the iris 1124a by the process <NUM> at block <NUM>. <FIG>, panels f shows a binary mask image for an irrelevant area in the eye image by the process <NUM> at block <NUM>. The binary mask image includes a region 1136a that corresponds to the iris region 1112a and the pupil region 1116a of the semantically segmented eye image shown in <FIG>, panel b. The binary mask image also includes a region 1140a that corresponds to the background region 1104a and the sclera region 1108a.

Similar to <FIG>, panels a-f, <FIG>, panels g-l show example results of determining an iris contour, a pupil contour, and a mask for irrelevant image area of another eye image.

<FIG> show example results of training a convolutional neural network (CNN) with a triplet network architecture on iris images in polar coordinates obtained after fitting pupil contours and iris contours with the example processes shown in <FIG> and <FIG>. The triplet network architecture is shown in <FIG> and described in greater detail below.

<FIG> is a histogram plot of the probability density vs. embedding distance. The iris images of the same subjects were closer together in the embedding space, and the iris images of different subjects were further away from one another in the embedding space. <FIG> is a receiver characteristic (ROC) curve of true positive rate (TPR) vs. false positive rate (FPR). The area under the ROC curve was <NUM>%. Using iris images in polar coordinates to train the CNN with a triplet network architecture, <NUM>% EER was achieved.

Using images of the human eye, a convolutional neural network (CNN) with a triplet network architecture can be trained to learn an embedding that maps from the higher dimensional eye image space to a lower dimensional embedding space. The dimension of the eye image space can be quite large. For example, an eye image of <NUM> pixels by <NUM> pixels can potentially include thousands or tens of thousands of degrees of freedom. <FIG> is a block diagram of an example convolutional neural network <NUM> with a triplet network architecture. A CNN <NUM> can be trained to learn an embedding <NUM> (Emb). The embedding <NUM> can be a function that maps an eye image (Img) <NUM> in the higher dimensional eye image space into an embedding space representation (EmbImg) of the eye image in a lower dimensional embedding space. For example, Emb(Img) = EmbImg. The eye image (Img) <NUM> can be an iris image in polar coordinates computed using a pupil contour and an iris contour determined with the example processes shown in <FIG> and <FIG>.

The embedding space representation, a representation of the eye image in the embedding space, can be an n-dimensional real number vectors. The embedding space representation of an eye image can be an n-dimensional eye description. The dimension of the representations in the embedding space can be different in different implementations. For example, the dimension can be in a range from <NUM> to <NUM>. In some implementations, n is <NUM>. The elements of the embedding space representations can be represented by real numbers. In some architectures, the embedding space representation is represented as n floating point numbers during training but it may be quantized to n bytes for authentication. Thus, in some cases, each eye image is represented by an n-byte representation. Representations in an embedding space with larger dimension may perform better than those with lower dimension but may require more training. The embedding space representation can have, for example, unit length.

The CNN <NUM> can be trained to learn the embedding <NUM> such that the distance between eye images, independent of imaging conditions, of one person (or of one person's left or right eye) in the embedding space is small because they are clustered together in the embedding space. In contrast, the distance between a pair of eye images of different persons (or of a person's different eye) can be large in the embedding space because they are not clustered together in the embedding space. Thus, the distance between the eye images from the same person in the embedding space, the embedding distance, can be smaller than the distance between the eye images from different persons in the embedding space. The distance between two eye images can be, for example, the Euclidian distance (a L2 norm) between the embedding space representations of the two eye images.

The distance between two eye images of one person, for example an anchor eye image (ImgA) 1312a and a positive eye image (ImgP) 1312p, can be small in the embedding space. The distance between two eye images of different persons, for example the anchor eye image (ImgA) 1312a and a negative eye image (ImgN) 1312n can be larger in the embedding space. The ImgA 1312a is an "anchor" image because its embedding space representation can be compared to embedding space representations of eye images of the same person (e.g., the ImgP 1312p) and different persons (e.g., ImgN 1312n). ImgA 1312p is a "positive" image because the ImgP 1312p and the ImgA 1312a are eye images of the same person. The ImgN 1312n is a "negative" image because the ImgN 1312n and the ImgA 1312a are eye images of different persons. Thus, the distance between the ImgA 1312a and the ImgP 1312p in the embedding space can be smaller than the distance between the ImgA 1312a and the ImgN 1312N in the embedding space.

The embedding network (Emb) <NUM> can map the ImgA 1312a, the ImgP 1312p, and the ImgN 1312n in the higher dimensional eye image space into an anchor embedding image (EmbA) 1316a, a positive embedding image (EmbP) 1316a, and a negative embedding image (EmbN) 1316n. For example, Emb(ImgA) = EmbA; Emb(ImgP) = EmbP; and Emb(ImgN) = EmbN. Thus, the distance between the EmbA 1316a and the EmbP 1316a in the embedding space can be smaller than the distance between EmbP 1316a and EmbN 1316n in the embedding space.

To learn the embedding <NUM>, a training set T1 of eye images <NUM> can be used. The eye images <NUM> can be iris images in polar coordinates computed using a pupil contour and an iris contour determined with the example processes shown in <FIG>. The eye images <NUM> can include the images of left eyes and right eyes. The eye images <NUM> can be associated with labels, where the labels distinguish the eye images of one person from eye images of another person. The labels can also distinguish the eye images of the left eye and the right eye of a person. The training set T1 can include pairs of eye image and label (Img; Label). The training set T1 of (Img; Label) pairs can be received from an eye image data store.

To learn the embedding <NUM>, the CNN <NUM> with a triplet network architecture can include three identical embedding networks, for example an anchor embedding network (ENetworkA) 1320a, a positive embedding network (ENetworkP) 1320p, and a negative embedding network (ENetworkN) 1320n. The embedding networks 1320a, 1320p, or 1320n can map eye images from the eye image space into embedding space representations of the eye images in the embedding space. For example, the ENetworkA 1320a can map an ImgA 1312a into an EmbA 1316a. The ENetworkA 1320p can map an ImgP 1312p into an EmbP 1316p. The ENetworkN 1320n can map an ImgN 1312n into an EmbN 1316n.

The convolutional neural network <NUM> with the triplet network architecture can learn the embedding <NUM> with a triplet training set T2 including triplets of eye images. Two eye images of a triplet are from the same person, for example the ImgA 1312a and the ImgP 1312p. The third eye image of the triplet is from a different person, for example the ImgN 1312n. The ENetworkA 1320a, the ENetworkP 1320p, and the ENetworkN 1320n can map triplets of (ImgA; ImgP; ImgN) into triplets of (EmbA; EmbP; EmbN). The eye authentication trainer <NUM> can generate the triplet training set T2 from the training set T1 of (Img; Label) pairs.

The ImgA 1312a, the ImgP 1312p, or the ImgN 1312n can be different in different implementations. For example, the ImgA 1312a and the ImgP 1312p can be eye images of one person, and the ImgN 1312n can be an eye image of another person. As another example, the ImgA 1312a and the ImgP 1312p can be images of one person's left eye, and the ImgN 1312n can be an image of the person's right eye or an eye image of another person.

The triplet network architecture can be used to learn the embedding <NUM> such that an eye image of a person in the embedding space is closer to all other eye images of the same person in the embedding space than it is to an eye image of any other person in the embedding space. For example, |EmbA - EmbP| < |EmbA - EmbN|, where |EmbA - EmbP| denotes the absolute distance between the EmbA 1316a and the EmbP 1316p in the embedding space, and |EmbA - EmbN| denotes the absolute distance between the EmbA 1316a and the EmbN 1316n in the embedding space.

In some implementations, the triplet network architecture can be used to learn the embedding <NUM> such that an image of a person's left eye in the embedding space is closer to all images of the same person's left eye in the embedding space than it is to any image of the person's right eye or any eye image of another person in the embedding space.

The dimension of the embedding space representations can be different in different implementations. The dimension of the EmbA 1316a, EmbP 1316p, and EmbN 1316n can be the same, for example <NUM>. The length of the embedding space representation can be different in different implementations. For example, the EmbA 1316a, EmbP 1316p, or EmbN 1316n can be normalized to have unit length in the embedding space using L2 normalization. Thus, the embedding space representations of the eye images are on a hypersphere in the embedding space.

The triplet network architecture can include a triplet loss layer <NUM> configured to compare the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n. The embedding <NUM> learned with the triplet loss layer <NUM> can map eye images of one person onto a single point or a cluster of points in close proximity in the embedding space. The triplet loss layer <NUM> can minimize the distance between eye images of the same person in the embedding space, for example the EmbA 1316a and the EmbP 1316p. The triplet loss layer <NUM> can maximize the distance between eye images of different persons in the embedding space, for example EmbA 1316a, and the EmbN 1316n.

The triplet loss layer <NUM> can compare the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n in a number of ways. For example, the triplet loss layer <NUM> can compare the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n by computing:<MAT> where |EmbA - EmbP| denotes the absolute distance between the EmbA 1316a and the EmbP 1316p in the embedding space, |EmbA - EmbN| denotes the absolute distance between the EmbA 1316a and the EmbN 1316n, and m denotes a margin. The margin can be different in different implementations. For example, the margin can be <NUM> or another number in a range from <NUM> to <NUM>. Thus, in some implementations, the embedding <NUM> can be learned from eye images of a plurality of persons, such that the distance in the embedding space between the eye images from the same person is smaller than the distance in the embedding space between eye images from different persons. In terms of the particular implementation of Equation (<NUM>), the squared distance in the embedding space between all eye images from the same person is small, and the squared distance in the embedding space between a pair of eye images from different persons is large.

The function of the margin m used in comparing the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n can be different in different implementations. For example, the margin m can enforce a margin between each pair of eye images of one person and eye images of all other persons in the embedding space. Accordingly, the embedding space representations of one person's eye images can be clustered closely together in the embedding space. At the same time, the embedding space representations of different persons' eye images can be maintained or maximized. As another example, the margin m can enforce a margin between each pair of images of one person's left eye and images of the person's right eye or eye images of all other persons.

During an iteration of the learning of the embedding <NUM>, the triplet loss layer <NUM> can compare the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n for different numbers of triplets. For example, the triplet loss layer <NUM> can compare the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n for all triplets (EmbA; EmbP; EmbN) in the triplet training set T2. As another example, the triplet loss layer <NUM> can compare the EmbA 1316a, the EmbP 1316p, and EmbN 1316n for a batch of triplets (EmbA; EmbP; EmbN) in the triplet training set T2. The number of triplets in the batch can be different in different implementations. For example, the batch can include <NUM> triplets of (EmbA; EmbP; EmbN). As another example, the batch can include all the triplets (EmbA; EmbP; EmbN) in the triplet training set T2.

During an iteration of learning the embedding <NUM>, the triplet loss layer <NUM> can compare the EmbA 1316a, the EmbP 1316p, and the EmbN 1316n for a batch of triplets (EmbA; EmbP; EmbN) by computing a triplet loss. The triplet loss can be, for example, <MAT> where n denotes the number of triplets in the batch of triplets; and EmbA(i), EmbP(i), and EmbN(i) denotes the ith EmbA 1316a, EmbP 1316p, and EmbN 1316n in the batch of triplets.

During the learning of the embedding <NUM>, the eye authentication trainer <NUM> can update the ENetworkA 1320a, the ENetworkP 1320p, and the ENetworkN 1320n based on the comparison between a batch of triplets (EmbA; EmbP; EmbN), for example the triplet loss between a batch of triplets (EmbA; EmbP; EmbN). The eye authentication trainer <NUM> can update the ENetworkA 1320a, the ENetworkP 1320p, and the ENetworkN 1320n periodically, for example every iteration or every <NUM>,<NUM> iterations. The eye authentication trainer <NUM> can update the ENetworkA 1320a, the ENetworkP 1320p, and the ENetworkN 1320n to optimize the embedding space. Optimizing the embedding space can be different in different implementations. For example, optimizing the embedding space can include minimizing Equation (<NUM>). As another example, optimizing the embedding space can include minimizing the distance between the EmbA 1316a and the EmbP 1316p and maximizing the distance between the EmbA 1316a and the EmbN 1316n.

After iterations of optimizing the embedding space, one or more of the following can be computed: an embedding <NUM> that maps eye images from the higher dimensional eye image space into representations of the eye images in a lower dimensional embedding space; or a threshold value <NUM> for a user device to determine whether the embedding space representation of an user's eye image is similar enough to an authorized user's eye image in the embedding space such that the user should be authenticated as the authorized user. The embedding <NUM> or the threshold value <NUM> can be determined without specifying the features of eye images that can or should use in computing the embedding <NUM> or the threshold value <NUM>.

The threshold value <NUM> can be different in different implementations. For example, the threshold value <NUM> can be the largest distance between eye images of the same person determined from the (ImgA; ImgP; ImgN) triplets during the last iteration of learning the embedding <NUM>. As another example, the threshold value <NUM> can be the median distance between eye images of the same person determined from the (ImgA; ImgP; ImgN) triplets during the last iteration of learning the embedding <NUM>. As yet another example, the threshold value <NUM> can be smaller than the largest distance between eye images of the different persons determined from the (ImgA; ImgP; ImgN) triplets during the last iteration of learning the embedding <NUM>.

The number of iterations required to learn the embedding <NUM> can be different in different implementations. For example, the number of iterations can be <NUM>,<NUM>. As another example, the number of iterations may not be predetermined and can depend on iterations required to learn an embedding <NUM> with satisfactory characteristics such as having an equal error rate (EER) of <NUM>%. As yet another example, the number of iterations can depend on iterations required to obtain a satisfactory triplet loss.

The ability of the embedding <NUM> to distinguish unauthorized users and authorized users can be different in different implementations. For example, the false positive rate (FPR) of the embedding <NUM> can be <NUM>%; and the true positive rate (TPR) of the embedding <NUM> can be <NUM>%. As another example, the false negative rate (FNR) of the embedding <NUM> can be <NUM>%; and the true negative rate (TNR) of the embedding <NUM> can be <NUM>%. The equal error rate (EER) of the embedding <NUM> can be <NUM>%, for example.

In some embodiments, a user device can be, or can be included, in a wearable display device, which may advantageously provide a more immersive virtual reality (VR), augmented reality (AR), or mixed reality (MR) experience, where digitally reproduced images or portions thereof are presented to a wearer in a manner wherein they seem to be, or may be perceived as, real.

Without being limited by theory, it is believed that the human eye typically can interpret a finite number of depth planes to provide depth perception. Consequently, a highly believable simulation of perceived depth may be achieved by providing, to the eye, different presentations of an image corresponding to each of these limited number of depth planes. For example, displays containing a stack of waveguides may be configured to be worn positioned in front of the eyes of a user, or viewer. The stack of waveguides may be utilized to provide three-dimensional perception to the eye/brain by using a plurality of waveguides to direct light from an image injection device (e.g., discrete displays or output ends of a multiplexed display which pipe image information via one or more optical fibers) to the viewer's eye at particular angles (and amounts of divergence) corresponding to the depth plane associated with a particular waveguide.

In some embodiments, two stacks of waveguides, one for each eye of a viewer, may be utilized to provide different images to each eye. As one example, an augmented reality scene may be such that a wearer of an AR technology sees a real-world park-like setting featuring people, trees, buildings in the background, and a concrete platform. In addition to these items, the wearer of the AR technology may also perceive that he "sees" a robot statue standing upon the real-world platform, and a cartoon-like avatar character flying by which seems to be a personification of a bumble bee, even though the robot statue and the bumble bee do not exist in the real world. The stack(s) of waveguides may be used to generate a light field corresponding to an input image and in some implementations, the wearable display comprises a wearable light field display. Examples of wearable display device and waveguide stacks for providing light field images are described in <CIT>.

<FIG> illustrates an example of a wearable display system <NUM> that can be used to present a VR, AR, or MR experience to a display system wearer or viewer <NUM>. The wearable display system <NUM> may be programmed to perform any of the applications or embodiments described herein (e.g., eye image segmentation, eye image quality estimation, pupil contour determination, or iris contour determination). The display system <NUM> includes a display <NUM>, and various mechanical and electronic modules and systems to support the functioning of that display <NUM>. The display <NUM> may be coupled to a frame <NUM>, which is wearable by the display system wearer or viewer <NUM> and which is configured to position the display <NUM> in front of the eyes of the wearer <NUM>. The display <NUM> may be a light field display. A speaker <NUM> is coupled to the frame <NUM> and positioned adjacent the ear canal of the user, another speaker, not shown, is positioned adjacent the other ear canal of the user to provide for stereo/shapeable sound control. The display <NUM> is operatively coupled <NUM>, such as by a wired lead or wireless connectivity, to a local data processing module <NUM> which may be mounted in a variety of configurations, such as fixedly attached to the frame <NUM>, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user <NUM> (e.g., in a backpack-style configuration, in a belt-coupling style configuration).

The local processing and data module <NUM> may comprise a hardware processor, as well as non-transitory digital memory, such as non-volatile memory e.g., flash memory, both of which may be utilized to assist in the processing, caching, and storage of data. The data include data (a) captured from sensors (which may be, e.g., operatively coupled to the frame <NUM> or otherwise attached to the wearer <NUM>), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros; and/or (b) acquired and/or processed using remote processing module <NUM> and/or remote data repository <NUM>, possibly for passage to the display <NUM> after such processing or retrieval. The local processing and data module <NUM> may be operatively coupled to the remote processing module <NUM> and remote data repository <NUM> by communication links <NUM>, <NUM>, such as via a wired or wireless communication links, such that these remote modules <NUM>, <NUM> are operatively coupled to each other and available as resources to the local processing and data module <NUM>. The image capture device(s) can be used to capture the eye images used in the eye image segmentation, eye image quality estimation, pupil contour determination, or iris contour determination procedures.

The remote processing module <NUM> may comprise one or more processors configured to analyze and process data and/or image information such as video information captured by an image capture device. The video data may be stored locally in the local processing and data module <NUM> and/or in the remote data repository <NUM>. The remote data repository <NUM> may comprise a digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration. All data is stored and all computations are performed in the local processing and data module <NUM>, allowing fully autonomous use from a remote module.

In some implementations, the local processing and data module <NUM> and/or the remote processing module <NUM> are programmed to perform embodiments of eye image segmentation, eye image quality estimation, pupil contour determination, or iris contour determination disclosed herein. For example, the local processing and data module <NUM> and/or the remote processing module <NUM> can be programmed to perform embodiments of the processes <NUM>, <NUM>, <NUM>, or <NUM> described with reference to <FIG>, <FIG>, <FIG>, or <FIG>. The local processing and data module <NUM> and/or the remote processing module <NUM> can be programmed to use the eye image segmentation, eye image quality estimation, pupil contour determination, or iris contour determination techniques disclosed herein in biometric extraction, for example to identify or authenticate the identity of the wearer <NUM>. The image capture device can capture video for a particular application (e.g., video of the wearer's eye for an eye-tracking application or video of a wearer's hand or finger for a gesture identification application). The video can be analyzed using the CNN <NUM> by one or both of the processing modules <NUM>, <NUM>. In some cases, off-loading at least some of the eye image segmentation, eye image quality estimation, pupil contour determination, or iris contour determination to a remote processing module (e.g., in the "cloud") may improve efficiency or speed of the computations. The parameters of the CNN <NUM> (e.g., weights, bias terms, subsampling factors for pooling layers, number and size of kernels in different layers, number of feature maps, etc.) can be stored in data modules <NUM> and/or <NUM>.

The results of the video analysis (e.g., the output of the CNN <NUM>) can be used by one or both of the processing modules <NUM>, <NUM> for additional operations or processing. For example, in various CNN applications, biometric identification, eye-tracking, recognition or classification of gestures, objects, poses, etc. may be used by the wearable display system <NUM>. For example, video of the wearer's eye(s) can be used for eye image segmentation or image quality estimation, which, in turn, can be used by the processing modules <NUM>, <NUM> for iris contour determination or pupil contour determination of the wearer <NUM> through the display <NUM>. The processing modules <NUM>, <NUM> of the wearable display system <NUM> can be programmed with one or more embodiments of eye image segmentation, eye image quality estimation, pupil contour determination, or iris contour determination to perform any of the video or image processing applications described herein.

Embodiments of the CNN <NUM> can be used to segment eye images and provide image quality estimation in other biometric applications. For example, an eye scanner in a biometric security system (such as, e.g., those used at transportation depots such as airports, train stations, etc., or in secure facilities) that is used to scan and analyze the eyes of users (such as, e.g., passengers or workers at the secure facility) can include an eye-imaging camera and hardware programmed to process eye images using embodiments of the CNN <NUM>. Other applications of the CNN <NUM> are possible such as for biometric identification (e.g., generating iris codes), eye gaze tracking, and so forth.

Each of the processes, methods, and algorithms described herein and/or depicted in the attached figures may be embodied in, and fully or partially automated by, code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuitry, and/or electronic hardware configured to execute specific and particular computer instructions. For example, computing systems can include general purpose computers (e.g., servers) programmed with specific computer instructions or special purpose computers, special purpose circuitry, and so forth. A code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some implementations, particular operations and methods may be performed by circuitry that is specific to a given function.

Further, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technically complex that application-specific hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be necessary to perform the functionality, for example, due to the volume or complexity of the calculations involved or to provide results substantially in real-time. For example, a video may include many frames, with each frame having millions of pixels, and specifically programmed computer hardware is necessary to process the video data to provide a desired image processing task (e.g., eye image segmentation and quality estimation using the CNN <NUM> with the merged architecture) or application in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disc, volatile or non-volatile storage, combinations of the same and/or the like. The methods and modules (or data) may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored, persistently or otherwise, in any type of non-transitory, tangible computer storage or may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific functions (e.g., logical or arithmetical) or steps in the process. The various processes, blocks, states, steps, or functionalities can be combined, rearranged, added to, deleted from, modified, or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionalities described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states relating thereto can be performed in other sequences that are appropriate, for example, in serial, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed example embodiments. Moreover, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LAN), wide area networks (WAN), personal area networks (PAN), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network may be a wired or a wireless network or any other type of communication network.

The systems and methods of the disclosure each have several innovative aspects, no single one of which is solely responsible or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another, or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with this disclosure, the principles and the novel features disclosed herein.

Certain features that are described in this specification in the context of separate implementations also can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation also can be implemented in multiple implementations separately or in any suitable subcombination.

Conditional language used herein, such as, among others, "can," "could," "might," "may," "e.g.," and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. In addition, the articles "a," "an," and "the" as used in this application and the appended claims are to be construed to mean "one or more" or "at least one" unless specified otherwise.

Claim 1:
A system (<NUM>) for eye image segmentation and image quality estimation, the system comprising:
an eye-imaging camera configured to obtain an eye image (<NUM>);
non-transitory memory configured to store the eye image (<NUM>);
a hardware processor (<NUM>) in communication with the non-transitory memory, the hardware processor programmed to
receive (<NUM>) the eye image (<NUM>), and
process the eye image (<NUM>) using a convolution neural network (<NUM>) to generate a segmentation of the eye image (<NUM>) and quality estimation of the eye image (<NUM>);
wherein the convolution neural network comprises a segmentation tower (<NUM>) to generate the segmentation of the eye image (<NUM>) and a quality estimation tower (<NUM>) to generate the quality estimation of the eye image (<NUM>),
wherein the segmentation tower (<NUM>) comprises segmentation layers (<NUM>) and shared layers (<NUM>),
wherein the quality estimation tower (<NUM>) comprises quality estimation layers (<NUM>) and the shared layers (<NUM>),
wherein a first output layer (330a) of the shared layers (<NUM>) is connected to a first input layer (332a) of the segmentation layers (<NUM>) and to a second input layer (336a) of the segmentation layers (<NUM>), the second input layer (336a) comprising a concatenation layer (336a) that concatenates input from the first output layer (330a) of the shared layers (<NUM>) and at least one other layer (334a) of the segmentation layers (<NUM>),
wherein the first output layer (330a) of the shared layers (<NUM>) is connected to an input layer (368a) of the quality estimation layers (<NUM>),
wherein an output of a first intermediate layer (324a) of the shared layer (<NUM>) is connected to a third input layer (344a) of the segmentation layers (<NUM>), and
wherein the eye image (<NUM>) is received by an input layer (302a) of the shared layers (<NUM>).