Patent ID: 12200233

The appended drawings are not necessarily to scale and may present a somewhat simplified representation of various preferred features of the present disclosure as disclosed herein, including, for example, specific dimensions, orientations, locations, and shapes. Details associated with such features will be determined in part by the particular intended application and use environment.

DETAILED DESCRIPTION

The present disclosure is susceptible of embodiments in many different forms. Representative examples of the disclosure are shown in the drawings and described herein in detail as non-limiting examples of the disclosed principles. To that end, elements and limitations described in the Abstract, Introduction, Summary, and Detailed Description sections, but not explicitly set forth in the claims, should not be incorporated into the claims, singly or collectively, by implication, inference, or otherwise.

For purposes of the present description, unless specifically disclaimed, use of the singular includes the plural and vice versa, the terms “and” and “or” shall be both conjunctive and disjunctive, and the words “including”, “containing”, “comprising”, “having”, and the like shall mean “including without limitation”. Moreover, words of approximation such as “about”, “almost”, “substantially”, “generally”, “approximately”, etc., may be used herein in the sense of “at, near, or nearly at”, or “within 0-5% of”, or “within acceptable manufacturing tolerances”, or logical combinations thereof. As used herein, a component that is “configured to” perform a specified function is capable of performing the specified function without alteration, rather than merely having potential to perform the specified function after further modification. In other words, the described hardware, when expressly configured to perform the specified function, is specifically selected, created, implemented, utilized, programmed, and/or designed for the purpose of performing the specified function.

Referring to the drawings, the left most digit of a reference number identifies the drawing in which the reference number first appears (e.g., a reference number ‘310’ indicates that the element so numbered is first labeled or first appears inFIG.3). Additionally, elements which have the same reference number, followed by a different letter of the alphabet or other distinctive marking (e.g., an apostrophe), indicate elements which may be the same in structure, operation, or form but may be identified as being in different locations in space or recurring at different points in time (e.g., reference numbers “110a” and “110b” may indicate two different input devices which may be functionally the same, but may be located at different points in a simulation arena).

Vehicles have become computationally advanced and equipped with multiple microcontrollers, sensors, processors, and control systems, including for example, autonomous vehicle and advanced driver assistance systems (AV/ADAS) such as adaptive cruise control, automated parking, automatic brake hold, automatic braking, evasive steering assist, lane keeping assist, adaptive headlights, backup assist, blind spot detection, cross traffic alert, local hazard alert, and rear automatic braking may depend on information obtained from cameras and sensors on a vehicle.

Further, during roadway operation of a vehicle by a vehicle operator, semi-autonomously or fully autonomous, the vehicle may be an observer in a driving scene which includes a driving environment, for example the roadway, surrounding infrastructure, objects, signs, hazards, and other vehicles sharing the roadway collectively referred to herein as objects or targets. Objects may be static, such as road signage, or dynamic, such as another vehicle traversing the roadway. Driver, operator, vehicle operator are terms that are meant to be interchangeable and are not meant to limit the scope of the disclosure.

FIG.1is an illustration of gaze pattern110and gaze pattern120from the perspective of a vehicle operator looking out of a front windshield, according to an embodiment of the present disclosure. Gaze pattern110may contain a single area, for example target area115. Target area115represents a cluster of eye gaze directional points. Further, eye gaze data points are continuous, and may also be referred to as floats. Target area115may represent a vehicle operator's gaze straight ahead, for example when driving down a straight highway. In such a scenario there may be little need to look far right or far left. Further, such a gaze may also infer that traffic may be fairly light as there appears to be little eye gaze direction towards a rearview mirror or side mirrors, if so equipped.

In contrast, gaze pattern120represents four distinct areas. Target area122may represent looking forward, but it is wider than target area115and therefore represents an eye gaze pattern of scanning a wider field of view than that of target area115. Further, gaze pattern120may also include target area128representing the vehicle operator looking at the rearview mirror. In addition, target area124may represent the vehicle operator looking at the left-hand side mirror and target area126may represent the vehicle operator looking at the right-hand side mirror. Accordingly, gaze pattern120does not appear to represent the same pattern, and hence the same driving environment as gaze pattern110. Gaze pattern120may more closely correspond to city driving, or a more congested highway driving scenario where the driver may tend to check all the mirrors on a fairly frequent basis.

Eye gaze direction data is a sequence that may be represented as follows:

(x0,y0),(x1,y1),…,(xt-1,yt-1),(xt,yt)
Where the subscript denotes time and the pair is a point on a two-dimensional plane. The joint distribution from which these points are taken may be approximated as a Gaussian mixture as follows:

fX,Y❘Θ(x,y❘Θ)=∑i=1Mρi⁢fX,Y❘θi(x,y❘θi)
Where ρiare the components weights and the component densities are Gaussian, thus θi=μi, Cirepresents the mean and variance of the ithcomponent.

FIG.2illustrates a single view eye tracker system200, according to an embodiment of the present disclosure. System200may include an inward facing camera component210that may include a camera212directed to the face and eyes213of the vehicle operator. Camera212may be another type of imaging device including but not limited to an optical camera, an infrared imaging device, a light-emitting-diode (LED) device, an ultrasound sensor and the like.

System200may also include an eye tracker component220, an encoder230, a decoder240, and an efficiency estimation component250. In an embodiment, system200may operate in an offline mode where camera212produces an image stream of the vehicle operator's face including the eyes213and forwards that image stream to the eye tracker component220. Eye tracker component220may then analyze the image stream to estimate the direction of gaze of the vehicle operator. Encoder230may then encode the gaze data that may then be transmitted over a bandwidth limited channel to the decoder240and the efficiency estimation component250. Encoder230may include a codebook where the codebook may contain a set of encoding and decoding parameters. The encoding/decoding parameters of a codebook may be optimized for a particular driving scenario. As discussed inFIG.1, gaze pattern110may be associated with driving on a straight highway as there is a single cluster of gaze direction in the middle of gaze pattern110. InFIG.2, if encoder230receives an image stream that contains a similar gaze pattern then the codebook may be well suited with its encoding/decoding parameters to efficiently encode and decode the data for transmitting over a limited bandwidth channel.

However, if the image stream inFIG.2is more consistent with gaze pattern120then the codebook may not efficiently encode and decode the estimated direction of gaze data resulting in a poor or corrupted restoration from the decoder. In such a situation the efficiency estimation component250may report that the present codebook being used in system200is not optimal and that a different codebook may be needed. Such reporting may be done to a back-office system over the cloud or other means of transmission.

FIG.3illustrates a structure of an encoder-decoder300, according to an embodiment of the present disclosure. Encoder310may include a two-dimensional quantizer314, an arithmetic encoder316that may accept input data312, and output data318. Decoder320may include an arithmetic decoder324, a quantizer326, input data322, and output data318:

Two-dimensional quantizer314may partition a two-dimensional space into a number of Voronoi cells and the finding a centroid of each cell including estimating a probability of each cell. This may be done using Lloyd clustering using training data. In addition, another approach may be to approximate the distribution model of the source for the creation of synthetic, e.g., pseudo randomly generated, points, for example, when a training set is minimal making the design of the vector quantization difficult.

The arithmetic encoder316, given a discrete set and its probabilities from the two-dimensional quantizer314, may produce a sequence over this set that can be compressed, for example, with a compression ratio approaching the theoretical limit of the source entropy. The resulting binary sequence, e.g., codewords k, may vary in length, e.g., block to variable encoding. The output data318from the arithmetic encoder316may then be sent to the arithmetic decoder324and received as input data322.

The arithmetic decoder324, may already know the set and probabilities and thus may losslessly decode the binary sequence encoded by the arithmetic encoder316. The result of the decoding may be then be input to the quantizer326to restore the original centroid. The output of the quantizer326, output328, may be an index representing the original Voronoi cell. This index may be taken from a discrete finite set, of size K, where the arithmetic encoder316may encode the index into a binary sequence. The arithmetic decoder324may then receive the codeword output and extract the index of the Voronoi cell. However, the restoration may not result in the exact centroid of the Voronoi cell and thus the overall encode and decoding process is lossy.

Regarding the efficiency estimation component250, the efficiency estimation is targeted to assess the efficiency of the communication of the encoder/decoder process. The efficiency estimation component250may not have any knowledge of the specific codebook being used in the system200, but it is known that the number of Voronoi cells in the quantization are bounded by the limit K as used inFIG.3. The method of efficiency estimation may include applying a Context Tree Weighted (CTW) approach for the estimation of the entropy. The estimated normalized entropy may be shown as follows:HCTW(X), where X is the binary sequence communicated.

Further, the efficiency of the communication may be determined by:

1log⁢K-HCTW(X)
However, if HCTW(X)≥log K then the estimation error may be too large and thus may be disregarded.

FIG.4is an illustration of an offline dual view eye tracker system400, according to an embodiment of the present disclosure. System400may include an inward facing camera component410that may include a camera412directed to the face and eyes413of the vehicle operator. Camera412may be any type of imaging device including but not limited to an optical camera, an infrared imaging device, a light-emitting-diode (LED) device and the like. Camera412may also be another type of sensor, such as a sound recording device or movement sensor.

System400may also include an eye tracker component420, an encoder430, a decoder440, an efficiency estimation component450, and an outward facing camera component460, a situation awareness component470, and a codebook selection component480. Outward facing camera component460may also include an outward facing camera462directed towards the surrounding environment, for example a straight roadway464or a curved roadway466. The surrounding environment may include other attributes, for example an intersection, other vehicles, buildings, people, or other object. Further, inward facing camera412and outward facing camera462may include additional image capture devices facing inwards and/or outwards. For example, outward facing camera component460may include multiple image capture devices situated around the outside of a vehicle to provide a three-hundred-sixty degree view. In a similar manner, inward facing camera component410may include multiple image capture devices and controllers situated around the inside of a vehicle. Further, in addition to including additional image capture device inside and outside of the vehicle, in an embodiment, a single image capture device producing a video stream may include multiple controllers and processors where each controller or processor may be dedicated to a specific function regarding the video stream. For example, processor to analyze a soundtrack associated with the video stream.

System400may include where outward facing camera462sends one or more video streams to the situation awareness component470. The situation awareness component470may use a variety of sensors and controllers in the vehicle to analyze the surrounding scene in the vicinity of the vehicle. Based on such an analysis, the situation awareness component470may select the most similar scene from a fixed list of known scenes. For example, outward facing camera462may determine that the scene in front of the vehicle is a straight highway, such as straight roadway464and thus may select a “straight roadway” scene from a list of known scenes. Further, situation awareness component470may use other data in analyzing surrounding scenes and selecting the most appropriate scene from the fixed list of known scenes. The other data may include global positioning system data, for example, location and speed to indicate the vehicle is in an urban or rural area and thus may have different codebooks. In addition, different vehicle speeds may also determine a selection of a different scene and thus different codebooks by codebook selection component480.

Additional data sources for the situation awareness component470, which may also be referred to as a controller, may include components such as an inertial measurement unit that may indicate acceleration or deceleration where the codebook may be adjusted to fit the scenario, or lateral acceleration to indicate steering wheel turns. Pedal control positions or turn signal may also be used, for example a left turn may indicate a particular codebook while a right turn would necessitate a different codebook. Object sensors such as radar may call for different codebooks for different kinds of objects in the scene, for example, animals, pedestrians, trucks, or emergency vehicles to name a few. Or, the existence of free space when the vehicle may not move due to other vehicles blocking its path, such as in a parking lot where the free space may be located in any direction relative to the vehicles direction of travel.

The situation awareness component470may then forward its selected scene, from a list of fixed, predefined scenes to the codebook selection component480. Codebook selection component480may then, based on the selected scene, select a codebook from a set of possible codebooks that best matches the selected predefined scene. The codebook selection component480may then provide the selected codebook to both encoder430and decoder440.

Canonical state signals from the situation awareness component470may be used by the codebook selection component480to assess the mean and variance of distributions, O, and the number of Gaussian distributions in the model mixture, M. Assuming a set of quantizers, based on different Gaussian mixture models, the best, or closest, fit for the parameters may be found. Then, the index of the selected codebook may be communicated by the codebook selection component480to the decoder440as discrete values at low frequencies. The chosen quantizer may then be used by both encoder430and decoder440of the next T eye gaze direction data points.

The inward facing camera component410may include a camera412directed to the face and eyes413of the vehicle operator. Camera412may produce an image stream of the vehicle operator's face include the eyes413and forwards that image stream to the eye tracker component420. Eye tracker component420may then analyze the image stream to estimate the direction of gaze of the vehicle operator. The estimation direction of gaze may then be sent to encoder430for encoding using the codebook selected by codebook selection component480as previously discussed. The encoded data may then be transmitted, e.g., using a transmitter located within the vehicle, over the bandwidth limited channel by encoder430to the decoder440for decoding based on the codebook sent by the codebook selection component480. Encoder430may also send the encoded data to efficiency estimation component450where, if the efficiency of the encoding/decoding process is not as expected, e.g., less than a predetermined threshold, the efficiency estimation component450may signal to the codebook selection component480to select a different codebook to be used by encoder430and decoder440with the goal to increase efficiency, as discussed inFIG.2.

Further, a driver state may act as an additional input factor in the selection of a codebook as shown by input485. Driver state scores may be associated with risk and workload that may have a significant effect on gaze patterns selection.

FIG.5is an illustration of an offline dual view multi-attribute tracker system500, according to an embodiment of the present disclosure. System500may include a number of the same components as discussed inFIG.4. For example, system500may include the eye tracker component420, the encoder430, the decoder440, the efficiency estimation component450, the outward facing camera component460, the situation awareness component470, and the codebook selection component480. Outward facing camera component460may also include an outward facing camera462directed towards the surrounding environment, for example a straight roadway464or a curved roadway466.

System500may include some additional components, for example an inward facing camera512, a hand position controller522directed to capture a position and movement of hand515, a head orientation controller524directed to capture a position and movement of head517, and a body pose controller526directed to capture a position and movement of body519of the vehicle operator.

Inward facing camera512may include multiple imaging devices creating multiple video streams. Or alternately, inward facing camera512may include a single imaging device with multiple controllers situated to analyze different aspects of the vehicle operator. The hand position controller may track a position and movement of hand515. The head orientation controller524may track a position and movement of head517. The body pose controller may track a position and movement of body519of the vehicle operator. Eye gaze, as discussed inFIG.4, may be a significant indicator of behavior in a situation, but other actions of a vehicle operator may provide additional insight. Accordingly, eye tracker component420, hand position controller522, head orientation controller524, and body pose controller526may output their data to encoder to be processed as discussed inFIG.4. In addition to tracking eye, hand, head orientation, and body position, the use of images and sequences of images may also be associated with a situation, for example, the turning of a head from center to right may be an indication of a right turn in a vehicle.

FIGS.2-5have primarily been directed to an off-line approach of context-based encoding of eye and body gaze. However, as an alternative, an on-line approach that includes designing specific codebooks rather than predefined codebooks may also be used. In addition, either off-line or on-line methods may also include further improvement by a use of decoding of sequences utilizing a delta encoding approach.

As previously discussed, canonical state signals from the situation awareness component470may be used by the codebook selection component480to assess the mean and variance of distributions, Θ, and the number of Gaussian distributions in the model mixture, M. Using this approach, the canonic state signals may be communicated to decoder440. As such communication may be done at lower frequencies, a reasonable resolution with limited averaged rate may be achieved. Given the canonical state signals and the L previously communicated signals, centroids may be represented as:

(x^t-1-L,y^t-1-L),(x^t-L,y^t-L),…,(x^t-1,y^t-1)

Both encoder430and decoder440may estimate a Generalized Method of Moments (GMM) and design a two-dimensional quantizer, based on estimation maximization using canonical state signals, and a vector quantizer design based on Voronoi cells, centroids, and probabilities.

A standard approach to the estimation of a GMM may be the estimation-maximization (EM) method where this method may be used to monotonically increase the log-likelihood. Thus, assume the L previous data point may be denoted as xn, for n∈{1, 2, . . . L}. Accordingly, for the GMM the EM algorithm iteratively yields estimates as follows:

ρi(k+1)=1L⁢∑n=1Lvi(k)(n)μi(k+1)=Σn=1L⁢vi(k)(n)⁢xnΣn=1L⁢vi(k)(n)Ci(k+1)=Σn=1L⁢vi(k)(n)⁢(xn-μi(k+1))⁢(xn-μi(k+1))Σn=1L⁢vi(k)(n)vi(k)(n)=ρi(k)⁢fX❘μi(k),Ci(k)(xn❘μi(k),Ci(k))Σj=1M⁢ρj(k)⁢fX❘μj(k),Cj(k)(xn❘μj(k),Cj(k))

The canonical state signals may be used in two ways. First, assuming an assessment of the mean and/or variance and/or component weights and/or M are exact and perform the estimation with these values are constants. Second, the assessment may be used as an initial educated guess of the values. However, in both cases the approach is bound to improve a convergence rate and an ultimate final estimation.

FIG.6is an illustration of a delta encoding method600, according to an embodiment of the present disclosure. A delta encoding may also be used for an on-line approach of context-based encoding of eye and body gaze. Eye-gaze data points may be initially thought to be independent, but in actuality may be considered highly dependent. In such a situation delta encoding may be more beneficial. One approach may be to use a dynamic delta encoding approach that may facilitate both an off-line as well as an on-line approach. Method600may include an input605that may be sent to a predictor610. Input605may include a set of data points prior to time t, e.g., ({circumflex over (x)}0, ŷ0), ({circumflex over (x)}1, ŷ1), . . . , ({circumflex over (x)}t-1, ŷt-1), where predictor610then generates a predicted value612for time t. Comparator620then may then compare the predicted values612with the actual value615(xt, yt) and outputs the difference as an error amount errort. Truncator630may then truncate errortshown aserrort-1, in an effort to minimize data transmission from truncator630. Thus, when the error is small enough a single bit flag may be transmitted. Decision640may determine that if the truncated error is greater than a threshold amount then the actual (xt, yt) value is sent. If the truncated error is less than the threshold amount then the actual truncated error amount,errort-1, is sent.

FIG.7illustrates a detail flowchart of a method700for limited rate context-based eye gaze encoding, according to an embodiment of the present disclosure. Step705may include capturing and sending, using an outward looking camera situated in a vehicle, a first video stream of a surrounding environment to a controller. As discussed inFIG.4, system400may include an outward facing camera462directed towards the surrounding environment, for example a straight roadway464or a curved roadway466. The surrounding environment may include other attributes, for example an intersection, other vehicles, buildings, people, or other object. Outward facing camera component460may include multiple image capture devices situated around the outside of a vehicle to provide a three-hundred-sixty-degree view.

Step710may include generating, by the controller, a scene description based on the first video stream. Step715may include selecting, by the controller, based on the scene description, a corresponding scene from a predefined list of known scenes. As discussed inFIG.4, the situation awareness component470may use a variety of sensors and controllers in the vehicle to analyze the surrounding scene in the vicinity of the vehicle. Based on such an analysis, the situation awareness component470may select the most similar scene from a fixed list of known scenes. For example, outward facing camera462may determine that the scene in front of the vehicle is a straight highway, such as straight roadway464and thus may select a “straight roadway” scene from a list of known scenes. Further, the situation awareness component470may use other data in analyzing surrounding scenes and selecting the most appropriate scene from the fixed list of known scenes.

Step720may include selecting, by the controller, based on the selected corresponding scene, a codebook of encoding and decoding parameters from a plurality of predefined codebooks. As discussed inFIG.4, the situation awareness component470may then forward its selected scene, from a list of fixed, predefined scenes to the codebook selection component480. Codebook selection component480may then, based on the selected scene, select a codebook from a set of possible codebooks.

Step725may include capturing and sending, using an inward looking camera situated in the vehicle, a second video stream of a face of a driver to an eye tracker controller. As discussed inFIG.4, the inward facing camera component410may include a camera412directed to the face and eyes413of the vehicle operator, where camera412may produce an image stream of the vehicle operator's face include the eyes413and forwards that image stream to the eye tracker component420.

Step730may include estimating, by the eye tracker controller, based on the second video stream, a gaze direction of the driver. As discussed inFIG.4, eye tracker component420may then analyze the image stream to estimate the direction of gaze of the vehicle operator.

Step735may include outputting, by an encoder, based on the selected codebook, an encoded data comprising the estimated gaze direction of the driver. As discussed inFIG.4, the situation awareness component470may then forward its selected scene, from a list of fixed, predefined scenes to the codebook selection component480. Codebook selection component480may then, based on the selected scene, select a codebook from a set of possible codebooks that best matches the selected predefined scene. The codebook selection component480may then provide the selected codebook to both encoder430and decoder440.

Step740may include sending, by a transmitter within the vehicle, the encoded data over a bandwidth limited channel to a decoder. As discussed inFIG.4, the encoded data may then be transmitted, e.g., using a transmitter located within the vehicle, over the bandwidth limited channel by encoder430to the decoder440for decoding based on the codebook sent by the codebook selection component480.

Step745may include decoding, by the decoder, based on the selected codebook, the encoded data. As discussed inFIG.4, the encoder430may also send the encoded data to efficiency estimation component450where, if the efficiency of the encoding/decoding process is not as expected, e.g., less than a predetermined threshold, the efficiency estimation component450may signal to the codebook selection component480to select a different codebook to be used by encoder430and decoder440with the goal to increase efficiency, as discussed inFIG.2.

Method700may then end.

The description and abstract sections may set forth one or more embodiments of the present disclosure as contemplated by the inventor(s), and thus, are not intended to limit the present disclosure and the appended claims.

Embodiments of the present disclosure have been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries may be defined so long as the specified functions and relationships thereof may be appropriately performed.

The foregoing description of the specific embodiments will so fully reveal the general nature of the disclosure that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present disclosure. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance.

The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments.

Exemplary embodiments of the present disclosure have been presented. The disclosure is not limited to these examples. These examples are presented herein for purposes of illustration, and not limitation. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosure.