Systems and methods for tracking user interaction with objects based upon eye movements

One or more computing devices, systems, and/or methods for tracking user interaction with objects based upon eye gazing movements are provided. A canvas is populated with grids corresponding to locations of objects and is populated with an eye cursor positioned to track a gaze of a user. A plurality of image frames captured by a camera are evaluated to track interaction of users with the objects. For an image frame, a midpoint between eyes of a user are identified based upon facial landmarks of the user. A current position of the eye cursor is set within the canvas based upon the midpoint. In response to detecting that the eye cursor intersects a first grid in the canvas, the user is identified as interacting with a first object associated with the first grid.

BACKGROUND

Cameras are used to obtain various types of information that may be used to improve the operation of devices. For example, a camera may capture imagery of a location, such as the inside of a store. The imagery may depict a user within the store. Accordingly, the imagery may be evaluated to track information related to the user. This information may be used to track the presence, movement, and/or actions of the user while in the store, such as for security reasons or customer tracking purposes.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Subject matter will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific example embodiments. This description is not intended as an extensive or detailed discussion of known concepts. Details that are well known may have been omitted, or may be handled in summary fashion.

One or more systems and/or techniques for tracking user interaction with objects based upon eye gazing movements are provided. In order to track eye gazing movements, one or more cameras may be positioned so that the cameras can capture image frames of where the objects are located, such as of a table where smart watches, headphones, tablets, laptops, food, books, items in a store, bikes, items within a home, outdoor objects such as a tree or trash can, and/or any other objects or devices are located. These image frames may be processed using a model that has been trained to identify facial landmarks of users, such as where a user walks up to the table and starts to look at the devices on the table. The facial landmarks may correspond to x, y, z coordinate points on a face of the user. The facial landmarks, such as facial landmarks of irises and pupils of the user, may be used to determine a midpoint between the eyes of the user. This midpoint is used to track where the user is gazing and eye gazing movements. In this way, the midpoint is used to detect whether the user is interacting with (e.g., gazing at, touching, picking up, trying on, etc.) the devices.

Various analytics and actions may be performed based upon whether users are interacting with objects. In an example, an interest or disinterest of the user with one of the devices may be determined based upon whether and/or how the user interacted with the object/device (e.g., the user picked up a book, the user put a grocery item in a shopping cart, the user turned on a stove in their kitchen, etc.) or ignored the object/device (e.g., the user walked past a bike section of a store without looking at the bikes). In another example, if a determination is made that the user is interacting with a device (e.g., a real-time determination that the user is currently trying on a pair of headphones), a command may be generated and transmitted to the device to modify operation of the device (e.g., turn on the pair of headphones to play music for the user).

Modifying operation of objects such as electronic devices based upon real-time detection of users interacting with the electronic devices can improve operation of the electronic devices by controlling the electronic devices based upon a current scenario of whether the electronic devices are being interacted with or not. For example, this can also improve battery and power consumption where the electronic devices can be turned off or placed into a low power mode when a determination is made that there are no users interacting with or looking at the electronic devices. The electronic device can then be turned on and/or controlled (e.g., controlling a smart watch to play a demo when a user puts on the smart watch) in real-time based upon a determination that a user is interacting with or looking at the electronic device.

FIG.1illustrates an example of a system100for tracking user interaction with objects based upon eye gazing movements. A platform101may be configured to track user interactions with the objects. In some embodiments, the platform101may be implemented as a cross platform application that may run on a server, a laptop, a smart phone, a smart watch, a tablet, a television, a container such as a Kubernetes container, hardware, software, or combination thereof. In some embodiments, the platform101may be connected to a camera122, integrated into the camera122, or capable of accessing storage within which the camera122stores image frames. The platform101may be configured to process image frames from imagery103captured by the camera122and/or other cameras, such as imagery152from a camera154, in order to track interactions of users with objects depicted by the image frames.

The camera122may be oriented towards a surface114on which one or more objects are located. For example, a laptop116, headphones118, and a mobile device120may be positioned on the surface114. The camera122may be oriented (e.g., located between about 3 feet and about 10 feet from the surface114or any other distance) such that the camera122can capture the imagery103of the surface114, the objects on the surface114, and users nearby the surface114. For example, the camera122may capture the imagery103comprising image frames of a user102nearby the surface114. The platform101may analyze these image frames using a model to determine whether the user102is interacting with any of the objects, such as gazing at one of the objects. The model may have been trained to identify landmarks of humans, such as facial landmarks of noses, lips, eyes, and/or other facial features, body feature for pose and movement detection, etc. The landmarks may corresponds to x, y, z coordinates of these features.

Using the facial features, the platform101can identify a left eye106and a right eye104of the user102, such as a left pupil110or left iris and a right pupil108or right iris. The platform101is configured to calculate a midpoint113along a calculated line112between the left pupil110and the right pupil108. The midpoint113corresponds to a current gaze of the user102, which can be used to track eye gaze movements of the user102. In this way, the midpoint113and changes in location of the midpoint113can be used to determine whether the user102is interacting with, such as looking at, an object from the surface114. If the platform101detects that the user102is interacting with an object such as looking at the mobile device120on the surface114, then the platform101may track an interest of the user102in the mobile device120. The platform101may perform an action based upon the interest, such as by transmitting a command to the mobile device120to turn on or wake up the mobile device120and execute a demo application on the mobile device120, which may prompt the user102to pick up and try out the mobile device120.

FIG.2illustrates an example of a method200for tracking user interaction with objects based upon eye gazing movements, which is further described in conjunction with system300ofFIGS.3A and3B. A platform301may be configured to track user interactions with objects on a table306. The objects may correspond to any objects such as headphones310, a mobile device312, a laptop308, and/or a wide variety of other types of objects (e.g., items for sale within a store such as grocery items, items within a building such as a door or desk, items within an outdoor location such as a trash can or water fountain, items within a home such as an appliance or a book, etc.). The platform301is configured to process image frames of imagery303captured by a camera304and/or other cameras in order to identify and track user interactions with the objects on the table306.

During operation202of method200, a canvas may be generated. In some embodiments, the canvas is a data structure (e.g., an internal data structure used by the platform301to store information) used by the platform301to track data and information used to track and associate user gaze and eye gaze movements with locations of objects. In some embodiments, the canvas is a user interface, such as a graphical user interface. Examples and further details of canvass will be subsequently described in relation toFIGS.5A-5F.

The canvas may be populated with grids corresponding to locations of the objects on the table306. For example, the canvas may be populated with a first grid corresponding to a location of the laptop308, a second grid corresponding to the location of the headphones310, and a third grid corresponding to a location of the mobile device312. In some embodiments, the location, orientation, size, and shape of a grid may be defined and/or redefined/adjusted based upon user input. For example, a user may drag and drop a grid onto the canvas as a user interface element. The user may either input values or use a cursor to drag and drop the canvas in order to position, size, and/or orient the canvas. The user may subsequently resize, relocate, and/or reorient the grids within the canvas. In some embodiments, the location, orientation, size, and shape of a grid may be automatically defined based upon image recognition functionality used to process the imagery303. That is, the image recognition functionality to identify and locate objects on the table306, which is used to define grids representing locations of those objects.

The canvas may be populated with eye cursors for users detected in the image frames. For example, when the platform301identifies the user302within an image frame, an eye cursor representing a gaze of the user302may be populated within the canvas. If the canvas is a graphical user interface, then the eye cursor may be a user interface element. If the canvas is an internal data structure used to track data and information associating user gaze and eye gaze movements with locations of objects, then the eye cursor may be coordinate values of the user's gaze. If more than one user is detected, then multiple eye cursors may be associated with the canvas and used to track gazes of the user.

During operation204of method200, a plurality of image frames, captured by the camera304as the imagery303, may be evaluated by the platform301to track interaction of the user302(and/or other users that are detected) with the objects on the table306. In some embodiments, the platform301may process the image frames in real-time as the camera304captures the imagery303of the user302so that the platform301may implement real-time actions in response to detected user interaction with the objects. Also, the platform301may evaluate the plurality of image frames to track real-time3D object orientations and locations/movement of the objects (e.g., the user302may move or pick up an object).

A model, trained to identify facial landmarks of human faces in images and/or body landmarks of human bodies, may be used to process each image frame to identify facial landmarks, body landmarks, and/or other landmarks of the user. The facial landmarks may correspond to z, y, z coordinates of points on a face of the user302, which may correspond to facial features of the user302. For example, the landmarks may be identified based upon features extracted from an image frame. The features may correspond to iris features, body features, voice features, and/or facial features. The iris features may be used to track iris and head movement of the user302over time. The body features may be used to track body poses of the user302over time, such as the user picking up an object, the user putting on and wearing an object, etc. The voice features and the facial features may be evaluated for tone sentiment, text sentiment, and/or facial emotional expressions, such as where the user302smiles while trying on the headphones310, the user302laughs while typing on the laptop308, the user302looks bored or frowns when gazing at the mobile device312, etc. These features may be used to help determine whether the user302has a positive or negative interest in an object.

Once the facial landmarks have been identified by the platform301using the model, the facial landmarks may be used to identify a left eye and a right eye of the user302, such as a left pupil and a right pupil of the user302. During operation206of method200, a midpoint between the left eye (left pupil) and the right eye (right pupil) may be identified (e.g., a midpoint along a calculated line between the left and right pupils of the user302). The midpoint may be used to represent a gaze of the user302, and thus can be used to determine whether the user302is interacting with (gazing/looking at) an object on the table306.

During operation208of method200, a current position of the eye cursor within the canvas is set based upon the midpoint identified from the image frame. If the canvas is a data structure, then setting the current position may correspond to storing a coordinate value within an eye cursor field in the data structure. If the canvas is a graphical user interface, then the eye cursor may be visually positioned at the coordinate value.

During operation210of method200, the current position of the eye cursor may be compared to the grids within the canvas to determine whether the eye cursor intersects any of the grids (or whether the coordinate value of the eye cursor overlaps coordinate values of a grid where the canvas is a data structure). If the eye cursor intersects a grid, then a determination may be made that the user302is interacting with an object whose location is represented by the grid. For example, the eye cursor may intersect the second grid representing the location of the headphones310(or the coordinate value of the eye cursor overlaps coordinate values of the second grid where the canvas is a data structure), and thus the platform301may determine that the user302is interacting with such as looking/gazing at the headphones310. In some embodiments where the canvas is implemented as the graphical user interface, a display property (e.g., a color, shading, etc.) of the second grid may be modified based upon the eye cursor intersecting with the second grid.

During operation212of method200, the platform301may implement an action based upon a determination that the user302is interacting with the headphones310. In some embodiments, the action may correspond to that generation of tracking information indicating that the user302interacted with the headphones310or has an interest in the headphones310. The action may correspond to generating and transmitting a notification to a remote device that the user302is/was interacting with the headphones310and/or has an interest in the headphones310. The notification may be transmitted and displayed by a remote device that is nearby the table306or the user302(e.g., displayed to an employee of a store where the table306is located). In some embodiments, the action may correspond to tracking a timespan of the interaction by the user302with the headphones310by analyzing multiple image frames, and then outputting a level of interest of the user302in the headphones310based upon the timeframe (e.g., the longer the user302interacts with the headphones310, the higher the interest). In some embodiments, the action may correspond to transmitting a command to the headphones310to modify operation of the headphones310such as to turn on and play music on the headphones310in response to the user interaction corresponding to the user302putting on the headphones310, as illustrated byFIG.3B.

The platform301may analyze user interactions with the objects over time using various analytics in order to generate various outputs. In some embodiments of performing the analytics, a heat map may be generated for the second grid based upon user interaction with the headphones310. The heat map may comprise information that is based upon a length of user interaction, types of user interaction (e.g., a user gazing at the headphones310, a user ignoring the headphones310, a user picking up the headphones310, a user trying out the headphones310, etc.), a number of users interacting with the headphones310, etc. The heat map may indicate a level of interest in the headphones310. In response to the heat map indicating a level of interest that exceeds an interest threshold, an alert about the interest in the headphones310may be created and transmitted to a remote device, such as to the remote device of the employee nearby the user302.

In some embodiments of performing the analytics, an idle time of the user302interacting with the headphones310may be determined based upon a time span of the eye cursor intersecting the second grid. In response to the idle time exceeding a threshold timespan (e.g., the user has gazed at the headphones310for over 8 seconds), an alert about the interest in the headphones310may be created and transmitted to a remote device, such as to the remote device of the employee nearby the user302. In some embodiments of performing the analytics, the alert may be created and transmitted based upon the platform301detecting a threshold amount of changes in movement of the eye cursor over a time frame (e.g., rapid eye movement of the user302may be indicative of a higher interest in the headphones310). In some embodiments of performing the analytics, a pattern of user interaction changes amongst the objects on the table306may be tracked based upon an order and duration of the eye cursor intersecting the grids of the canvas (e.g., the user302may look at the headphones310for 8 seconds, look at the laptop308of 1 second, and return to looking at the headphones310for another 15 seconds). The pattern may be used to determine an interest of the user302, such as a high interest in the headphones310and little to no interest in the laptop308.

FIG.4illustrates a system400for tracking user interaction with objects by multiple users based upon eye gazing movements. A platform401may be configured to process image frames of imagery403detected by a camera404and/or other cameras not illustrated. The image frames may depict users interacting with objects, such as a first object408, a second object410, and a third object412on a surface406. The platform401may be capable of identifying and distinguishing between a first user402and a second user414. In this way, the platform401may process the image frames to detect whether the first user402is interacting with any of the objects, and whether the second user414is interacting with any of the objects.

FIGS.5A-5Fillustrate an example500of a canvas502. The canvas502may be populated with one or more grids, such as a first grid504, a second grid506, a third grid508, and a fourth grid510, as illustrated byFIG.5A. The first grid504may correspond to a location within a room where laptops are located. The second grid506may correspond to a location within the room where phones are located. The third grid508may correspond to a location within the room where tablets are located. The fourth grid510may correspond to a location within the room where watches are located. In some embodiments, the first grid504, the second grid506, the third grid508, and the fourth grid510may be been created and placed within the canvas502by a user. For example, the user may have dragged and dropped grid user interface elements into the canvas502, and may have specified size, shapes, and/or orientations for the grids. In some embodiments, the first grid504, the second grid506, the third grid508, and the fourth grid510may have been automatically and/or programmatically created based upon a platform evaluating imagery of the room. Features of laptops depicted by the imagery may be extracted and used to create the first grid504. Features of phones depicted by the imagery may be extracted and used to create the second grid506. Features of tablets depicted by the imagery may be extracted and used to create the third grid508. Features of watches depicted by the imagery may be extracted and used to create the fourth grid510.

The canvas502may be populated with an eye cursor512. The eye cursor512may represent a current gaze location of a user depicted in image frames being processed by the platform. At a first point in time, the platform may process a first image frame to determine a midpoint between eyes of the user at the first point in time. A current position of the eye cursor512may be set to a first current position within the canvas502based upon the midpoint at the first point in time (e.g., the user may be looking towards the laptops), as illustrated byFIG.5B.

At a second point in time, the platform may process a second image frame to determine a midpoint between eyes of the user at the second point in time. A current position of the eye cursor512may be set to a second current position within the canvas502based upon the midpoint at the second point in time (e.g., the user may be looking at a particular laptop), as illustrated byFIG.5C. Because the eye cursor512(e.g., a pupil representation of the eye cursor512) now intersects the first grid504, the platform may determine that the user is interacting with the laptop such as looking at the laptop. In some embodiments, the platform may modify a display property of the first grid504(e.g., changing a fill color, charging a shading property, bolding text “laptops,” increasing a line size of an outline of the first grid504, etc.) based upon the intersection.

At a third point in time, the platform may process a third image frame to determine a midpoint between eyes of the user at the third point in time. A current position of the eye cursor512may be set to a third current position within the canvas502based upon the midpoint at the third point in time (e.g., the user may be looking towards the phones), as illustrated byFIG.5D.

At a fourth point in time, the platform may process a fourth image frame to determine a midpoint between eyes of the user at the fourth point in time. A current position of the eye cursor512may be set to a fourth current position within the canvas502based upon the midpoint at the fourth point in time (e.g., the user may be looking at a particular phone), as illustrated byFIG.5E. Because the eye cursor512(e.g., a pupil representation of the eye cursor512) now intersects the second grid506, the platform may determine that the user is interacting with the phone such as by picking up the phone. In some embodiments, the platform may modify a display property of the second grid506(e.g., changing a fill color, charging a shading property, bolding text “phones,” increasing a line size of an outline of the second grid506, etc.) based upon the intersection.

At a fifth point in time, the platform may process a fifth image frame to determine a first midpoint between eyes of the user and a second midpoint between eyes of a second user at the fifth point in time. A current position of the eye cursor512may be set to a fifth current position within the canvas502based upon the first midpoint at the fifth point in time (e.g., the user may still be holding the phone), as illustrated byFIG.5F. Because the eye cursor512(e.g., a pupil representation of the eye cursor512) intersects the second grid506, the platform may determine that the user is still interacting with the phone. In some embodiments, the platform may modify a display property of the second grid506based upon the intersection. The canvas502may be populated with a second eye cursor520representing a gaze of the second user. A current position of the second eye cursor520may be set to a current position within the canvas502based upon the second midpoint at the fifth point in time (e.g., the second user may be looking at a tablet). Because the second eye cursor520(e.g., a pupil representation of the second eye cursor520) intersects the third grid508, the platform may determine that the second user is looking at the tablet. In some embodiments, the platform may modify a display property of the third grid508based upon the intersection.

In some embodiments, a platform may be used to authenticate users. For example, the platform may evaluate image frames of imagery captured by a camera to determine whether a user has interacted with objects according to an authentication code. In particular, the authentication code may be generated to specify a particular ordering with which a user is to interact with a particular set of objects. For example, a user may be authenticated if the user first touches their nose, then holds up 3 fingers, and finally blinks twice at a mobile device in order to unlock the mobile device. The objects may correspond to body parts of the user or may correspond to other types of objects (e.g., the user must first touch a box, then hold a pen, and finally stare at a keyboard for at least 4 seconds). In this way, the user may authenticate using the authentication code by interacting with one or more objects according to a particular order of interactions. Various actions may be performed once the user is authenticated, such as granting the user access to a device, an application, a website, an account, a door, a vehicle, etc.

In some embodiments, object tracking and gaze detection is implemented by a platform. In an example, while a user is gazing at a section on a table and picks up an object, the platform detects the object to understand the type and name of the object that the user is holding. The platform also understands the actions of the user with respect to the object by implementing pose detection. For example, if the user is picking up a mobile phone and looks around at the mobile phone, then then platform calibrates a pose of the user along with object (the mobile phone) that the user is holding. The platform generates a notification about the user's interest in the mobile phone. The platform also details out to which areas the user is gazing as the platform understands a relative distance between irises of the user to the object and an orientation of the object with respect to a screen. Similarly, if the user is lifting a headphone and brings the headphone closer to the ears of the user, then another trigger is generated for understanding the pose of the user in that the user is holding a headphone as the user brings the headphone nearer to the ears. This trigger can be used to trigger various actions such as to turn on music to automatically play through the headphone. Similarly, if the user is trying to wear a smart watch on a wrist, then a wrist pose is detected and this generates a trigger about the user's interest in trying out the smart watch. Displacement of a product from one grid to other grid is also tracked when the user lifts a product from one grid/section and displaces the product to another grid/section. This implies that the user is interested in multiple products at each of these grids/sections.

In some embodiments, real-time3D object orientation is detected by the platform. In an example of mobile phone pose detection, a neural network may be trained to recognize parts of the human body. Landmarks may be assigned using the neural network to the connected parts of a user's body depicted by an image frame. Real-world coordinates of the landmarks may be tracked over time from image frames. A distance (e.g., Euclidean) between a wrist and an eye view point (a midpoint between eyes of a user) may be calculated. An object (a mobile phone) may be detected, and a distance between the object and the eye view point may be calculated. A pose of the user may be classified. The classified pose can be stored, and a slope of a finger can be stored in memory. In this way, the pose of the user may be tracked over time to track user interaction with the mobile phone.

In an example of headphone pose detection, a neural network is trained to recognize parts of the human body. Landmarks can be assigned using the neural network to the connected parts of a user's body depicted by an image frame. Real-world coordinates of the landmarks can be tracked over time from image frames. A distance (e.g., Euclidean) between both wrists of the user's hands and an eye view point (a midpoint between eyes of a user) may be calculated as well as other identified markers, such as head tilt, a wrist, or elbow angle etc. An object (headphones) may be detected, and a pose of the user may be classified. The classified pose may be stored, and a slope of a finger or other positional information of other body parts may be stored in memory (e.g., positional information of fingers/hands relating to the user picking up and putting on headphones using hands/fingers of the user). In this way, the pose of the user may be tracked over time for tracking user interaction with the headphones.

In an example of smart watch pose detection, a neural network can be trained to recognize parts of the human body. Landmarks may be assigned using the neural network to the connected parts of a user's body depicted by an image frame. Real-world coordinates of the landmarks can be tracked over time from image frames. A distance between a wrist landmark can be calculated. An object (smart watch) is detected. A pose of the user can be classified based upon the wrist landmark (e.g., a location of the wrist with respect to a location of the smart watch). The classified pose can be stored, and a slope of a finger or other positional information of other body parts such as the wrist is stored in memory. In this way, the pose of the user may be tracked over time for tracking user interaction of the smart watch.

In some embodiments, a viewpoint is identified by the platform. A neural network can be trained to detect facial landmarks. The neural network can be used to identify landmarks of eyes, irises, and pupils. A right diameter of the right eye and a left diameter of the left eye can be determined using the landmarks of the eyes, and a midpoint will correspond to pupil coordinates. The midpoint of the pupils of both eyes can be used to obtain x, y, z coordinates. A length from the center of both pupils corresponds to a length passing from the left eye to the right eye. A midpoint of this length corresponds to a center point of the left and right pupils. This midpoint is tracked as the viewpoint of the user, which can be used to determine current positions of an eye cursor or tracking pad on a canvas or webpage.

In some embodiments, head direction can be identified by the platform. A midpoint between eyes of the user is calculated (e.g., the eyes of the user may be projected onto a virtual screen, which is used to identify the midpoint between the eyes). The viewpoint that was previously determined can be used to find an angle between the midpoint between the eyes of the user and the viewpoint of the user's gaze. This angle provides an angle or measurement used to understand a direction of a head with respect to the canvas. If the user moves the user's head diagonally up in a left or right direction, then a theta is determined with respect to the viewpoint. This change can be mapped to a quadrant that provides a direction in which the head is moving. Similarly, if the user moves the user's head diagonally down in a left or right direction, then this can be also tracked. A dtheta/time (or a change in theta with respect to time) helps determine a speed at which the user is changing a direction of head movement and if the user is moving backwards or forwards.

In some embodiments, the platform can be configured to detect an eye blink. Coordinates of the midpoint (viewpoint coordinates) can be used to obtain a z axis that is used to track changes in eye lids. When the midpoint moves towards a negative y direction downwards, then the eyes may be closing/closed. The eyes may be opening/opened when the viewpoint or midpoint moves upwards or comes to an origin point inline to a center of the pupils of both eyes. This can be used to calculate a time elapsed from the close to open and from the open to close of the eyes. A normal blink is generally less than a second long, and a time elapsed can be determined, such as where if the eyes are detected as being closed, then a timeout is set to about a 100 ms to trigger an open eye check function to see if the eyes are now open. If the eyes are now open, then a blink is detected.

In some embodiments, the platform can be configured to detect head movement. The viewpoint (midpoint) is detected and measured along a z axis. A change in the z axis (dz) while moving the viewpoint gives a change in head position. The angle or direction previously calculated can be used to determine an angle with respect to a screen midpoint, which is used to project the viewpoint to a quadrant. If the dz is positive, and dz and the angle is greater than about 250 degrees, then the head position is typically towards the left. If dz and the angle are less than about 250 degrees, then the head position is typically to the right. If dz is negative or less than about 10 degrees, then the head position is typically facing downwards. If dz is greater than about 10 degrees, then the head position is typically facing upwards.

In some embodiments, the platform can be configured to detect a distance from the user to the screen. A midpoint may be calculate using the formula: Midpoint=(Math.sqrt(Math.pow(righteyemidpoint[0]−lefteyemidpoint[0],2)+Math.pow(righteyemidpoint[1]−lefteyemidpoint[1],2))). If the angle of the head is constant and a floor value of the midpoint is increasing, then a distance of the viewpoint from the user is moving closer to the camera or to the canvas. Similarly, if the angle of the head position is constant and the floor value of the midpoint is decreasing or tending to zero, then a distance of the viewpoint from the user is moving further away from the camera or the canvas.

According to some embodiments, a method may be provided. The method may include generating a canvas populated with a first grid corresponding to a location of a first object. The canvas is populated with an eye cursor positioned to track a gaze of a user. A plurality of image frames captured by a camera is evaluated to track interaction of the user with the first object. For an image frame, a model is utilized to process the image frame to identify facial landmarks of the user. A midpoint between a left eye and a right eye of the user is determined based upon the facial landmarks. A current position of the eye cursor is set within the canvas based upon the midpoint. In response to detecting that the eye cursor intersects the first grid in the canvas, the user may be determined to be interacting with the first object. In response to the user interacting with the first object, an action may be performed based upon the interaction.

According to some embodiments, the object comprises an electronic device. The method comprises transmitting a command to the electronic device to modify operation of the electronic device.

According to some embodiments, the method includes tracking a timespan of the interaction. A level of interest of the user with the first object is outputted based upon the timespan.

According to some embodiments, the method includes populating the canvas with a plurality of grids corresponding to locations of objects. The canvas is populated with a second grid corresponding to a location of a second object. A determination is made as to whether the user interacts with the second object based upon whether the eye cursor intersects with the second grid.

According to some embodiments, the method includes populating the canvas with a plurality of grids corresponding to locations of objects. An authentication code corresponding to an order of interactions with one or more of the objects is generated. The user is authenticated based upon the user interacting with the one or more of the objects according to the order of interactions.

According to some embodiments, the method includes modifying a display property of the first grid in response to the eye cursor intersecting the first grid.

According to some embodiments, the method includes populating the canvas with a plurality of grids corresponding to locations of objects. Interactions of users with the objects is tracked within a data structure. Analytical processing is executed upon the data structure to determine users interests with the objects based upon the interactions, types of interactions, and lengths of the interactions.

According to some embodiments, the method includes receiving first user input defining the first grid, wherein the user input specifies a location, orientation, and size of the first grid; and populating the canvas with the first grid based upon the first user input.

According to some embodiments, the method includes in response to receiving second user input modifying the first grid, modifying at least one of the location, the orientation, or the size of the first grid based upon the second user input.

According to some embodiments, a device is provided. The device comprises a processor. The processor is configured to generate a canvas populated with grids corresponding to locations of objects. The canvas is populated with an eye cursor positioned to track a gaze of a user. A plurality of image frames captured by a camera are evaluated to track interaction of users with the objects. For an image frame, a model is utilized to process the image frame to identify facial landmarks of the user. A midpoint between a left eye and a right eye of the user is determined based upon the facial landmarks. A current position of the eye cursor is set within the canvas based upon the midpoint. In response to detecting that the eye cursor intersects a first grid in the canvas, the user may be determined as interacting with a first object represented by the first grid. In response to the user interacting with the first object, an action is performed based upon the interaction.

According to some embodiments, the processor is configured to generate a heat map for the first grid based upon user interaction with the first object. In response to the heat map indicating a level of interest in the first object that exceeds an interest threshold, an alert about an interest in the first object is generated and transmitted.

According to some embodiments, the processor is configured to determine an idle time of the user interacting with the first object based upon a timespan of the eye cursor intersecting the first grid. In response to the idle time exceeding a threshold timespan, an alert about an interest of the user with the first object is generated and transmitted.

According to some embodiments, in response to detecting a threshold amount of changes in movement of the eye cursor over a time frame, generating and transmitting an alert about an interest of the user with the first object.

According to some embodiments, the processor is configured to track a pattern of user interaction changes amongst the objects based upon an order and duration of the eye cursor intersecting the grids. An interest of the user is determined based upon the pattern.

According to some embodiments, a non-transitory computer-readable medium storing instructions that when executed facilitate performance of operations, is provided. The operations include generating a canvas populated with grids corresponding to locations of objects. The canvas is populated with an eye cursor positioned to track a gaze of a user. A plurality of image frames captured by a camera is evaluated to track interaction of users with the objects. For an image frame, a midpoint between a left eye and a right eye of the user is determined based upon facial landmarks of the user identified from the image frame. A current position of the eye cursor is set within the canvas based upon the midpoint. In response to detecting that the eye cursor intersects a first grid in the canvas, the user is determined to be interacting with a first object represented by the first grid. In response to the user interacting with the first object, an action may be performed based upon the interaction.

According to some embodiments, the operations include evaluating the plurality of image frames to track real-time3D object orientation of the objects.

According to some embodiments, the operations include extracting features from the plurality of image frames corresponding to iris features, body features, voice features, and facial features.

According to some embodiments, the operations include evaluating the iris features to track iris and head movements.

According to some embodiments, the operations include evaluating the body features to track body poses; and evaluating the voice features and the facial features to for tone sentiment, text sentiment, and facial emotional expression.

According to some embodiments, the operations include populating the canvas with a plurality of eye cursors to track gazes of a plurality of users.

FIG.6is an illustration of a scenario600involving an example non-transitory machine readable medium602. The non-transitory machine readable medium602may comprise processor-executable instructions612that when executed by a processor616cause performance (e.g., by the processor616) of at least some of the provisions herein. The non-transitory machine readable medium602may comprise a memory semiconductor (e.g., a semiconductor utilizing static random access memory (SRAM), dynamic random access memory (DRAM), and/or synchronous dynamic random access memory (SDRAM) technologies), a platter of a hard disk drive, a flash memory device, or a magnetic or optical disc (such as a compact disk (CD), a digital versatile disk (DVD), or floppy disk). The example non-transitory machine readable medium602stores computer-readable data604that, when subjected to reading606by a reader610of a device608(e.g., a read head of a hard disk drive, or a read operation invoked on a solid-state storage device), express the processor-executable instructions612. In some embodiments, the processor-executable instructions612, when executed cause performance of operations, such as at least some of the example method200ofFIG.2, for example. In some embodiments, the processor-executable instructions612are configured to cause implementation of a system, such as at least some of the example system100ofFIG.1, at least some of the example system300ofFIGS.3A and3B, at least some of the example system400ofFIG.4, and/or at least some of the example system500ofFIGS.5A-5F.

FIG.7is an interaction diagram of a scenario700illustrating a service702provided by a set of computers704to a set of client devices710via various types of transmission mediums. The computers704and/or client devices710may be capable of transmitting, receiving, processing, and/or storing many types of signals, such as in memory as physical memory states.

The computers704of the service702may be communicatively coupled together, such as for exchange of communications using a transmission medium706. The transmission medium706may be organized according to one or more network architectures, such as computer/client, peer-to-peer, and/or mesh architectures, and/or a variety of roles, such as administrative computers, authentication computers, security monitor computers, data stores for objects such as files and databases, business logic computers, time synchronization computers, and/or front-end computers providing a user-facing interface for the service702.

Likewise, the transmission medium706may comprise one or more sub-networks, such as may employ different architectures, may be compliant or compatible with differing protocols and/or may interoperate within the transmission medium706. Additionally, various types of transmission medium706may be interconnected (e.g., a router may provide a link between otherwise separate and independent transmission medium706).

In scenario700ofFIG.7, the transmission medium706of the service702is connected to a transmission medium708that allows the service702to exchange data with other services702and/or client devices710. The transmission medium708may encompass various combinations of devices with varying levels of distribution and exposure, such as a public wide-area network and/or a private network (e.g., a virtual private network (VPN) of a distributed enterprise).

In the scenario700ofFIG.7, the service702may be accessed via the transmission medium708by a user712of one or more client devices710, such as a portable media player (e.g., an electronic text reader, an audio device, or a portable gaming, exercise, or navigation device); a portable communication device (e.g., a camera, a phone, a wearable or a text chatting device); a workstation; and/or a laptop form factor computer. The respective client devices710may communicate with the service702via various communicative couplings to the transmission medium708. As a first such example, one or more client devices710may comprise a cellular communicator and may communicate with the service702by connecting to the transmission medium708via a transmission medium707provided by a cellular provider. As a second such example, one or more client devices710may communicate with the service702by connecting to the transmission medium708via a transmission medium709provided by a location such as the user's home or workplace (e.g., a WiFi (Institute of Electrical and Electronics Engineers (IEEE) Standard 702.11) network or a Bluetooth (IEEE Standard 702.15.1) personal area network). In this manner, the computers704and the client devices710may communicate over various types of transmission mediums.

FIG.8presents a schematic architecture diagram800of a computer704that may utilize at least a portion of the techniques provided herein. Such a computer704may vary widely in configuration or capabilities, alone or in conjunction with other computers, in order to provide a service such as the service702.

The computer704may comprise one or more processors810that process instructions. The one or more processors810may optionally include a plurality of cores; one or more coprocessors, such as a mathematics coprocessor or an integrated graphical processing unit (GPU); and/or one or more layers of local cache memory. The computer704may comprise memory802storing various forms of applications, such as an operating system804; one or more computer applications806; and/or various forms of data, such as a database808or a file system. The computer704may comprise a variety of peripheral components, such as a wired and/or wireless network adapter814connectible to a local area network and/or wide area network; one or more storage components816, such as a hard disk drive, a solid-state storage device (SSD), a flash memory device, and/or a magnetic and/or optical disk reader.

The computer704may comprise a mainboard featuring one or more communication buses812that interconnect the processor810, the memory802, and various peripherals, using a variety of bus technologies, such as a variant of a serial or parallel AT Attachment (ATA) bus protocol; a Uniform Serial Bus (USB) protocol; and/or Small Computer System Interface (SCI) bus protocol. In a multibus scenario, a communication bus812may interconnect the computer704with at least one other computer. Other components that may optionally be included with the computer704(though not shown in the schematic architecture diagram800ofFIG.8) include a display; a display adapter, such as a graphical processing unit (GPU); input peripherals, such as a keyboard and/or mouse; and a flash memory device that may store a basic input/output system (BIOS) routine that facilitates booting the computer704to a state of readiness.

The computer704may operate in various physical enclosures, such as a desktop or tower, and/or may be integrated with a display as an “all-in-one” device. The computer704may be mounted horizontally and/or in a cabinet or rack, and/or may simply comprise an interconnected set of components. The computer704may comprise a dedicated and/or shared power supply818that supplies and/or regulates power for the other components. The computer704may provide power to and/or receive power from another computer and/or other devices. The computer704may comprise a shared and/or dedicated climate control unit820that regulates climate properties, such as temperature, humidity, and/or airflow. Many such computers704may be configured and/or adapted to utilize at least a portion of the techniques presented herein.

FIG.9presents a schematic architecture diagram900of a client device710whereupon at least a portion of the techniques presented herein may be implemented. Such a client device710may vary widely in configuration or capabilities, in order to provide a variety of functionality to a user such as the user712. The client device710may be provided in a variety of form factors, such as a desktop or tower workstation; an “all-in-one” device integrated with a display908; a laptop, tablet, convertible tablet, or palmtop device; a wearable device mountable in a headset, eyeglass, earpiece, and/or wristwatch, and/or integrated with an article of clothing; and/or a component of a piece of furniture, such as a tabletop, and/or of another device, such as a vehicle or residence. The client device710may serve the user in a variety of roles, such as a workstation, kiosk, media player, gaming device, and/or appliance.

The client device710may comprise one or more processors910that process instructions. The one or more processors910may optionally include a plurality of cores; one or more coprocessors, such as a mathematics coprocessor or an integrated graphical processing unit (GPU); and/or one or more layers of local cache memory. The client device710may comprise memory901storing various forms of applications, such as an operating system903; one or more user applications902, such as document applications, media applications, file and/or data access applications, communication applications such as web browsers and/or email clients, utilities, and/or games; and/or drivers for various peripherals. The client device710may comprise a variety of peripheral components, such as a wired and/or wireless network adapter906connectible to a local area network and/or wide area network; one or more output components, such as a display908coupled with a display adapter (optionally including a graphical processing unit (GPU)), a sound adapter coupled with a speaker, and/or a printer; input devices for receiving input from the user, such as a keyboard911, a mouse, a microphone, a camera, and/or a touch-sensitive component of the display908; and/or environmental sensors, such as a global positioning system (GPS) receiver919that detects the location, velocity, and/or acceleration of the client device710, a compass, accelerometer, and/or gyroscope that detects a physical orientation of the client device710. Other components that may optionally be included with the client device710(though not shown in the schematic architecture diagram900ofFIG.9) include one or more storage components, such as a hard disk drive, a solid-state storage device (SSD), a flash memory device, and/or a magnetic and/or optical disk reader; and/or a flash memory device that may store a basic input/output system (BIOS) routine that facilitates booting the client device710to a state of readiness; and a climate control unit that regulates climate properties, such as temperature, humidity, and airflow.

The client device710may comprise a mainboard featuring one or more communication buses912that interconnect the processor910, the memory901, and various peripherals, using a variety of bus technologies, such as a variant of a serial or parallel AT Attachment (ATA) bus protocol; the Uniform Serial Bus (USB) protocol; and/or the Small Computer System Interface (SCI) bus protocol. The client device710may comprise a dedicated and/or shared power supply918that supplies and/or regulates power for other components, and/or a battery904that stores power for use while the client device710is not connected to a power source via the power supply918. The client device710may provide power to and/or receive power from other client devices.

Various operations of embodiments are provided herein. In an embodiment, one or more of the operations described may constitute computer readable instructions stored on one or more computer readable media, which if executed by a computing device, will cause the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering may be implemented without departing from the scope of the disclosure. Further, it will be understood that not all operations are necessarily present in each embodiment provided herein. Also, it will be understood that not all operations are necessary in some embodiments.

Also, although the disclosure has been shown and described with respect to one or more implementations, alterations and modifications may be made thereto and additional embodiments may be implemented based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications, alterations and additional embodiments and is limited only by the scope of the following claims. The specification and drawings are accordingly to be regarded in an illustrative rather than restrictive sense. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.