Gestural object selection

A posture for a user is derived and used in conjunction with a known physical location of a tangible object or a presented virtual object to determine when the user selects at least a portion of the object. The derived posture comprises a model having a user origin and a gestural terminus, from which a designation vector intersecting with the selected object is determined. The designation vector may extend from a user origin of the dominant eye through a gestural terminus comprising a tip of an outstretched finger. For example, the user may point to a chair in the room to select the physical chair, or point to a picture of a chair presented via a display device to select the picture.

BACKGROUND

Augmented reality allows interaction among users and real-world tangible objects and virtual or computer-generated objects and information within an augmented reality environment. Within the augmented reality environment, interactions may include electronic input, verbal input, physical input relating to the manipulation of tangible and virtual objects, and so forth.

DETAILED DESCRIPTION

An augmented reality system may be configured to provide for interactions between objects such as tangible real-world objects and presented virtual objects. The augmented reality system may be configured to accept gestural input. Gestural input is a natural communication mode for humans. For example, one person referring another person to a restaurant across the street is likely to point in the direction of the restaurant. Gestures such as pointing, grasping, sweeping an arm, and so forth may be used as inputs within the augmented reality environment.

Disclosed herein are techniques and devices suitable for gestural object selection whereby a particular object or portion thereof is indicated by a gesture. For example, a user may point to a virtual object such as a photograph presented via a display device to select that particular photograph. As described herein, the particular object or portion thereof is selected by determining a designation vector based at least in part upon a derived posture of the user. The derived posture may include a user origin, a gestural terminus, location of the object, and in some implementations the orientation of the gestural terminus.

The user origin is a reference point or volumetric location associated with the physicality of a user. For example, in some implementations the user origin may be the spatial location (e.g., in X-Y-Z space) of the dominant eyeball. The user origin and its designation are discussed below in more detail.

The gestural terminus is one or more spatial points or volumetric locations associated with where a gesturing appendage of the user or tool ends. For example, the gestural terminus of a user pointing with an index finger is the tip of their extended finger. While a single gestural terminus is described herein, it is understood that a gestural input may include a plurality of gestural termini, such as when two hands are involved, multiple fingers, head and hand, and so forth. The gestural terminus is also discussed below in more detail.

The designation vector exists relative to the user origin and the gestural terminus. This vector includes a direction as well as a magnitude. The magnitude may be proportionate to a distance between the user origin and the gestural terminus. Orientation of the gestural terminus or other factors may also affect the magnitude of the designation vector.

By determining the designation vector and determining where that vector corresponds with an object or a portion thereof, the user may select or otherwise interact with that object in the augmented reality environment. For example, the user may point with palm-up to a displayed picture of a chair presented by a display device. The designation vector intersects the image of the chair, while the magnitude of the vector in response to the gestural orientation may be such that the selection is of the opposite and unseen side of the chair. Upon selection, the displayed image of the chair may be changed to show the previously hidden back of the chair.

ILLUSTRATIVE ENVIRONMENT

FIG. 1shows an illustrative augmented reality environment100that includes an augmented reality functional node (ARFN)102with associated computing device. One or more ARFNs102may be positioned on the ceiling of the room. In other implementations, the ARFN102(s) may be positioned in other locations within the scene, such as on walls, in lamp fixtures, on tables, and so forth. When active, the ARFN102may generate an augmented reality environment incorporating the scene. In some implementations, more or fewer ARFNs102may be used.

The ARFN102is coupled to a computing device104. This computing device104may be within the ARFN102, or disposed at another location and connected to the ARFN102. The computing device104comprises a processor106, an input/output interface108, and a memory110. The processor106may comprise one or more processors configured to execute instructions. The instructions may be stored in memory110, or in other memory accessible to the processor106.

The input/output interface108may be configured to couple the computing device104to other components such as projector, cameras, microphones, other ARFNs102, other computing devices, and so forth. The coupling between the computing device104and the devices may be via wire, fiber optic cable, or wireless connection.

The memory110may include computer-readable storage media (“CRSM”). The CRSM may be any available physical media accessible by a computing device to implement the instructions stored thereon. CRSM may include, but is not limited to, random access memory (“RAM”), read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technology, compact disk read-only memory (“CD-ROM”), digital versatile disks (“DVD”) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.

Several modules such as instructions, datastores, and so forth may be stored within the memory110and configured to execute on a processor, such as the processor106. An operating system module112is configured to manage hardware and services within and coupled to the computing device104for the benefit of other modules. Modules may be stored in the memory of the ARFN102, storage devices accessible on the local network, or cloud storage accessible via a wide area network.

A spatial analysis module114is configured to perform several functions which may include analyzing a scene to generate a topology, recognizing objects in the scene, dimensioning the objects, modeling the scene, and so forth. Characterization may be facilitated using several technologies including structured light, light detection and ranging (LIDAR), optical time-of-flight, ultrasonic ranging, stereoscopic imaging, radar, and so forth either alone or in combination with one another. For convenience, and not by way of limitation, the examples in this disclosure refer to structured light. The spatial analysis module114may receive data from input and output components, such as described below with regards toFIG. 2.

An object datastore116in the memory110is configured to maintain information about objects within the scene. This information may be acquired from the ARFN102, other input devices, or via manual input. For example, the object datastore may include surface contours of the object. The object datastore118may include a library of pre-loaded reference objects or physiological models of users. These models are discussed in more detail below with regards to the posture module122. The object datastore118may be stored on one or more of the memory of the ARFN102, storage devices accessible on the local network, or cloud storage accessible via a wide area network. As used in this disclosure, datastore includes lists, arrays, databases, and other data structures used to provide storage and retrieval of data.

An augmented reality module118is configured to generate augmented reality output in concert with the physical environment. A tracking and control module120may be configured to identify objects within the scene and track their position and, in some implementations, orientation. The augmented reality module118may access input/output devices within one or more ARFN nodes102to generate output, receive input, or both.

The tracking and control module122is configured to identify objects within the scene, track their position, and accept gestures as inputs. The tracking and control module122may utilize data obtained by the spatial analysis module114. The tracking and control module122may include a posture module122and a selection module124.

The posture module122is configured to derive a posture of a user in the augmented reality environment. As used herein, posture refers to the physical arrangement of a user's body, such as arms, hands, legs, torso, head, and so forth. For example a user with right arm extended and right index finger extended is in one posture, while the user holding both hands over his head is in another posture. The posture module122is configured to accept data from the augmented reality environment and derive the posture of one or more users. In some implementations, the derived posture may comprise a simplified model of the human body's gross orientation and operational limits. The posture module122may be used to determine the position and placement of portions of the user's anatomy in space. For example, few users can touch a sole of their foot to the back of their head while standing. As a result, with the derived posture model, it is possible to assume that the user's appendage which is touching the back of that same user's head is likely a hand and remove otherwise incongruous results. The derived posture and its determination are discussed below in more detail with regards toFIG. 5.

A selection module124generates a designation vector and determines what object that vector corresponds to. The selection module124may use the derived posture from the posture module122and data from the spatial analysis module114to determine that a user is gesturing to select a particular object or portion of that object. The designation vector and the selection are discussed below in additional detail.

FIG. 2shows an illustrative schematic200of the augmented reality functional node102and selected components. The ARFN node102is configured to scan at least a portion of a scene202and the objects therein. The ARFN102may also be configured to provide output.

A chassis204is configured to hold the components of the ARFN102. Within the chassis may be disposed a projector206. The projector206is configured to generate images. These images may be visible light images perceptible to the user, visible light images imperceptible to the user, images with non-visible light, or a combination thereof. This projector206may comprise a digital micromirror device (DMD), liquid crystal on silicon display (LCOS), liquid crystal display, 3LCD, and so forth configured to generate an image and project it onto a surface within the environment. In some implementations other display devices may be used in place of or in addition to the projector206. These display devices may include emissive, reflective, or other display technologies suitable for presenting electronically generated visual information such as bitmaps, raster graphics, and so forth to a user.

The projector206has a projector field of view208which describes a particular solid angle. The projector field of view208may vary according to changes in the configuration of the projector. For example, the projector field of view208may narrow upon application of an optical zoom to the projector. In some implementations a plurality of projectors206may be used.

A camera210or other imager may also be disposed within the chassis204. The camera210is configured to image the scene in visible light wavelengths, non-visible light wavelengths, or both. The camera210has a camera field of view212which describes a particular solid angle. The camera field of view212may vary according to changes in the configuration of the camera210. For example, an optical zoom of the camera may narrow the camera field of view212. In some implementations a plurality of cameras210may be used.

The chassis204may be mounted with a fixed orientation, or be coupled via an actuator to a fixture such that the chassis204may move. Actuators may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis204or components therein such as the projector206and/or the camera210. The actuator may comprise a pan motor214, tilt motor216, and so forth. The pan motor214is configured to rotate the chassis204in a yawing motion. The tilt motor216is configured to change the pitch of the chassis204. By panning and/or tilting the chassis204, different views of the scene may be acquired. The spatial analysis module114may use the different views to monitor objects within the environment.

One or more microphones218may be disposed within the chassis204, or elsewhere within the scene. These microphones218may be used to acquire input from the user, for echolocation, location determination of a sound, or to otherwise aid in the characterization of and receipt of input from the scene. For example, the user may make a particular noise such as a tap on a wall or snap of the fingers which are pre-designated as inputs within the augmented reality environment. The user may alternatively use voice commands.

One or more speakers220may also be present to provide for audible output. For example, the speakers220may be used to provide output from a text-to-speech module or to playback pre-recorded audio.

A transducer222may be present within the ARFN102, or elsewhere within the environment, and configured to detect and/or generate inaudible signals, such as infrasound or ultrasound. These inaudible signals may be used to provide for signaling between accessory devices and the ARFN102.

A ranging system224may also be provided in the ARFN102. The ranging system224is configured to provide distance information from the ARFN102to a scanned object or set of objects. The ranging system224may comprise radar, light detection and ranging (LIDAR), ultrasonic ranging, stereoscopic ranging, and so forth. In some implementations the transducer222, the microphones218, the speaker220, or a combination thereof may be configured to use echolocation or echo-ranging to determine distance and spatial characteristics.

In this illustration, the computing device104is shown within the chassis204. However, in other implementations all or a portion of the computing device104may be disposed in another location and coupled to the ARFN102. This coupling may occur via wire, fiber optic cable, wirelessly, or a combination thereof. Furthermore, additional resources external to the ARFN102may be accessed, such as resources in another ARFN102accessible via a local area network, cloud resources accessible via a wide area network connection, or a combination thereof.

Also shown in this illustration is a projector/camera linear offset designated “O”. This is a linear distance between the projector206and the camera210. Placement of the projector206and the camera210at distance “O” from one another aids in the recovery of structured light data from the scene. The known projector/camera linear offset “O” may also be used to calculate distances, dimensioning, and otherwise aid in the characterization of objects within the scene202. In other implementations the relative angle and size of the projector field of view208and camera field of view212may vary. Also, the angle of the projector206and the camera210relative to the chassis204may vary.

In other implementations the components of the ARFN102may be distributed in one or more locations within the environment100. As mentioned above, microphones218and speakers220may be distributed throughout the scene. The projector206and the camera210may also be located in separate chassis204. The ARFN102may also include discrete portable signaling devices used by users to issue inputs. For example, these may be acoustic clickers (audible or ultrasonic), electronic signaling devices such as infrared emitters, radio transmitters, and so forth.

FIG. 3illustrates a structured light pattern300impinging on the scene. In this illustration, the projector206within the ARFN102projects a structured light pattern302onto the scene202. In some implementations a sequence of different structured light patterns302may be used. This structured light pattern302may be in wavelengths which are visible to the user, non-visible to the user, or a combination thereof. The structured light pattern304is shown in this example, but not by way of limitation, as a grid. In other implementations other patterns, such as bars, dots, pseudorandom noise, and so forth may be used. Pseudorandom noise (PN) patterns are particularly useful because a particular point within the PN pattern may be specifically identified. A PN function is deterministic in that given a specific set of variables, a particular output is defined. This deterministic behavior allows the specific identification and placement of a point or block of pixels within the PN pattern.

A sphere304is shown positioned between the projector206and a wall in the scene202. A shadow306from the sphere304appears on the wall. Inspection of the sphere304shows a deformation effect308of the structured light pattern302as it interacts with the curved surface of the sphere304.

In some implementations other effects, such as dispersion of the structured light pattern302may be used to provide information on the topography of the scene. Where the projector206and camera210have differing fields of view, such as shown inFIG. 2, the dispersion or change in the “density” of the structured light pattern302may be used to determine depth of field.

The camera210detects the interaction of the structured light pattern302with objects within the scene. For example, the deformation effect308on the sphere304is detected by the camera210. The camera210is configured to sense or detect the structured light. In some implementations the camera210may also sense or detect wavelengths other than those used for structured light pattern302. The image captured by the camera210is processed by the spatial analysis module114to characterize the scene202. In some implementations, multiple cameras may be used to acquire the image.

Illustrative Scene

FIG. 4illustrates a scene400with two users in an augmented reality environment, such as maintained at least in part by one or more of the ARFNs102. A first user402(1) is shown in a discussion with a second user402(2). A display404is shown. This display404may be provided by a display device such as a projector206, emissive display, reflective display, and so forth. The display404may be configured to present a two- or three-dimensional image. For example, the display404may comprise a liquid crystal display, light emitting diode display, or image projected by one or more of the ARFNs102. This image may be two-dimensional, or provide for stereoscopic three-dimensional viewing. In other implementations the display404may comprise a space filling three-dimensional display system.

Presented by the display404are several objects406. In this and the following examples, the objects are virtual objects. However, the same techniques and devices described herein may be used to select tangible objects. The first user402(1) is looking along a gaze direction408(1) to a particular user-perceived designated point410(1). The gaze direction408is the overall average direction of the user's central or foveal vision. Meanwhile the second user402(2) is also looking along her gaze direction408(2) to her user-perceived designated point410(2). The user-perceived designated point410is the point the user associates with the particular gesture. The first user402(1) is thus pointing to the rightmost object406presented on the display404, while the second user402(2) is pointing to the leftmost object406presented on the display404.

FIG. 5illustrates derived postures500for the two users ofFIG. 4. The ARFN102may be used to determine the spatial position and orientation of the users402, as well as other physical objects in the scene. As described above, the spatial analysis module114may provide data to the posture module122which in turn generates derived postures for one or more users in the scene. In this illustration a derived posture502(1) for the first user402(1) is shown as well as a derived posture502(2) for the second user402(2).

The derived posture may comprise a simplified biomechanical model of the user's gross physiological makeup. For example, as shown here, the derived posture502reflects length, position, and orientation of the user's joints and extremities. In some implementations this may be visualized as a wireframe model of the users as shown here.

The derived posture502provides a simplified model which may be used in the augmented reality environment. For example, as shown here and discussed below in more detail, the position of a gestural terminus504(1) (such as the fingertips of the left hand) of the first user402(1) and a user origin such as the user's dominant eye may be used to determine the designation vector and determine the user-perceived designation point410(1). The derived posture502also includes physiological constraints. For example, the user cannot detach a limb, and the user's arm reach extends to a definable maximum in relation to their shoulder. In another example, a user's wrist cannot rotate a complete 360 degrees. Such physiological constraints allow the environment to disregard contradictory or spurious input data, resolve ambiguities which may result from the scanning and user observation techniques in use, and improve the overall quality of the user interaction.

The derived posture502is a simplified model in that it is a less than comprehensive of the totality of the user and may omit details not used by the environment. For example, in one implementation as shown here, position of the user's jaw may not be a factor in the designation vector, and thus it is not a part of the derived posture502. Generation of the derived posture is described in more detail below with regards toFIGS. 9-11.

One or more gestural termini504may be associated with the derived posture502. For example, the gestural terminus504(1) is volume where the user's thumb and index finger are proximate to one another, such as with the gesture of the first user402(1) picking up the object406. The gestural terminus504(2) may also comprise a volume proximate to or encompassing the user's fingertip, such as the end of the second user's402(2) right index finger which is extended. The gestural terminus504may in some implementations comprise a volume associated with a hand, arm, shoulder, head, foot, and so forth.

FIG. 6illustrates a derived posture and calibration600configured to determine a dominant eye of the user. Humans have binocular vision resulting from two eyes which are placed a fixed distance apart. For relatively close objects, such as within about 10 meters, this binocular vision provides for depth perception. Because of the distance between the eyes, parallax effects occur, particularly for objects which are relatively close. Stated another way, each eye sees a different version of the same scene, shifted due to the linear distance between the eyes. This may be readily visualized by reaching out your hand to cover a distant object. By closing one eye and leaving the other open, then alternating, your hand appears to “jump”.

In most of the human population, the human brain generally selects and uses a dominant eye. The dominant eye, or ocular dominance, is the tendency for the user's brain to prefer visual input from one eye over visual input from the other eye. Which eye is dominant may remain constant over the user's lifetime, or may vary depending upon the direction of the gaze, individual physiological variations, and so forth. As a result, eye dominance may be periodically re-assessed.

As shown inFIG. 6, the user is presented with a calibration object602. The calibration object602has a known position relative to the derived posture502. The user may be prompted to select this calibration object602, or the calibration object602may be a commonly used control placed such that its selection is unambiguous regardless of eye dominance. As depicted here, the user is left eye dominant and has a dominant eye gaze604emanating from the left eye towards the calibration object602. A non-dominant right eye provides a non-dominant eye gaze606.

From the user's402(2) point of view, she is reaching out the index finger of her right hand to select the calibration object602. In fact, the gestural terminus504associated with that index finger is in the path of the dominant eye gaze604. This presence in the gaze path may thus be used at least in part to determine eye dominance. Meanwhile, the non-dominant eye gaze606is unobscured. But, due to the user's eye dominance and perception, the index finger appears to be placed over the calibration object602.

By determining the derived posture502including the location of the user's eyes, knowing the position of the gestural terminus504, and the position of the calibration object602, the environment may thus determine which eye is dominant. Such a determination may be used by the environment to generate the designation vector and determine what object or portion thereof has been selected.

FIG. 7is a plan-view illustration700of the derived posture of a user and selection of a particular object. This view shows the eyes of the derived posture502. For clarity of illustration and not by way of limitation, the remainder of the derived posture502has been omitted.

A left eye702is designated here as a user origin704, which is centered on that eye. As mentioned above, the user origin is a reference point or volumetric location associated with the physicality of a user. The user origin may be part of the derived posture502. For example, as shown here, the user origin704is the spatial location of the dominant (left) eyeball. Also shown is the right eye,706which is non-dominant in this illustration.

Shown here is a gestural terminus504. The gestural terminus504in conjunction with the user origin704may be used to determine a designation vector708. The designation vector708provides a direction and a magnitude and may be defined relative to the user origin704and the gestural terminus504. The magnitude may be proportionate to a distance between the user origin704and the gestural terminus504. For example, the user402may point to a stack of objects406starting with arm midway extended, and extend the arm out to maximum reach to indicate selection of the object406deepest in the stack.

The gestural terminus504may also include an orientation710. In some implementations, the orientation of the user's hand in space may alter the orientation of the gestural terminus710. For example, the user pointing with their index finger and palm down may be one orientation while pointing with the index finger and palm up may be another. In some implementations, this orientation may alter the magnitude of the designation vector. For example, the orientation710with the index finger extended and palm down may select the closest side of object, or closest object such as a selected object712. As described above, particular objects or portions of objects may be selected or indicated by gestures. These objects may be virtual or tangible. Once selected, an object or portion thereof may initiate a function, be designated for further processing, and so forth.

The selected object712or portion thereof is that which corresponds to the designation vector708. For example, as shown here, the correspondence is indicated by the intersection of the designation vector708and the object. Unselected objects such as one in the foreground714(1) do not intersect the designation vector708. Unselected objects in the background714(2) are also not intersected by the designation vector708. As described above, the user may “push” through by moving the gestural terminus504relative to the user origin704, altering orientation710of the gestural terminus, and so forth to modify the magnitude of the designation vector708. For example, a two-dimensional display404may present a representation of three-dimensional objects. By the user rotating a hand to place the palm up, the magnitude of the designation vector708may be increased to select the previously unselected background object714(2) while leaving the foreground object712unselected. Similarly, in a three-dimensional display such as an auto stereoscopic projection or a volumetric display, the user may select objects at varying depth based at least in part on changing the magnitude of the designation vector708.

In some implementations, the augmented reality environment may be configured to visually or audibly indicate the object or portion thereof which has been selected. For example, where the object is a physical object, the projector206may be configured to present a visual indicator such as a glowing ring that is projected around the object indicating selection. Where the object is a virtual object such as presented on screen or projected on a wall, the object or portion thereof selected may be indicated by a cursor or other visual indicator. In other instances, an ARFN may audibly state the name of the object that the system believes the user has selected. For instance, in response to a user selecting a chair with a particular gesture, the ARFN may state the term “chair” over the speakers, allowing the user to confirm or deny the system's interpretation of the gesture.

The designation vector708may thus extend through at least a portion of the object to designate a portion of the object non-visible to the user or to other objects which are non-visible to the user. For example, the user may thus select the opposite side of a three-dimensional object or select a user interface screen which is present “under” several other overlapping screens.

In some, but not all implementations, the designation vector708may be determined at least in part by the gaze direction408as well as the position of the gestural terminus. For example, where the gaze direction408is coincident with the line extending from the user origin704through the gestural terminus504, the designation vector708may be considered to have a high degree of accuracy as to what the user intends to select.

FIG. 8is a plan-view illustration800of the derived posture of a user and selection of a particular object which is outside of the user's field of view. Each eye has a particular field of view, which describes a solid angle within which an image of the scene is formed. This field of view is limited by the shape of the user's lenses in their eyes, presence or absence of corrective lenses, facial structure, retinal health, and so forth. The fields of view of the left and right eyes generally overlap, but do not do so completely. A healthy user typically has a field of central or foveal vision as well as peripheral vision which exists around the outer edges of each eye's field of view. This illustration depicts a left eye field of view802and a right eye field of view804.

Users may make gestures which are outside of their field of vision. Consider a virtual object which provides a user interface for playing back music. A user may find the music distracting, select the virtual object with a gesture, and with a gesture drag it around and behind them and thus outside of their field of vision. The environment recognizes this gesture and correspondingly moves the presentation of that virtual object to that extreme position and as previously instructed, stops the playback of the audio.

When the gestural terminus504is outside of the field of view of the dominant eye or both eyes as shown here, the user origin704may be placed at a pre-determined point. This point may remain with the dominant eye, or as shown here, move. For example, as shown here, the user origin704has been placed at a midpoint between the user's eyes. The designation vector708extends from the user origin704towards the gestural terminus504.

As described above, the orientation of the gestural terminus710may be used to modify the magnitude of the designation vector708. As shown here, the orientation710is such that the far side of the object712has been selected.

FIG. 9illustrates object selection factors902. As described above, the selection of an object or portion thereof is determined by the intersection of the object with the designation vector708.

To determine whether the designation vector708intersects an object, the environment determines a spatial location of the object904(e.g., in X-Y-Z space). As described above, this spatial location may be determined using structured light, time delay of arrival, machine vision systems, LIDAR, and so forth. Where the object is virtual and presented via a display device, the spatial location may be determined by calculation using factors such as known position of the projection or display, address within the image, and so forth. In some implementations the image may include a watermark or other encoded signal allowing for direct determination of the physical location of the presented image.

As described above, the derived posture502comprises a model of the user's physical arrangement. From the derived posture502, the gestural terminus504, user origin704, and forth may be determined.

The derived posture502may include information about head position/motion906. For example, is the user facing forwards, or turning his head to one side. In some implementations the field of view may be determined based upon the head position/motion906. The derived posture502may designate the position of a portion of the user's body at a given instant of time, motion over a pre-determined interval, or a combination thereof. For example, the motion of a hand moving or a shoulder shrug may comprise a gesture.

Shoulder position/motion908may be part of the derived posture502. Hand position/motion910may be included in the derived posture502. For example, as discussed above the orientation of the user's hand may change the gestural orientation710.

Aspects of the user's visual system may be included in the derived posture502such as a gaze direction912indicating direction of the user's central vision. This gaze may comprise orientation or directionality of one or both eyes. As discussed above, a dominant eye914may be considered, as well as field of view916. A focus point918may be determined which indicates a distance at which the user's eye or eyes are focused. For example, the user may be focused on an object that is five meters from the user. The derived posture502may take into account other factors920such as hand dominance, physical disability, foot position/motion, torso position/motion, and so forth.

Illustrative Processes

FIG. 10is an illustrative process1000of determining a selected object based at least in part upon a gesture. This gesture may be free from contact with another object, such as made in space. At1002, an object is identified as a human user in the augmented reality environment. For example, the camera210may recognize the face of the user as a human face, or the microphone218may recognize a sound from a particular location as a human voice.

At1004, a posture of the user is derived. For example, the posture module122may use data from the spatial analysis module114to generate the derived posture502. This derived posture may be a model comprising the relative position of the user origin704and the gestural terminus504such as directly measured by sensors in the augmented reality environment. In some implementations a more detailed anatomically-based model may be constructed, such as illustrated at502.

At1006, a location of a user origin is determined relative to the derived posture. For example, the dominant eye of the user may be designated as the user origin704.

At1008, a location of one or more gestural termini is determined based at least in part on the derived posture. For example, the tip of an extended finger of an extended hand may be designated as a gestural terminus504.

At1010, the designation vector708may be determined based at least in part upon the location of the user origin704and the location of the gestural terminus504. As mentioned above with regards toFIG. 9, in some implementations the designation vector708may comprise motion of the gestural terminus504, such as a direction in which a user is moving a hand where the hand is the gestural terminus504.

At1012, at least a portion of the object which corresponds to the designation vector708is determined. For example, this correspondence may be indicated by an endpoint of the designation vector708intersecting a particular portion of the object as determined by the selection module124.

At1014, at least a portion of the object corresponding to the designation vector708is designated as a selection. For example, the selection module124may generate data for use by the environment indicating that the far side of the virtual object has been selected, allowing further manipulation of that object. This manipulation may include designating for further functions, illuminating with the projector, changing presentation of a virtual object, displaying information or functions associated with the object, and so forth.

FIG. 11is an illustrative process1100of deriving a posture. As mentioned above, the posture may be derived from data acquired in the augmented reality environment, such as from sensors of the ARFN102. This derived posture may comprise a more detailed anatomical model containing a plurality of segments, such as illustrated at502and described next.

At1102, a baseline human physical arrangement is accessed. This baseline human physical arrangement may contain basic physiological data such as a human generally has two eyes, two arms, two legs, is bilaterally symmetrical, bipedal, and that joints have particular ranges of motion. A library of baselines may be stored to account for various different body types.

At1104, data is acquired describing physical characteristics of a particular user in the environment. For example, the user's height, arm length, finger size, and so forth may be measured using structured light generated and acquired by the ARFN102.

At1106, the baseline human physical arrangement is scaled to correspond to the data. For example, the baseline may be scaled to reflect the user's actual height, actual arm length, and so forth. In some implementations this scaling may be made using a subset of available measurements. For example, given a user's overall height and the distance from their elbow to finger tip, assumptions may be made about other physical elements reducing or eliminating the need to measure or acquire data such as the length of the user's lower leg. The baseline may also reflect a standard or reference position, such as the user standing, facing forward, arms extended to the sides.

At1108, a derived posture of the user is generated, based at least in part upon the scaled baseline. For example, the baseline may have been scaled to reflect the size of the actual user, and the position of the joints adjusted to reflect the actual position of the user. As mentioned above, the derived posture504may be used to designate a user origin704, determine the gestural terminus504, and so forth.

FIG. 12is an illustrative process1200of determining the user origin704. The user origin704may be statically defined, such as a particular point within the derived posture502, or it may be dynamic and vary location depending upon the derived posture502or other factors in the environment.

In some implementations such as described in this figure, the user origin704may vary in location. At1202, a field of view of a user is determined. In some implementations the field of view may be pre-determined such as provided by the human baseline described above with regards toFIG. 11. In other implementations, the field of view may be determined by presenting visual stimuli at known positions relative to the user and determining the user's reaction thereto. The field of view may be for one or both eyes.

At1204, a determination is made as to when the gestural terminus504is within the field of view. When the gestural terminus504is within the field of view, the process may proceed to1206. At1206, the dominant eye of the user may be designated as the user origin704. When at1184the gestural terminus504is outside of the user's field of view, the process proceeds to1208.

At1208, a point in the derived posture502other than the dominant eye is designated as the user origin704. For example, a point midway between the user's eyes or the position of a shoulder associated with a dominant hand may be designated as the user origin.

CONCLUSION