Architecture for augmented reality environment

An architecture has one or more systems for generating augmented reality environments configured to access cloud services over a network. A user is authenticated within the environments, and once authenticated is permitted to interact with the cloud services using the augmented reality environments as well as client resources provided within the environments. The client resources may include devices or things that are primary intended for use within the environments, and devices or things that are not typically associated with augmented reality environments. The architecture enables the client resources to function as thin client interfaces to the cloud services.

BACKGROUND

Augmented reality allows interaction among users, real-world objects, and virtual or computer-generated objects and information within an augmented reality environment. Within the augmented reality environment, images may be presented on various objects and users may interact with the images and/or objects in a number of ways. However, maintaining images on these objects, receiving input from a user's interaction with these objects, and so forth often pose challenges. What is desired is an augmented reality environment where essentially any type of device or object may become part of the environment and useable for interaction with users.

DETAILED DESCRIPTION

Augmented reality environments allow users to interact with physical and virtual objects in a physical space. Augmented reality environments may be resource intensive, including resources such as cameras, projectors, computing devices with processing and memory capabilities, and so forth. In addition to these resources used to define the augmented reality environments, other devices or objects may be present within an environment that may, at times, be used in some manner to further facilitate user interaction within the augmented reality environment. For example, a room that is equipped to provide an augmented reality environment may also include other devices that are not typically nor intended to be part of the environment. Such devices may range widely in functionality and capabilities, with example devices being televisions, computers, portable devices (phones, PDAs, tablets, etc.), limited-functioning clients (i.e., clients with limited processing and/or memory capabilities), passive devices (i.e., devices that can provide some functionality either mechanically or under power from an external source) and even non-electronic devices (e.g., surface, pad, etc.).

Described herein is an architecture to create augmented reality environments (ARE) and leverage other non-ARE resources that may be available within the environments to further extend users' abilities to interact within the environments. Within this architecture, the system used to create the augmented reality environment is connected to networking and computing resources, such as cloud services, which are external to the system. Certain types of non-ARE resources may be connected to cloud services as well, allowing those resources to be leveraged to further enhance the augmented reality environment.

The architecture may be implemented in many ways, several examples of which are described below. The following discussion begins with a description of the system used to create an augmented reality environment, and then proceeds with a description of an integrated architecture involving ARE resources and non-ARE resources.

Illustrative Environment

FIG. 1shows an illustrative augmented reality environment100which includes an augmented reality functional node (ARFN)102with associated computing device104. In this illustration, multiple ARFNs102are positioned around the scene, in this case a room. In other implementations, a single ARFN102may be used and positioned in any number of arrangements, such as on the ceiling, in a lamp, on the wall, and so forth. One implementation of the ARFN102is provided below in more detail with reference toFIG. 2.

The ARFN102is coupled to the computing device104, which may be located within the environment100or disposed at another location. The ARFN may be connected to the computing device104via a wired network, a wireless network, or a combination of the two. The computing device104has a processor106, an input/output interface108, and a memory110. The processor106may include one or more processors configured to execute instructions. The instructions may be stored in memory110, or in other memory accessible to the processor106, such as storage in the cloud.

The input/output interface108may be configured to couple the computing device104to other components, such as projector, cameras, microphones, other ARFNs102, other computing devices, and so forth. The input/output interface108may further include a network interface109that facilitates connection to a remote computing system, such as cloud computing resources. The network interface109enables access to one or more network types, including wired and wireless networks. More generally, the coupling between the computing device104and any components may be via wired technologies (e.g., wires, fiber optic cable, etc.), wireless technologies (e.g., RF, cellular, satellite, Bluetooth, etc.), or other connection technologies.

The memory110may include computer-readable storage media (“CRSM”). The CRSM may be any available physical media accessible by a computing device to implement the instructions stored thereon. CRSM may include, but is not limited to, random access memory (“RAM”), read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technology, compact disk read-only memory (“CD-ROM”), digital versatile disks (“DVD”) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.

Several modules such as instruction, datastores, and so forth may be stored within the memory110and configured to execute on a processor, such as the processor106. An operating system module112is configured to manage hardware and services within and coupled to the computing device104for the benefit of other modules.

A spatial analysis module114is configured to perform several functions which may include analyzing a scene to generate a topology, recognizing objects in the scene, dimensioning the objects, and creating a 3D model of the scene. Characterization may be facilitated using several technologies including structured light, light detection and ranging (LIDAR), optical time-of-flight, ultrasonic ranging, stereoscopic imaging, radar, and so forth either alone or in combination with one another. For convenience, and not by way of limitation, the examples in this disclosure refer to structured light. The spatial analysis module114provides the information used within the augmented reality environment to provide an interface between the physicality of the scene and virtual objects and information.

A system parameters datastore116is configured to maintain information about the state of the computing device104, the input/output devices of the ARFN102, and so forth. For example, system parameters may include current pan and tilt settings of the cameras and projectors. As used in this disclosure, the datastore includes lists, arrays, databases, and other data structures used to provide storage and retrieval of data.

An object parameters datastore118in the memory110is configured to maintain information about the state of objects within the scene. The object parameters may include the surface contour of the object, overall reflectivity, color, and so forth. This information may be acquired from the ARFN102, other input devices, or via manual input and stored within the object parameters datastore118.

An object datastore120is configured to maintain a library of pre-loaded reference objects. This information may include assumptions about the object, dimensions, and so forth. For example, the object datastore120may include a reference object of a beverage can and include the assumptions that beverage cans are either held by a user or sit on a surface, and are not present on walls or ceilings. The spatial analysis module114may use this data maintained in the datastore120to test dimensional assumptions when determining the dimensions of objects within the scene. In some implementations, the object parameters in the object parameters datastore118may be incorporated into the object datastore120. For example, objects in the scene which are temporally persistent, such as walls, a particular table, particular users, and so forth may be stored within the object datastore120. The object datastore120may be stored on one or more of the memories of the ARFN102, storage devices accessible on the local network, or cloud storage accessible via a wide area network.

A user identification and authentication module122is stored in memory110and executed on the processor(s)106to use one or more techniques to verify users within the environment100. In this example, a user124is shown within the room. In one implementation, the ARFN102may capture an image of the user's face and the spatial analysis module114reconstructs 3D representations of the user's face. Rather than 3D representations, other biometric profiles may be computed, such as a face profile that includes key biometric parameters such as distance between eyes, location of nose relative to eyes, etc. In such profiles, less data is used than full reconstructed 3D images. The user identification and authentication module122can then match the reconstructed images (or other biometric parameters) against a database of images (or parameters), which may be stored locally or remotely on a storage system or in the cloud, for purposes of authenticating the user. If a match is detected, the user is permitted to interact with the system.

In some implementations, multiple levels of authentication may be used. For instance, authentication may involve primary user data in the form of a biometric parameter (e.g., face image, face profile, fingerprint, etc.) and secondary user data in a different form. Examples of secondary user data include, for example, a non-vocal audio command (e.g., sequence of claps, or snaps) or a detectable body movement. Secondary user data may also be confirmation that the user's cell phone, or unique identifier fob, is also present in the environment.

In another implementation, the room may be equipped with other mechanisms used to capture one or more biometric parameters pertaining to the user, and feed this information to the user identification and authentication module122. For instance, a scanner may be mounted on the wall or embedded in the ARFN to scan the users fingerprint, or hand profile, or retina. In other implementations, the user may use verbal input and the module122verifies the user through an audio profile match. In still other implementations, the user may enter a pass code via a keypad or other input mechanism within the environment100.

An augmented reality module126is configured to generate augmented reality output in concert with the physical environment. The augmented reality module126may employ essentially any surface, object, or device within the environment100to interact with the user124. In this example, the room has walls128, a floor130, a chair132, a TV134, tables136and138, a computing device140, and a projection accessory display device (PADD)142. A PADD142may be essentially any device for use within an augmented reality environment, and may be provided in several form factors, including a tablet, coaster, placemat, tablecloth, countertop, tabletop, and so forth. A projection surface on the PADD facilitates presentation of an image generated by an image projector, such as the projector that is part of an augmented reality functional node (ARFN)102. The PADD may range from entirely non-active, non-electronic, mechanical surfaces to full functioning, full processing and electronic devices. Example PADDs are described in more detail with reference to U.S. patent application Ser. No. 12/977,949, which is entitled “Powered Augmented Reality Projection Accessory Display Device,” and was filed on Dec. 23, 2010, and to U.S. patent application Ser. No. 12/977,992, which is entitled “Unpowered Augmented Reality Projection Accessory Display Device,” and was filed on Dec. 23, 2010. These applications are hereby incorporated by reference.

Some of the things in the room may be intended to be part of the augmented reality environment (ARE) upon which images may be projected. Examples of such things may be the walls128, floor130, and tables136,138, and PADD142. These items will be referred to as ARE items. Other things may not normally be intended to be part of the augmented reality environment. These items will be referred to generally as non-ARE items. Such non-ARE items may include the TV134and computing device140. However, these non-ARE items may be temporarily used within the augmented reality environment when such devices are accessible, directly or indirectly, by the system of ARFNs102(1)-(N) and computing devices104.

Accordingly, the augmented reality module126may be used to track not only items within the environment that were previously identified by the spatial analysis module114, and stored in the various datastores116-120, but also the non-ARE items that reside in the environment or are ported into the environment. The augmented reality module126includes a tracking and control module144configured to track one or more ARE and/or non-ARE items within the scene and accept inputs from or relating to the ARE and/or non-ARE items.

FIG. 2shows an illustrative schematic200of the augmented reality functional node102and selected components. The ARFN102is configured to scan at least a portion of a scene202and the objects therein. The ARFN102may also be configured to provide augmented reality output, such as images, sounds, and so forth.

A chassis204holds the components of the ARFN102. Within the chassis204may be disposed a projector206that generates and projects images into the scene202. These images may be visible light images perceptible to the user, visible light images imperceptible to the user, images with non-visible light, or a combination thereof. This projector206may be implemented with any number of technologies capable of generating an image and projecting that image onto a surface within the environment. Suitable technologies include a digital micromirror device (DMD), liquid crystal on silicon display (LCOS), liquid crystal display, 3LCD, and so forth. The projector206has a projector field of view208which describes a particular solid angle. The projector field of view208may vary according to changes in the configuration of the projector. For example, the projector field of view208may narrow upon application of an optical zoom to the projector. In some implementations, a plurality of projectors206may be used.

A camera210may also be disposed within the chassis204. The camera210is configured to image the scene in visible light wavelengths, non-visible light wavelengths, or both. The camera210has a camera field of view212which describes a particular solid angle. The camera field of view212may vary according to changes in the configuration of the camera210. For example, an optical zoom of the camera may narrow the camera field of view212. In some implementations, a plurality of cameras210may be used.

The chassis204may be mounted with a fixed orientation, or be coupled via an actuator to a fixture such that the chassis204may move. Actuators may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis204or components therein such as the projector206and/or the camera210. For example, in one implementation, the actuator may comprise a pan motor214, tilt motor216, and so forth. The pan motor214is configured to rotate the chassis204in a yawing motion. The tilt motor216is configured to change the pitch of the chassis204. By panning and/or tilting the chassis204, different views of the scene may be acquired. The spatial analysis module114may use the different views to monitor objects within the environment.

One or more microphones218may be disposed within the chassis204, or elsewhere within the scene. These microphones218may be used to acquire input from the user, for echolocation, location determination of a sound, or to otherwise aid in the characterization of and receipt of input from the scene. For example, the user may make a particular noise, such as a tap on a wall or snap of the fingers, which are pre-designated to initiate an augmented reality function. The user may alternatively use voice commands. Such audio inputs may be located within the scene using time-of-arrival differences among the microphones and used to summon an active zone within the augmented reality environment. Further, the microphones218may be used to receive voice input from the user for purposes of identifying and authenticating the user. The voice input may be received and passed to the user identification and authentication module122in the computing device104for analysis and verification.

One or more speakers220may also be present to provide for audible output. For example, the speakers220may be used to provide output from a text-to-speech module, to playback pre-recorded audio, etc.

A transducer222may be present within the ARFN102, or elsewhere within the environment, and configured to detect and/or generate inaudible signals, such as infrasound or ultrasound. The transducer may also employ visible light, non-visible light, RF, or other ways to facilitate communication. These inaudible signals may be used to provide for signaling between accessory devices and the ARFN102.

A ranging system224may also be provided in the ARFN102to provide distance information from the ARFN102to an object or set of objects. The ranging system224may comprise radar, light detection and ranging (LIDAR), ultrasonic ranging, stereoscopic ranging, and so forth. In some implementations, the transducer222, the microphones218, the speaker220, or a combination thereof may be configured to use echolocation or echo-ranging to determine distance and spatial characteristics.

A wireless power transmitter226may also be present in the ARFN102, or elsewhere within the augmented reality environment. The wireless power transmitter226is configured to transmit electromagnetic fields suitable for recovery by a wireless power receiver and conversion into electrical power for use by active components within the PADD142. The wireless power transmitter226may also be configured to transmit visible light, non-visible light, RF, or other forms to communicate power. The wireless power transmitter226may utilize inductive coupling, resonant coupling, capacitive coupling, and so forth.

In this illustration, the computing device104is shown within the chassis204. However, in other implementations all or a portion of the computing device104may be disposed in another location and coupled to the ARFN102. This coupling may occur via wire, fiber optic cable, wirelessly, or a combination thereof. Furthermore, additional resources external to the ARFN102may be accessed, such as resources in another ARFN102accessible via a local area network, cloud resources accessible via a wide area network connection, or a combination thereof.

Also shown in this illustration is a projector/camera linear offset designated “O”. This is a linear distance between the projector206and the camera210. Placement of the projector206and the camera210at distance “O” from one another aids in the recovery of structured light data from the scene. The known projector/camera linear offset “O” may also be used to calculate distances, dimensioning, and otherwise aid in the characterization of objects within the scene202. In other implementations, the relative angle and size of the projector field of view208and camera field of view212may vary. Also, the angle of the projector206and the camera210relative to the chassis204may vary.

In other implementations, the components of the ARFN102may be distributed in one or more locations within the environment100. As mentioned above, microphones218and speakers220may be distributed throughout the scene. The projector206and the camera210may also be located in separate chassis204.

FIG. 3illustrates a structured light pattern300impinging on the scene202. In this illustration, the projector206within the ARFN102projects a structured light pattern302onto the scene202. In some implementations, a sequence of different structure light patterns302may be used. This structured light pattern302may be in wavelengths which are visible to the user, non-visible to the user, or a combination thereof. The structured light pattern304is shown as a grid in this example, but not by way of limitation. In other implementations other patterns may be used, such as bars, dots, pseudorandom noise, and so forth. Pseudorandom noise (PN) patterns are particularly useful because a particular point within the PN pattern may be specifically identified. A PN function is deterministic in that given a specific set of variables, a particular output is defined. This deterministic behavior allows the specific identification and placement of a point or block of pixels within the PN pattern.

The user124is shown within the scene202such that the user's face304is between the projector206and a wall. A shadow306from the user's body appears on the wall. Further, a deformation effect308is produced on the shape of the user's face304as the structured light pattern302interacts with the facial features. This deformation effect308is detected by the camera210, which is further configured to sense or detect the structured light. In some implementations, the camera210may also sense or detect wavelengths other than those used for structured light pattern302.

The images captured by the camera210may be used for any number of things. For instances, some images of the scene are processed by the spatial analysis module114to characterize the scene202. In some implementations, multiple cameras may be used to acquire the image. In other instances, the images of the user's face304(or other body contours, such as hand shape) may be processed by the spatial analysis module114to reconstruct 3D images of the user, which are then passed to the user identification and authentication module122for purposes of verifying the user.

Certain features of objects within the scene202may not be readily determined based upon the geometry of the ARFN102, shape of the objects, distance between the ARFN102and the objects, and so forth. As a result, the spatial analysis module114may be configured to make one or more assumptions about the scene, and test those assumptions to constrain the dimensions of the scene202and maintain the model of the scene.

Illustrative Architecture

FIG. 4shows an architecture400in which the ARFNs102(1)-(4) residing in the room are further connected to cloud services402via a network404. In this arrangement, the ARFNs102(1)-(4) may be integrated into a larger architecture involving the cloud services402to provide an even richer user experience. Cloud services generally refer to the computing infrastructure of processors, storage, software, data access, and so forth that is maintained and accessible via a network such as the Internet. Cloud services402do not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with cloud services include “on-demand computing”, “software as a service (SaaS)”, “platform computing”, and so forth.

As shown inFIG. 4, the cloud services402may include processing capabilities, as represented by servers406(1)-(S), and storage capabilities, as represented by data storage408. Applications410may be stored and executed on the servers406(1)-(S) to provide services to requesting users over the network404. Essentially any type of application may be executed on the cloud services402. One possible application is a user identification and authentication application. This application receives user data from an augmented reality environment, and attempts to verify a specific user from that data. The user data may be an image of certain physical features, such as facial features captured by the ARFN102. The user data may be a hand profile, or a fingerprint, or a voice file, or a retina scan, or other biometric data, or a pass code. The user data may be processed against a collection of authorized user profiles stored in the data storage408to determine whether this particular user should be authorized. In this manner, the cloud services402perform the user authentication task in addition, or as opposed, to the ARFN-resident module122shown inFIG. 1. Other examples of cloud services applications include sales applications, programming tools, office productivity applications, search tools, mapping and other reference applications, media distribution, social networking, and so on.

The network404is representative of any number of network configurations, including wired networks (e.g., cable, fiber optic, etc.) and wireless networks (e.g., cellular, RF, satellite, etc.). Parts of the network may further be supported by local wireless technologies, such as Bluetooth, ultra-wide band radio communication, wife, and so forth.

By connecting the ARFNs102(1)-(4) to the cloud services402, the architecture400allows the ARFNs102and computing devices104associated with a particular environment, such as the illustrated room, to access essentially any number of services. Consider, for instance, a scenario where the user124is in the process of drafting code for a particular software application. Upon entering the room, the user124is first authenticated using one of the techniques described above. For instance, the user's facial image is captured by one or more of the ARFNs102(1)-(4) and processed using the facial reconstruction techniques described above with respect toFIGS. 1 and 3.

Once authenticated, the user124can interact with the augmented reality environment to request the code under development from the cloud services402. The code may be projected by an ARFN102onto any viewable surface, such as the wall, floor, or table top. This is represented by ARFN102(3) projecting a portion of code412onto the right wall. Additionally or alternatively, the architecture400may, at least temporarily, leverage non-ARE items, such as the TV134, to assist in interacting with the user within the augmented reality environment. The TV134may be connected via a network to the cloud services402, or alternatively, the ARFN102and computing device104may communicate with the TV through local technologies (e.g., LAN, WLAN, Bluetooth, RF, HDMI, IR, etc.). In this example, the TV is used as a display device to present another portion of code414that is being worked on by the user124.

The architecture allows the user to interact more comfortably with coding tasks. This is particularly true for a team coding situation. Suppose that the user124is joined by two colleagues, programmers416and418. The other two users416and418may be identified and authenticated upon entering the room according to the techniques described herein. In this manner, each user in the environment is uniquely verified.

Different levels of access may be implemented to take advantage of multi-user authentication. For instance, the architecture may prohibit or limit access to certain applications or information when not all users in the environment have been authenticated. Alternatively, the users may be granted different access to applications and data. For instance, one user124may be allowed access to view content on any surface or any device within the environment, whereas the users416and418may be given access only to the code being presented more publicly on the walls and TV.

Other access polices may allow pooling or aggregation of rights. For instance, the collection of users within the environment may be given collective access to all applications and data that are accessible by at least one authenticated user. As an example, the users play lists of music or games may be aggregated to offer a broader opportunity to enjoy entertainment as a group. Alternatively, the policy may be to allow access only to the applications and data to which all three can access individually. Further, another possible access policy is to permit use of applications and data only when a collection of the users are present. In other words, all users are present and authenticated before access is granted. This may be useful, for example, during executive or board meetings where voting quorums and the like are held.

In this case, suppose each user124,416, and418is independently authenticated prior to allowing access to, and presenting code in a viewable manner. The three programmers can comfortably team program a piece of code through the augmented reality environment. With code portion414presented on the TV134and code portion412projected onto the wall, the team of three programmers124,416,418may see more code and comfortably move about the room while coding. The programmers may employ any number of ways to enter new lines of code or navigate, including voice or motion commands. Alternatively, the programmers may utilize a keyboard420to enter code. The keyboard420may be a physical keyboard or an image projected onto a surface by the ARFN102and user input is captured by the same or different ARFN102. As more code is entered by the programming team, that code is saved to local computing device104and/or to the cloud services402.

In this coding scenario, the cloud services402leverages the augmented reality environment created within the room and additional resources (e.g., TV, keyboard, etc.) as clients for interacting with the users. Further, by coupling individual environments to the cloud services402, the architecture400extends the augmented reality environment for the users from one scene to other scenes in other physical locations. This allows the architecture400to support essentially any activity at essentially any locations where an augmented reality environment may be created and the user can be identified and authenticated.

FIG. 5illustrates a storyline500that presents three different scenes502,504, and506in which the same user interacts with services provided by the cloud services via different augmented reality environments. A first scene502shows the team coding scenario ofFIG. 4, where the user124drafts software code with two colleagues. The ARFNs102used to create an augmented reality environment within the room are also connected to the cloud services402via the network404.

After drafting code, suppose the user124has a meeting with two other managers508and510, as represented by the second scene504. In this scene, the user124is in a conference room that is at a different physical location than the room in scene502. At this meeting, the user124is authenticated by the local ARFNs512, which are also connected to the cloud services402. Once authenticated, the user can request data from the cloud services and that data may be presented, for instance, on the user's PADD142. The data may be projected onto the PADD142via one of the ARFNs508in the scene504. During the meeting, the user124may elect to share the data kept in a file with the other managers508,510. These managers may individually be authenticated by the ARFNs508and verified to the cloud services402. The user124may initiate a file transfer via a voice command or other audible signal (finger snap, clap, etc.) picked up by the microphones of the ARFNs512, or through use of hand motion that is captured by the cameras of the ARFNs, or through other interaction techniques.

The file may be transferred in any number of ways. In one case, the cloud services402share the user's file with the laptop514of the first manager508via the network404, to which the laptop514is connected. In another case, the data being projected onto the PADD128by one ARFN512is additionally projected onto the table surface516by a different ARFN for viewing by the second manager510.

After the manager meeting in scene504, the user departs for the airport for a business trip. In scene506, the user124approaches a kiosk518at the airport to check-in for a flight. The kiosk518is equipped with a ARFN520that creates a local augmented reality environment proximal to the kiosk. The ARFN520captures the user's facial features, and submits the images to the cloud services402for identification and authentication. In this manner, the kiosk518may be a comparatively low functioning machine, relying on the cloud services402for processing capabilities. Once the user124is identified and authenticated, ticketing and gate information may be provided to the user124.

As shown, all three scenes involve the same user124, who is able to move freely among multiple augmented reality environments. The user may be authenticated within each scene, but once this occurs, the user is able to access the same resources (e.g., applications and data) maintained at the cloud services402.

FIG. 6illustrates an abstraction600of the architecture in which an augmented reality environment and any number of resources therein may be leveraged as clients to interact with applications and data provided by the cloud services402. In this illustration, the cloud services402are coupled via the network404to a system of ARFNs, as represented by ARFN102, and/or to resources equipped with network access capabilities. ARE resources that may be employed by the ARFN102include essentially any surface onto which an image may be projected and/or any device that may depict an image. Examples include the surface of table138, the television134, and any number of implementations of the PADD142(1)-(P). As noted above, the PADDs may range from a simple piece of material having a projection surface, to mechanical tablet with some mechanical functionality (e.g., audible sound generation), to a low functioning electronic devices (e.g., limited memory and processing, and remotely powered), to powered electronic devices (e.g., display, processor, memory, etc), to full functioning computing devices.

The architecture may also make use of resources in the augmented reality environments that may not generally be intended for use within the environments. Such non-ARE devices may themselves be leveraged by the architecture through their connection with the cloud services402via the network404or through some form of communication with the ARFN102. Representative non-ARE devices include a tablet602, a portable computer140, a desktop computer604, and a server606.

Illustrative Process

At702, a user is detected upon entry to an augmented reality environment. This is represented pictorially inFIG. 7, where a user is detected entering a room equipped with one or more ARFNs. User detection may be executed by sensing disruption in the structured light pattern projected into an otherwise empty room. Other detection techniques may include motion sensing, sound reception, heat sensors, sensors on the door to the room, and so forth.

At704, the user's identity is ascertained and authenticated. User authentication may be performed in a number of ways, including using the ARFNs to capture the image of the user's face and to verify the features as belonging to the user. Other authentication techniques include voice recognition, hand profile, finger print identification, retina scan, entry of a code, and so forth. The user authentication may be performed locally within the environment, such as by computing device104, or remote from the environment, such as by the cloud services402, or a combination of the two.

In another implementation, different levels of authentication may be performed locally and remotely in the cloud. For instance, the architecture may capture an image of the users face (or other biometric parameters) and perform user verification locally. Upon local verification, the user may then request a service from the remote cloud services402that involves a second level of authorization. For instance, the cloud services402may request a spoken password, a sequence of non-vocal audio inputs (e.g., sequence of claps, taps, etc.), a pass code, other non-biometric data, or any combination of these and biometric data.

In yet another implementation, the architecture may have different levels of authentication at the local or cloud level. For instance, upon entering the room, the user may speak a password for initial verification. Once resources are powered on, one or more user biometric parameters are captured and analyzed for verification purposes. If the user then requests some functionality locally or from the cloud, the user may be prompted for yet another authentication parameter that may be used alone or in combination with the others.

If multiple users are present, each user is identified and authenticated. Consider, for example, the coding scenario depicted inFIG. 4. In this case, each user124,416, and418may be authenticated prior to the system presenting code in a viewable manner. In one implementation, the system may detect the presence of a new user and suppress presentation of the images unless and until that user is authenticated as well. Further, once authenticated, each user may be given different levels of access. For instance, the users416and418may be given access solely to the code being presented on the walls and TV, whereas the user124may be given access to this as well as additional information provided on a PADD or other device. As another example, each user may have access to certain lists or information. When all are authenticated, the system may present an aggregation of the user lists and information for all three users to consider jointly.

At706, once the user is authenticated, the user may interact with the augmented reality environment. As part of that interaction, the user may access the cloud services remote from the augmented reality environment. The user may request data, process data, store results, share data with others, and so forth. Any number of services available from the cloud services may be made available to the user using the ARFNs that form the augmented reality environment. For instance, a programming tool may use the ARFNs to project code onto a wall or other surface as illustrated inFIG. 4. As another example, a social networking application may organize contacts such that family members are projected onto one surface, and friends are projected onto another surface.

At708, resources available in the environment may also be utilized to interact with the cloud services. The resources may include both items intended for use within the environment (i.e., ARE resources) and those that are not generally intended for use within the environment (i.e., non-ARE resources). The types of resources range widely, as described above with respect toFIG. 6. The resources may be used in a number of ways, including presenting information to the user through audible and/or visual output, and receiving or capturing input from the user.

At710, once a task is completed, the user may exit the cloud services. The user may continue to interact with the augmented reality environment, engaging resources local to the environment. The user may eventually leave the environment, ending the augmented reality session. As part of exiting the cloud services or leaving the environment, the user may be affirmatively de-authenticated. For instance, the system may recognize that the user is no longer participating in the environment and effectively remove the user from an active, authenticated state. In this manner, the user would need to go through the authentication process again to be able to function within the augmented reality environment and/or gain access to the services.

CONCLUSION