HEAD-MOUNTED INTEGRATED INTERFACE

A head mounted integrated interface (HMII) is presented that may include a wearable head-mounted display unit supporting two compact high resolution screens for outputting a right eye and left eye image in support of the stereoscopic viewing, wireless communication circuits, three-dimensional positioning and motion sensors, and a processing system which is capable of independent software processing and/or processing streamed output from a remote server. The HMII may also include a graphics processing unit capable of also functioning as a general parallel processing system and cameras positioned to track hand gestures. The HMII may function as an independent computing system or as an interface to remote computer systems, external GPU clusters, or subscription computational services, The HMII is also capable linking and streaming to a remote display such as a large screen monitor.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to computer interfaces and, more specifically, to a head-mounted integrated interface.

2. Description of the Related Art

The computer interface has not changed significantly since the Apple Macintosh was introduced in 1984. Most computers support some incarnation of an alphanumeric keyboard, a pointer such as a mouse, and a 2D display or monitor. Typically, computers support some form of a user interface that combines the keyboard and mouse input and provides visual feedback to the user via the display. Virtual reality (VR) technology introduced new methods for interfacing with computer systems. For example, VR displays that are head-mounted present separate left and right eye images in order to generate stereoscopic 3-dimensional (3D) displays, include input pointer devices tracked in 3D space, and output synchronized audio to the user to provide a richer feeling of immersion in the scene being displayed.

User input devices for the VR system include data gloves, joysticks, belt mounted keypads, and hand-held wands. VR systems allow users to interact with environments ranging from a completely virtual environment generated entirely by a computer system to an augmented reality environment in which computer generated graphics are superimposed onto images of a real environment. Representative examples range from virtual environments such as molecular modeling and video games to augmented reality applications such as remote robotic operations and surgical systems.

One drawback in current head-mounted systems is that in order to provide high resolution input and display, the head-mounted interface systems are tethered via communication cables to high performance computer and graphic systems. The cables not only restrict the wearer's movements, the cables also impair the portability of the unit. Other drawbacks of head-mounted displays are bulkiness and long processing delays between tracker information input and visual display update which is commonly referred to as tracking latency. Extended use of current head-mounted systems can adversely affect the user causing physical pain and discomfort.

As the foregoing illustrates, what is needed in the art is a more versatile and user-friendly head-mounted display system.

SUMMARY OF THE INVENTION

One embodiment of the present invention is, generally, an apparatus configured to be wearable by a user and includes at least one display in front of the user's eyes on which a computer-generated image is displayed. The apparatus also incorporates at least two cameras with overlapping fields-of view to allow a gesture of the user to be captured. Additionally, a gesture input module is coupled to the cameras and is configured to receive visual data from the cameras and identify the user gesture within the visual data. The identified gesture is then used to affect the computer-generated image presented to the user.

Integrating the cameras and the gesture input module into a wearable apparatus improve the versatility of the device relative to other devices that require separate input devices (e.g., keyboards, mice, joysticks, and the like) to permit the user to interact with the virtual environment displayed by the apparatus. Doing so also improves the portability of the apparatus because, in at least one embodiment, the apparatus can function as a fully operable computer that can be used in a variety of different situations and scenarios where other wearable devices are ill-suited.

DETAILED DESCRIPTION

Head-Mounted Integrated Interface System Overview

FIG. 1is a diagram illustrating a head-mounted integrated interface (HMII)100configured to implement one or more aspects of the present invention. As shown, HMII100includes a frame102configured to be wearable on a user's head via some means such as straps120as shown. In one embodiment, the frame102may contain a processor system which will be described in greater detail below. Frame102includes a display that includes a left-eye display104positioned to be viewed by the user's left eye and a right-eye display106positioned to be viewed by the user's right eye. The displays104and106are not limited to any specific display technology and may include LCDs, LEDs, projection displays, and the like. Incorporated into frame102are additional input devices that include, but are not limited to, a left look-down camera108, a right look-down camera110, a right look-front camera114, a left look-front camera112, a left look-side camera116, and a right look-side camera118.

In one embodiment, gesture input is captured by the cameras (e.g., cameras108and110and processed by the processor system, or external systems with which the HMII100is in communication. Based on the detected gestures, the HMII100may update the image presented on the displays104and106accordingly. To do so, the fields-of-view of at least two of the cameras in the HIID100may be at least partially overlapping to define a gesture sensing region. For example, the user may perform a hand gesture that is captured in the overlapping fields-of-views and is used by the HMII100to alter the image presented on the displays104and106.

In one embodiment, the left look-down camera108and right look-down camera110are oriented so that at least a portion of the respective fields-of-view overlap in a downward direction—e.g., from the eyes of the user towards the feet of the user. Overlapping the fields-of-view may provide additional depth information for identifying the user gestures. The left look-front camera112and right look-front camera114may also be oriented so that at least a portion of their respective fields-of-view overlap. Thus, the visual data captured by cameras112and114may also be used to detect and identify user gestures. As shown generally inFIG. 1, the various cameras (including side looking cameras116and118) illustrate that different fields-of-view can be established to increase the gesture recognition region for detecting and identifying user gestures. Increasing the size of the gesture recognition region may also aid the HMII100to identify gestures made by others who near the user.

The camera1108,110,112,114,116, and118may also be used to augment the apparent field of view which may be presented in the left- and right-eye displays104and106. Doing so may aid the user to identify obstructions that would otherwise be outside the field of view presented in the displays104and106.

Situating at least two of the input cameras with overlapping fields of view, as for example left look-down camera108and right look-down camera110, enables stereoscopic views of hand motions that in turn permits very fine resolution, distinguishing user movements to sub-millimeter accuracy, and discrimination of movements not just from the visual context surrounding the gestures but also discriminating between gestures that have nearly identical features. Fine movement resolution thereby enables greater accuracy in interpreting the movements and thereby greater accuracy and reliability in translating gestures into command inputs. Some examples include: wielding virtual artifacts in gaming environments such as swords or staffs, coloring information in the environment in an augmented reality application, exchanging virtual tools or devices between multiple users in a shared virtual environment, etc. It should be noted that gestures are not restricted to hands of the user or even to hands generally and that these examples are not exhaustive. Gesture recognition can include other limb movements, object motions, and/or analogous gestures made by other users in a shared environment. Gestures may also include without limitation myelographic signals, electroencephalographic signals, eye tracking, breathing or puffing, hand motions, and so forth, whether from the wearer or another participant in a shared environment.

Another factor in the handling of gestures is the context of the virtual environment being displayed to the user when a particular gesture is made. The simple motion of pointing with an index finger when a word processing application is being executed on the HMII100could indicate a particular key to press or word to edit. The same motion in a game environment could indicate that a weapon is to be deployed or a direction of travel.

Gesture recognition offers a number of advantages for shared communication or networked interactivity in applications such as medical training, equipment operation, and remote or tele-operation guidance. Task specific gesture libraries or neural network machine learning could enable tool identification and feedback for a task. One example would be the use of a virtual tool that translates into remote, real actions. For example, manipulating a virtual drill within a virtual scene could translate to the remote operation of a drill on a robotic device deployed to search a collapsed building. Moreover, the gestures may also be customizable. That is, the HMII100may include a protocol for enabling a user to add a new gesture to a list of identifiable gestures associated with user actions.

In addition, the various cameras in the HMII100may be configurable to detect spectrum frequencies in addition to the visible wavelengths of the spectrum. Multi-spectral imaging capabilities in the input cameras allows position tracking of the user and/or objects by eliminating nonessential image features (e.g., background noise). For example, in augmented reality applications such as surgery, instruments and equipment can be tracked by their infrared reflectivity without the need for additional tracking aids. Moreover, HMII100could be employed in situations of low visibility where a “live feed” from the various cameras could be enhanced or augmented through computer analysis and displayed to the user as visual or audio cues.

In one embodiment, the HMII100is capable of an independent mode of operation where the HMII100does not perform any type of data communication with a remote computing system or need power cables. This is due in part to power unit which enables the HMII100to operate independently free from external power systems. In this embodiment, the HMII100may be completely cordless without a wired connection to an external computing device or a power supply. In a gaming application, this mode of operation would mean that a player could enjoy a full featured game anywhere without being tethered to an external computer or power unit. Another practical example of fully independent operation of the HMII100is a word processing application. In this example, the HMII100would present, using the displays104and106, a virtual keyboard, virtual mouse, and documents to the user via a virtual desktop or word processing scene. Using gesture recognition data captured by one or more of the cameras, the user may type on a virtual keyboard or move a virtual mouse using her hand which then alters the document presented on the displays104and105. Advantageously, the user has to carry only the HMII100rather than an actual keyboard and/or mouse when moving to a different location. Moreover, the fully contained display system offers the added advantage that documents are safer from prying eyes.

in operation, frame102is configured to fit over the user's eyes positioning the left-eye display104in front of the user's left eye and the right-eye display106in front of the user's right eye. In one embodiment, processing module is configured to fully support compression/decompression of video and audio signals. Also, the left-eye display104and right-eye display106are configured within frame102to provide separate left eye and right eye images in order to create the perception of a three-dimensional view. In one example, left-eye display104and right-eye display106have sufficient resolution so to provide a high-resolution field of view greater than 90 degrees relative to the user. In one embodiment, left-eye display104and right-eye display106positions in frame102are adjustable to match variations in eye separation between different users.

In another embodiment left-eye display104and right-eye display108may be configured using organic light-emitting diode or other effective technology to permit “view-through” displays. View-through displays, for example, permit a user to view the surrounding environment, through the display. This creates an effect of overlaying or integrating computer generated visual information into the visual field (referred to herein as “augmented reality”). Applications for augmented reality range from computer assisted operating manuals for complex mechanical systems to surgical training or telemedicine. With view-through display technology, frame102would optionally support an opaque shield that would be used to screen out the surrounding environment. One embodiment of the HMII100display output in the view-through mode is super-position of graphic information into a live feed of the user's surroundings, The graphic information may change based on the gestures provided by the user. For example, the user may point at different objects in the physical surroundings. In response, the HMII100may superimpose supplemental information on the displays104and106corresponding to the objects pointed to by the user. In one example, the integrated information may be displayed in a “Heads-up Display” (HUD) operation. The HUD mode may then be used in navigating unfamiliar locations, visually highlighting key components in machinery or an integrated chip design, alert the wearer to changing conditions, etc.

AlthoughFIG. 1illustrates six cameras, this is for illustration purposed only. Indeed, the HMII100may include less than six or more than six cameras and still perform the functions described herein.

Functional Components of the HMII

FIG. 2is a block diagram of the functional components in the HMII100, according to one embodiment of the present invention. To enhance its portability, the HMII100may include a power unit230within frame102. As discussed above, the cameras520may provide input to the gesture recognition process as well as visual data regarding the immediate environment. The cameras520communicate with the processor system500via the I/O bridge507as indicated. Specifically, the data captured by the cameras520may be transported by the bridge507to the processor system500for further processing.

The displays104and106described above are components of the display device510inFIG. 2. The displays104and106are contained in frame102and may be arranged to provide a two- or three-dimensional image to the user. In the augmented reality configuration, the user is able to concurrently view the external scene with the computer graphics overlaid in the presentation. Because the display device510is also coupled to the I/O bridge507, the processor system500is able to update the image presented on display104and106based on the data captured by the cameras520

For instance, based on a detected gesture, the processor system500may alter the image displayed to the user.

Audio output modules514and audio input modules522may enable three-dimensional aural environments and voice control capabilities respectively. Three-dimensional aural environments provide non-visual cues and realism to virtual environments. An advantage provided by this technology is signaling details about the virtual environment that are outside the current field of view. Left audio output222and right audio output224provide the processed audio data to the wearer such that the sounds contain the necessary cues such as (but not limited to) delays, phase shifts, attenuation, etc. to convey the sense of spatial localization. This can enhance the quality of video game play as well as provide audible cues when the HMII100may be employed in augmented reality situation with, for example, low light levels. Also high-end audio processing of the left audio input232and right audio input234could enable voice recognition not just of the wearer but other speakers as well. Voice recognition commands could augment gesture recognition by activating modality switching, adjusting sensitivity, enabling the user to add a new gesture, etc.

FIG. 2also shows that frame102contains motion tracking module226which communicates position, orientation, and motion tracking data to processor system500. In one embodiment, the motion tracking module226incorporates sensors to detect position, orientation, and motion of frame102. Frame102also contains interface points for external devices (not shown) to support haptic feedback devices, light sources, directional microphones, etc. The list of these devices is not meant to be exhaustive or limiting to just these examples.

A wireless module228is also contained within frame102and communicating to the processor system500. Wireless module228is configured to provide wireless communication between the processor system and remote systems. Wireless module228is dynamically configurable to support standard wireless protocols. Wireless module228communicates with processor system500through network adapter518.

Head-Mounted Integrated Interface Process Flow

FIG. 3is a flow diagram of detecting user gestures for changing the virtual environment presented to the user, according to one embodiment of the present invention. Although the steps are described in conjunction withFIGS. 1-6, persons skilled in the art will understand that any system configured to perform the steps, in any order, falls within the scope of the present invention.

As shown, a method300begins at step302, where the immediate or local environment of the user is sampled by at least one of the cameras in the HMII100. The visual data captured by the one or more of the cameras may include a user gesture.

At step304, the HMII100system detects a predefined user gesture or motion input based on the visual data captured by the cameras. A gesture can be as simple as pointing with an index finger or as complex as a sequence of motions such as American Sign Language, Gestures can also include gross limb movement or tracked eye movements. In one embodiment, accurate gesture interpretation may depend on the visual context in which the gesture is made, e.g. game environment gesture may indicate a direction of movement while the same gesture in an augmented reality application might request data output.

In one embodiment, to identify the gesture from the background image, the HMII100performs a technique for extracting the gesture data from the sampled real environment data. For instance, the HMII100may filter the scene to determine the gesture context, e.g. objects, motion paths, graphic displays, etc. that correspond to the situation in which the gestures were recorded. Filtering can, for example, remove jitter or other motion artifacts, reduce noise, and preprocess image data to highlight edges, regions-of-interest, etc. For example, in a gaming application if the HMII100is unable, because of low contrast, to distinguish between a user's hand and the background scene, user gestures may go undetected and frustrate the user. Contrast enhancement filters coupled with edge detection can obviate some of these problems.

At step306, the gesture input module of the HMII100interprets the gesture and/or motion data detected at step304. In one example, the HMII100includes a datastore or table for mapping the identified gesture to an action in the virtual environment being display to the user. For example, the certain gesture may map to a moving a virtual paintbrush. As the user moves her hand, the paintbrush in the virtual environment displayed by the HMII100follows the user's movements. In another embodiment, a single gesture may map to different actions. As discussed above, the HMII100may use a context of the virtual environment to determine which action maps to the particular gesture.

At step308, HMII100updates the visual display and audio output based on the action identified in step306by changing the virtual environment and/or generating a sound. For example, in response to a user swinging her arm (i.e., the user gesture), the HMII100may manipulate an avatar in the virtual environment to strike an opponent or object in the game environment (i.e., the virtual action). Sound and visual cues would accompany the action thus providing richer feedback to the user and enhancing the game experience.

HMII Operation with External Systems

FIG. 4Aillustrates one configuration for use of the HMII100with local resources, according to one embodiment of the present invention. Local resources, for example, can be those systems available over a secure data transfer channel (e.g., a direct communication link between the HMII100and the local resource) and/or network (e.g., a LAN). HMII100can wirelessly interface to external general purpose computer404and/or a Graphics Processing Unit (GPU) cluster402to augment the computational resources for HMII100. The processor system described below further enables data compression of the visual data both to and from the HMII100.

By integrating with external computer systems402and404, the HMII100is able to direct more of the “on-board” processing resources to the task of interfacing with complex virtual environments such as, but not limited to, integrated circuit analysis, computational fluid dynamics, gaming applications, molecular mechanics, teleoperation controls. Stated differently, the HMII100can use its wireless connection to the general purpose computer404or CPU cluster402to leverage the graphic processors on these components in order to, e.g., conserve battery power or perform more complex tasks.

Moreover, advances in display technology and video signal compression technology have enabled high quality video delivered wirelessly to large format displays such as large format wall mounted video systems. The HMII100is capable of wirelessly driving a large format display406similar to large format wall mounted systems. In addition to the capability to display wireless transmissions, HMII100has sufficient resolution and refresh rates to synchronize with the HMII100displays and the external display device such as the large format display406, concurrently. Thus, whatever is being displayed to the user via the display in the HMII may also be displayed on the external display406. As the HMII100uses, for example, gestures to alter the virtual environment, these alterations are also presented on the large format display406.

FIG. 4Billustrates a configuration for use of the HMII100with networked resources, according to one embodiment of the present invention. In this embodiment, the HMII100may use standard network protocols (e.g., TCP/IP) for communicating with the networked resources. Thus, the HMII100may be fully compatible with resources available over the Internet414including systems or software as a service (SaaS)410, subscription services for servers, and Internet II. The HMII100can connect via the Internet414with subscription service410that provides resources such as, but not limited to, GPU servers412. As described above, the GPU servers412may augment the computational resources for HMII100. By leveraging the graphics processing resources in the servers412to aid in generating the images displayed to the user, the HMII100may be able to perform more complex task than otherwise would be possible.

In one embodiment, the subscription service410may be a cloud service which further enhances the portability of the HMII100. So long as the user has an Internet connection, the HMII100can access the service410and take advantage of the GPU servers412included therein. Because multiple user subscribe to the service410, this may also defray the cost of upgrading the GPU servers410as new hardware/software is released.

Representative Processor Overview

FIG. 5is a block diagram illustrating a processor system500configured to implement one or more aspects of the present invention. Any processor known or to be developed that provides the minimal necessary capabilities such as (and without limitation) compute capacity, power consumption, and speed may be used in implementing one or more aspects of the present invention. As shown, processor system500includes, without limitation, a central processing unit (CPU)502and a system memory504coupled to a parallel processing subsystem512via a memory bridge505and a communication path513. Memory bridge505is further coupled to an I/O (input/output) bridge507via a communication path506, and I/O bridge507is, in turn, coupled to a switch516.

In operation, I/O bridge507is configured to receive information (e.g., user input information) from input cameras520, gesture input devices, and/or equivalent components, and forward the input information to CPU502for processing via communication path506and memory bridge505. Switch516is configured to provide connections between I/O bridge507and other components of the computer system500, such as a network adapter518. As also shown, I/O bridge507is coupled to an audio output modules514that may be configured to output audio signals synchronized with the displays. In one embodiment audio output modules514is configured to implement a 3D auditory environment. Finally, although not explicitly shown, other components, such as universal serial bus or other port connections, compact disc drives, digital versatile disc drives, film recording devices, and the like, may be connected to I/O bridge507as well.

In various embodiments, memory bridge505may be a Northbridge chip, and I/O bridge507may be a Southbrige chip. In addition, communication paths506and513, as well as other communication paths within computer system500, may be implemented using any technically suitable protocols, including, without limitation, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

In some embodiments, parallel processing subsystem512comprises a graphics subsystem that delivers pixels to a display device510that may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like. In such embodiments, the parallel processing subsystem512incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. As described in greater detail below inFIG. 6, such circuitry may be incorporated across one or more parallel processing units (PPUs) included within parallel processing subsystem512. In other embodiments, the parallel processing subsystem512incorporates circuitry optimized for general purpose and/or compute processing. Again, such circuitry may be incorporated across one or more PPUs included within parallel processing subsystem512that are configured to perform such general purpose and/or compute operations. In yet other embodiments, the one or more PPUs included within parallel processing subsystem512may be configured to perform graphics processing, general purpose processing, and compute processing operations. System memory504includes at least one device driver503configured to manage the processing operations of the one or more PPUs within parallel processing subsystem512. System memory504also includes a point of view engine501configured to receive information from input cameras520, motion tracker, or other type of sensor. The point of view engine501then computes field of view information, such as a field of view vector, a two-dimensional transform, a scaling factor, or a motion vector. The point of view information may then be forwarded to the display device510.

In various embodiments, parallel processing subsystem512may be integrated with one or more of the other elements ofFIG. 5to form a single system. For example, parallel processing subsystem512may be integrated with the, memory bridge505, I/O bridge507, and/or other connection circuitry on a single chip to form a system on chip (SoC).

It will be appreciated that the system shown herein is illustrative and that variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of CPUs502, and the number of parallel processing subsystems512, may be modified as desired. For example, in some embodiments, system memory504could be connected to CPU502directly rather than through memory bridge505, and other devices would communicate with system memory504via CPU502. In other alternative topologies, parallel processing subsystem512may be connected to I/O bridge507or directly to CPU502, rather than to memory bridge505. In still other embodiments, I/O bridge507and memory bridge505may be integrated into a single chip instead of existing as one or more discrete devices. Lastly, in certain embodiments, one or more components shown inFIG. 5may not be present. For example, switch516could be eliminated and network adapter518would connect directly to I/O bridge507.

FIG. 6is a block diagram of a parallel processing unit (PPU)602included in the parallel processing subsystem512ofFIG. 5, according to one embodiment of the present invention. AlthoughFIG. 6depicts one PPU602having a particular architecture, as indicated above, parallel processing subsystem512may include any number of PPUs602having the same or different architecture. As shown, PPU602is coupled to a local parallel processing (PP) memory604. PPU602and PP memory604may be implemented using one or more integrated circuit devices, such as programmable processors, application specific integrated circuits (ASICs), or memory devices, or in any other technically feasible fashion.

In some embodiments, PPU602comprises a graphics processing unit (CPU) that may be configured to implement a graphics rendering pipeline to perform various operations related to generating pixel data based on graphics data supplied by CPU502and/or system memory504. When processing graphics data, PP memory604can be used as graphics memory that stores one or more conventional frame buffers and, if needed, one or more other render targets as well. Among other things, PP memory604may be used to store and update pixel data and deliver final pixel data or display frames to display device510for display. In some embodiments, PPU602also may be configured for general-purpose processing and compute operations.

In operation, CPU502is the master processor of computer system500, controlling and coordinating operations of other system components. In particular, CPU502issues commands that control the operation of PPU602. In some embodiments, CPU502writes a stream of commands for PPU602to a data structure (not explicitly shown in eitherFIG. 5orFIG. 6) that may be located in system memory504, PP memory604, or another storage location accessible to both CPU502and PPU602. A pointer to the data structure is written to a pushbuffer to initiate processing of the stream of commands in the data structure. The PPU602reads command streams from the pushbuffer and then executes commands asynchronously relative to the operation of CPU502. In embodiments where multiple pushbuffers are generated, execution priorities may be specified for each pushbuffer by an application program via device driver503to control scheduling of the different pushbuffers.

As also shown, PPU602includes an input/output (I/O) unit605that communicates with the rest of computer system500via the communication path513and memory bridge505. I/O unit605generates packets (or other signals) for transmission on communication path513and also receives all incoming packets (or other signals) from communication path513, directing the incoming packets to appropriate components of PPU602. For example, commands related to processing tasks may be directed to a host interface606, while commands related to memory operations (e.g., reading from or writing to PP memory604) may be directed to a crossbar unit610. Host interface606reads each pushbuffer and transmits the command stream stored in the pushbuffer to a front end612.

As mentioned above in conjunction withFIG. 5, the connection of PPU602to the rest of computer system500may be varied. In some embodiments, parallel processing subsystem512, which includes at least one PPU602, is implemented as an add-in card that can be inserted into an expansion slot of computer system500. In other embodiments, PPU602can be integrated on a single chip with a bus bridge, such as memory bridge505or I/O bridge507. Again, in still other embodiments, some or all of the elements of PPU602may be included along with CPU502in a single integrated circuit or system of chip (SoC).

In operation, front end612transmits processing tasks received from host interface606to a work distribution unit (not shown) within task/work unit607. The work distribution unit receives pointers to processing tasks that are encoded as task metadata (TMD) and stored in memory. The pointers to TMDs are included in a command stream that is stored as a pushbuffer and received by the front end unit612from the host interface606. Processing tasks that may be encoded as TMDs include indices associated with the data to be processed as well as state parameters and commands that define how the data is to be processed. For example, the state parameters and commands could define the program to be executed on the data. The task/work unit607receives tasks from the front end612and ensures that GPCs608are configured to a valid state before the processing task specified by each one of the TMDs is initiated. A priority may be specified for each TMD that is used to schedule the execution of the processing task. Processing tasks also may be received from the processing cluster array630. Optionally, the TMD may include a parameter that controls whether the TMD is added to the head or the tail of a list of processing tasks (or to a list of pointers to the processing tasks), thereby providing another level of control over execution priority.

PPU602advantageously implements a highly parallel processing architecture based on a processing cluster array630that includes a set of C general processing clusters (GPCs)608, where C≧1. Each GPC608is capable of executing a large number (e.g., hundreds or thousands) of threads concurrently, where each thread is an instance of a program. In various applications, different GPCs608may be allocated for processing different types of programs or for performing different types of computations. The allocation of GPCs608may vary depending on the workload arising for each type of program or computation.

Memory interface614includes a set of D of partition units615, where D≧1. Each partition unit615is coupled to one or more dynamic random access memories (DRAMs)620residing within PPM memory604. In one embodiment, the number of partition units615equals the number of DRAMs620, and each partition unit615is coupled to a different DRAM620. In other embodiments, the number of partition units615may be different than the number of DRAMs620. Persons of ordinary skill in the art will appreciate that a DRAM620may be replaced with any other technically suitable storage device. In operation, various render targets, such as texture maps and frame buffers, may be stored across DRAMs620, allowing partition units615to write portions of each render target in parallel to efficiently use the available bandwidth of PP memory604.

A given GPCs608may process data to be written to any of the DRAMs620within PP memory604. Crossbar unit610is configured to route the output of each GPC608to the input of any partition unit615or to any other GPC608for further processing. GPCs608communicate with memory interface614via crossbar unit610to read from or write to various DRAMs620. In one embodiment, crossbar unit610has a connection to I/O unit605, in addition to a connection to PP memory604via memory interface614, thereby enabling the processing cores within the different GPCs608to communicate with system memory504or other memory not local to PPU602. In the embodiment ofFIG. 6, crossbar unit610is directly connected with I/O unit605. In various embodiments, crossbar unit610may use virtual channels to separate traffic streams between the GPCs608and partition units615.

Again, GPCs608can be programmed to execute processing tasks relating to a wide variety of applications, including, without limitation, linear and nonlinear data transforms, filtering of video and/or audio data, modeling operations (e.g., applying laws of physics to determine position, velocity and other attributes of objects), image rendering operations (e.g., tessellation shader, vertex shader, geometry shader, and/or pixel/fragment shader programs), general compute operations, etc. In operation, PPU602is configured to transfer data from system memory504and/or PP memory604to one or more on-chip memory units, process the data, and write result data back to system memory504and/or PP memory604. The result data may then be accessed by other system components, including CPU502, another PPU602within parallel processing subsystem512, or another parallel processing subsystem512within computer system500.

As noted above, any number of PPUs602may be included in a parallel processing subsystem512. For example, multiple PPUs602may be provided on a single add-in card, or multiple add-in cards may be connected to communication path513, or one or more of PPUs602may be integrated into a bridge chip. PPUs602in a multi-PPU system may be identical to or different from one another. For example, different PPUs602might have different numbers of processing cores and/or different amounts of PP memory604. In implementations where multiple PPUs602are present, those PPUs may be operated in parallel to process data at a higher throughput than is possible with a single PPU602. Systems incorporating one or more PPUs602may be implemented in a variety of configurations and form factors, including, without limitation, desktops, laptops, handheld personal computers or other handheld devices, servers, workstations, game consoles, embedded systems, and the like.

In sum, the HMII includes a wearable head-mounted display unit supporting two compact high resolution screens for outputting a right eye and left eye image in support of the stereoscopic viewing, wireless communication circuits, three-dimensional positioning and motion sensors, a high-end processor capable of: independent software processing, processing streamed output from a remote server, a graphics processing unit capable of also functioning as a general parallel processing system, multiple imaging input, cameras positioned to track hand gestures, and high definition audio output. The HMII cameras are oriented to record the surrounding environment for integrated display by the HMII. One embodiment of the HMII would incorporate audio channels supporting a three-dimensional auditory environment. The HMII would further be capable of linking with a GPU server, subscription computational service, e.g. “cloud” servers, or other networked computational and/or graphics resources, and be capable of linking and streaming to a remote display such as a large screen monitor.

One advantage of the HMII disclosed herein is that a user has a functionally complete wireless interface to a computer system. Furthermore, the gesture recognition capability obviates the requirement for additional hardware components such as data gloves, wands, or keypads. This permits unrestricted movement by the user/wearer as well as a full featured portable computer or gaming platform. In some embodiments, the HMII can be configured to link with remote systems and thereby take advantage of scalable resources or resource sharing. In other embodiments the may be configured to enhance or augment the perception the wearer's local environment as an aid in training, information presentation, or situational awareness.