Abstract:
Described are apparatus and methods for integrating gestural controllers in Head Mounted Displays, preferably for Virtual Reality and Augmented Reality applications. In a specific embodiment, multiple controllers are located on both sides of the HMD for ease of locating and wearing the controllers without having to remove the Head Mounted Display.

Description:
RELATED APPLICATIONS 
       [0001]    This application is a continuation-in-part of previously filed U.S. patent application Ser. No. 14/487,039 filed Sep. 15, 2014, which claims priority to U.S. Provisional Application No. 61/877,933 filed Sep. 13, 2013 and entitled “Methods and Apparatus for using the Human Body as an Input Device”, and which are explicitly incorporated herein by reference. 
     
    
     FIELD OF THE ART 
       [0002]    This disclosure relates to integrating gestural controllers into wearable head mounted displays (HMDs), preferably for Virtual Reality and Augmented Reality Applications. 
       BACKGROUND 
       [0003]    Many conventional Head Mounted Displays lack integrated gestural input systems and rely on existing conventional input mechanisms such as a Gamepad, Keyboard or Mouse. Such conventional input mechanisms are not intuitive since they are usually not visible in the Virtual Reality environments. 
       SUMMARY 
       [0004]    Described are apparatus and methods for integrating gestural controllers in Head Mounted Displays, preferably for Virtual Reality and Augmented Reality applications. 
         [0005]    In a specific embodiment, multiple controllers are located on one or more sides of the HMD for ease of locating and wearing the controllers without having to remove the Head Mounted Display. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]      FIGS. 1A-C  is an embodiment of a gestural controller according to an embodiment described herein 
           [0007]      FIG. 2  is a functional schematic of the gestural controller of  FIG. 1 . 
           [0008]      FIG. 3  is a perspective view of one embodiment of an HMD that has multiple gestural controllers of  FIG. 1  removably mounted thereon. 
           [0009]      FIG. 4  is a perspective view of another embodiment of an HMD that has multiple gestural controllers of  FIG. 1  removably mounted thereon. 
           [0010]      FIG. 5  is a top view of a gestural controller of  FIG. 1  removably mounted to a wrist strap. 
           [0011]      FIG. 6  is a view of a gestural controller of  FIG. 1  removably mounted to a finger strap. 
           [0012]      FIG. 7  is a top view of a gestural controller of  FIG. 1  inserted for charging and storage into a compartment of a computer or other electronic device. 
           [0013]      FIGS. 8A-B  are a perspective view of a game controller version of a gestural controller. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0014]    The embodiments described herein build on the concepts, hardware, software and functionality described in filed U.S. patent application Ser. No. 14/487,039 filed Sep. 14, 2014, which claims priority to U.S. Provisional Application No. 61/877,933 filed Sep. 13, 2013 and entitled “Methods and Apparatus for using the Human Body as an Input Device.” 
         [0015]    In one aspect, described herein is a gestural controller that can have more simplified functionality than the single sensor ring described in FIGS. 7 and 8 of U.S. patent application Ser. No. 14/487,039 mentioned above, though having the same or similar functionalities and structure, as well as different functionalities and structure consistent with the descriptions herein, are all within the intended scope. For purposes herein of distinguishing, the gestural controller described herein will be referred to as a single controller disk, though it is understood that various different embodiments of what is referred to herein as the single controller disk can exist, and the shapes can be many and are not limited to disks. 
         [0016]    As described further herein, a plurality of single controller disks can be combined with holders that allow easy placement of the single controller disk onto various parts of the body. Accordingly, the single controller disk can be strapped onto the aims or legs or attached to other electronics that are worn on the body (e.g: Virtual Reality Headsets, Helmets, watches etc). Single controller disks can be charged collectively, and also enabled to allow a plurality of them to be used to track different parts of the body, each single controller disk preferably] tracking the motion of the part of the body to which it is attached. Each single controller disk could, in a preferred embodiment, enter a lower power state if that part of the body isn&#39;t being actively tracked. 
         [0017]      FIGS. 1A-C  illustrate a single controller disk gestural controller  100  according to one embodiment. A single controller disk  100  is very small, typically having dimensions in the range of 25 mm*25 mm or less, and is a motion sensor unit that can track motion very accurately with MEMS sensor technology. A plurality of single controller disks have the ability to synchronize between themselves on when they can start tracking and when they go into a deep sleep state. In a preferred embodiment, one single controller disk can be used to track if there is any motion of the body and when there is motion, that single controller disk can wake up the other single controller disks to start tracking. 
         [0018]    As illustrated in  FIG. 1A , a single controller disk  100  in a preferred embodiment has elements A, B, X, Y and T 1  on a front face  110 , and elements T 2  and T 3  on a side face  120 . Elements A, B and X, Y are buttons that each provide a digital output (typically one bit corresponding to “pressed” or “not pressed”, but may also be multi-bit) whereas elements T 1 , T 2  and T 3  are touch sensitive tactile sensors, which allow for discrimination by touching, thereby creating a signal that can be used for control purposes. 
         [0019]    In a preferred embodiment, as shown in  FIG. 1A , the exposed front face surface of element B is flat, whereas in the embodiment of  FIG. 1B , the exposed front face surface of elements A, B, X and Y are each contoured, thereby providing for a tactile difference and assist a user in feeling the location of each merely by touch. In both these embodiments, T 1 , T 2  and T 3  are located on a side surface, as illustrated. 
         [0020]      FIG. 1C  illustrates a back-side  130  of a single controller disk  100 , which contains a band  140  that includes a metal insert  150  that allows for attachment to a ring, strap, or HMD device with a snap-fit, as described further herein. The metal insert also provides a ground reference for measuring the capacitive touch. 
         [0021]    Single controller disks  100  can also share information such as compass calibration and tracking information with each other to allow other single controller disks to converge faster in calibration since they have a baseline to start from. 
         [0022]      FIG. 2  is a functional schematic of the single controller disk gestural controller of  FIG. 1 . The internal components of the single controller disk  100 , to which each of the elements A, B, X, Y and T 1 , are further described, but generally include digital output buttons  210 —A, B, X, Y that each provide a digital output, touch sensitive tactile sensors  220 -T 1 , T 2  and T 3 , as well as one or more sensors. The sensors can be an accelerometer  230 , a gyroscope  240  and a magnetometer  250 . Optionally a compass  260  can be included as well. In a game controller version that is described herein, an analog controller switch/joystick device  270  that provides an analog input (that is then A/D converted) is also included. In a particular embodiment, a camera  280  provides a stream of image data that is used as described herein. Data from these various components are fed into a controller  290 , which operates on software as described herein, and has a processor, RAM and ROM memory, A/D and D/A conversion as necessary for the application, and other conventional software processing elements, such as a display and display interface elements for a version that contains that. In a preferred embodiment a microcontroller that has a processor, RAM and ROM all included within a single chip can be advantageous. 
         [0023]    Communication between the HMD and the disks is wireless, using Bluetooth or other conventional short-distance low-power means. 
         [0024]      FIG. 3  is a perspective view of one embodiment of an HMD  300  that has multiple single controller disk gestural controllers  100  of  FIG. 1  removably mounted thereon. As shown, the HMD  300  also contains a slot  310  for insertion of an electronic device such as a smartphone, which can include a viewable display. In this preferred embodiment, three different single controller disks  100  are removably inserted onto the HMD  300 , one on each side surface  300 L and  300 R, and one in the middle top surface  300 T. The HMD  300  also has an integrated projection delivery system  320  for illuminating the lens of the display of the HMD, as well as providing audio, in a manner that is conventional. The integrated projection delivery system  320  integrates with the various single controller disks  100  using control signals that are similar to those conventionally used when an HMD interfaces with a mouse and a keyboard, but instead receives the signals from single controller disks  100 , as described above. In a preferred embodiment, the single controller disk  100  attached to the middle top surface  300 T remains in place and is used to track head motion, whereas the single controller disks that are attached to the side surfaces  300 L and  300 R are removed, and each then attached to a finger of a different hand, as described further herein. As such, the three different single controller disks  100  are used to track the motion of the head, the left hand, and the right hand, which is particularly suitable for usage in a Virtual Reality and/or Augmented Reality application. 
         [0025]    It is understood that in other embodiments, the multiple single controller disk gestural controllers  100  need not be removably mounted to the HMD, but can be separate from physically, though when in use still in wireless communication with, the integrated projection delivery system  320  that is disposed on the HMD. 
         [0026]    Even more generally, data from N sensors that are placed at different positions and orientations relative to the centroid of the HMD is combined. Each sensor is some combination of accelerometers, gyroscopes and magnetometers. The positions and orientations are determined using a multi-objective cost function that weights overall accuracy, variance and robustness across multiple test conditions along with geometrical constraints, to name a few. The accelerometer, gyroscope and magnetometer readings are preferably averaged together to reduce the noise. In addition, Kalman filters are preferably used to accurately model the noise in these sensors to thus further reduce their noise. The accelerometers are used to emulate a gyro by re-constructing angular acceleration as well as angular velocity. The magnetometer array allows for a faster calibration time which translates to less heading drift and better performance in magnetically disturbed environments. 
         [0027]      FIG. 4  is a perspective view of another embodiment of an HMD  400  that has multiple single controller disk gestural controllers  100  of  FIG. 1  removably mounted thereon. This embodiment is similar to the embodiment described in  FIG. 3 , but instead of a slot to hold an electronic device that has a display therein, instead includes a grip  410 , which is preferably adjustable and has an outer rubber surface at least at points of contact with the electronic device, for firm holding thereof on the HMD  400 . 
         [0028]    In another embodiment shown in  FIG. 4 , the HMD contains a camera  440  outputting a video stream, which video stream is used for detecting the hand and extracting depth information to position the hand in front of the HMD. By processing this information from the camera  440 , the skeletal pose of one&#39;s own hand can be extracted, based upon the other teachings discussed herein. This skeletal pose can also be enhanced and tracked accurately by combining with the other multiple single controller disk gestural controllers  100  on the body. 
         [0029]    While the HMD embodiments are shown as a frame with goggles that cover both eyes, it is understood that the HMD as described herein can be adapted to be viewable by either or both eyes, and mounted accordingly on the frame of glasses or the like. 
         [0030]      FIG. 5  is a view of a single controller disk gestural controller of  FIG. 1  removably mounted to a wrist strap  500 . As shown, the wrist strap  500  includes a bendable and non-conductive strap portion  510 , with a metal band  520  disposed therein for providing a reference ground plane used for capacitive touch sensing. 
         [0031]      FIG. 6  is a view of a single controller disk gestural controller of  FIG. 1  removably mounted to a finger strap or metal ring. 
         [0032]      FIG. 7  is a top view of a single controller disk gestural controller of  FIG. 1  inserted for charging and storage into a compartment  710  of a computer  700  or other electronic device, in which charging occurs internally, with power being supplied from the computer or other electronic device. 
         [0033]      FIGS. 8A-B  are perspective views of a game controller version of a gestural controller  800 , with  FIG. 8A  illustrating the inner mechanical structure, and  FIG. 8B  showing the completed game controller version of a gestural controller. In addition to having A, B, X, Y inputs  810 , there is also a joystick  820 , as shown. This operates as a typical game controller, though is a miniature version, as it is wearable, as shown on a finger ring, using the hardware of, for example, the ring sensor described above and software is programmed to implement the conventional game controller inputs. 
         [0034]    Although the present inventions are described with respect to certain preferred embodiments, modifications thereto will be apparent to those skilled in the art.