Virtual reality / augmented reality handheld controller sensing

A method that includes employing several sensors associated with a handheld controller, where each of the sensors is made of one of a hover, touch, and force/pressure sensor, and generating, by one or more of the sensors, sensor data associated with the position of a user's hand and finger in relation to the handheld controller. The method continues with combining the sensor data from several sensors to form aggregate sensor data, sending the aggregate sensor data to a processor, and generating an estimated position of the user's hand and fingers based on the aggregate sensor data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. Provisional Patent Application 62/588,706, entitled “VIRTUAL REALITY/AUGMENTED REALITY HANDHELD CONTROLLER SENSING” and filed on Nov. 20, 2017, the entirety of which is incorporated by reference herein.

BACKGROUND

Field of the Disclosure

The present disclosure relates generally to virtual reality/augmented reality (VR/AR) systems, and more specifically to handheld controls for VR/AR systems.

Description of the Related Art

Handheld controllers are used in a variety of applications, including controlling media devices, remotely-controlled vehicles, and VR/AR systems. For example, in one application the handheld controller allows a user to move their hands and fingers to manipulate buttons and sensors to interact with objects within a VR/AR environment. To support flexible interaction with the VR/AR environment, various types of sensors can be employed within the handheld controller, including, but not limited to, contact switches, mechanical buttons, slider switches, and the like, which provides on/off and analog signals as inputs to the VR/AR system. However, existing handheld controller designs do not support sufficient immersion in the VR/AR environment, and therefore negatively impact the user experience.

DETAILED DESCRIPTION

FIGS. 1-6illustrate methods and apparatus for sensors placed on a handheld controller for use with VR/AR systems. The multiple sensors can employ several different modalities including capacitive sensing, inductive sensing, and haptic feedback to the user, and can be positioned on the handheld controller in geometric patterns such as circles or hexagons. In some embodiments, the system employing the handheld controller uses data from multiple sensors concurrently (also known as aggregate sensor data) to establish the location of a user's fingers and hands. This allows the system to use triangulation to determine the user's fingers position, and also improves the resolution and sensitivity of the sensors. In addition, inductive sensing can be used to improve the accuracy of capacitive sensing by detecting when a user's fingers are actually in contact with a sensor verses hovering over the sensor. In yet other embodiments, the palm of the user's hand provides a force to the handheld controller by squeezing, where the squeezing force can be detected by a sensor located in the grip of the handheld controller.

The handheld controller as described herein uses several types of sensors, including, but not limited to mechanical switch-type sensors, hover sensors, touch sensors, and force/pressure sensors. In at least some embodiments, the handheld controller uses a hover and touch sensor employing capacitive sensing. Capacitive sensing uses self-capacitive measurements and is employed as a proximity sensor and also as a contact sensor when the finger is near or in contact with the sensor. Another type of sensor is a force/pressure sensor that uses inductive sensing where a finger applies varying levels of pressure to the sensor. The pressure deflects elements within the sensor and provides a measure of variable force corresponding to the pressure applied. In some embodiments, sensors employing inductive sensing can also produce haptic feedback in response to pressure applied to the sensor. In the VR/AR environment, employing these hover, touch, and force sensors enhances the VR/AR experience by allowing for greater control of, and interaction with, objects within the VR/AR world. In one example, employing hover and force sensing allows the system to more precisely sense the position of the user's hand and fingers on the handheld controller. In another example, hover sensing and haptic feedback enables users to see in the VR/AR world the object they are about to touch before they actually make contact.

FIG. 1is a diagram illustrating a VR/AR system100that uses a plurality of sensors110located on the two handheld controllers132,134in accordance with some embodiments.FIG. 1illustrates the VR/AR system100for providing VR or AR content to a user in accordance with at least one embodiment of the present disclosure. The VR/AR system100includes a HMD device102and one or more external data sources104. The HMD device102includes a housing106to mount on a head108of a user. The housing106contains various electronic and optical components used to display visual content to the user, output audio content to the user, and track a pose (position and orientation) of the HMD device102, such as one or more near-eye displays112, an inertial measurement unit (IMU)114having one or more inertia/movement-based sensors, a processing subsystem (processor)116, and one or more image sensors118,120, as well as one or more audio speakers, lenses, or other optical elements, and the like (not shown).

As a general overview of the operation of the VR/AR system100, the HMD device102operates to display visual content via the one or more near-eye displays112and output audio content via one or more speakers (not shown). The visual and audio content are sourced from the one or more external data sources104, which may include, for example, a remote server, a local notebook computer or desktop computer, and the like. The visual and audio content are streamed to the processor116via any of a variety of wireless communications, such as one or more of the IEEE 802.11a/b/g/n/ac/ad specifications (also known as the WiFi specifications) to wirelessly connect to a corresponding wireless access point. As the video data are received via the WLAN link, the processor116executes software stored in one or more memories (not shown) to process the received video data to render sequences of image frames that are then displayed at the near-eye display112. Concurrently, the processor116executes software to continuously update the pose of the wireless handheld controllers132,134. The processor116may utilize imagery from one or more of the image sensors118,120, as well as depth information from one or more depth sensors (not shown), to determine spatial features in the environment of the HMD device102, and use various visual telemetry techniques to facilitate determination of the pose. The current pose of the wireless handheld controllers132,134typically is utilized by the processor116to control the perspective of a scene from which the sequences of images are rendered to provide an immersive VR/AR experience to the user.

In some embodiments, the VR/AR system100utilizes one or both wireless handheld controllers132,134to enable a user to provide gesture commands and other user inputs to control the operation of the VR/AR system100. As such, the handheld controllers132,134typically include an internal IMU (not shown) with one or more positional/inertial sensors to detect the user's manipulation of the handheld controllers132,134to detect such motion. The plurality of sensors110located on the handheld controllers132,134are operated by the user to control objects within the VR/AR environment. The plurality of sensors110can use, but are not limited to, capacitive sensing and inductive sensing, and also can provide haptic feedback to allow the user a detailed level of control over objects in the VR/AR world as described herein.

Capacitive sensing permits the user to position one or more fingers in contact with or near the sensor110, thus allowing the user to either position their finger a distance away from, or physically touch the sensor110. Stated another way, the finger can hover over the sensor110without touching the sensor110, or can touch the sensor110directly. Meanwhile, inductive sensing allows the user to apply varying levels of force or pressure by the user's fingers and hand to the sensor110, which then generates accurate force/pressure data to be sent to the processor116. In both capacitive and inductive sensing, the sensor110generates accurate position and force data due to the position and force/pressure of the user's fingers to the processor116.

Haptic feedback improves the user's experience within the VR/AR environment by generating a physical response to the user's hand using a handheld controller132. The physical response may be a clicking sensation, vibration, resistance to a button press, a bump when virtual objects are touched, and the like. Haptic feedback is generated by the handheld controller132when the user interacts with objects in the VR/AR environment. The user's control of objects, and their interactive VR/AR experience itself, is greatly enhanced by the user of capacitive sensing, inductive sensing, and haptic feedback.

FIG. 2is a diagram of a handheld controller200with a plurality of finger and palm sensors110in accordance with some embodiments. The handheld controller200is one embodiment of the handheld controller132,134ofFIG. 1, but other embodiments are possible. The handheld controller200is used to manipulate objects in the VR/AR system100ofFIG. 1by using a plurality of sensors110positioned on the handheld controller200.

The handheld controller200includes a hand grip202(grip), a touch pad204, a grip pad236, and a trigger240. The touch pad204includes the plurality of sensors110that sense the presence of the fingers of the user. In at least some embodiments, the sensors110are positioned in geometric patterns on the touch pad204as described herein. In at least one embodiment, the output of the sensors110are electrically combined together to form aggregate sensor data that is sent to the processor116ofFIG. 1for further analysis in determining which sensor110was activated, and to calculate the position and motion of the user's fingers. In a number of embodiments, the grip pad236includes a grip sensor238that detects the presence of a palm of the hand of the user, and, in some embodiments, can also detect the amount of force the hand is applying to the grip pad236. The grip sensor238can be a contact sensor, a capacitive sensor, or an inductive sensor with other embodiments possible. In at least some embodiments, the grip sensor238can also provide haptic feedback to the user's hand.

The touch pad204is positioned at a location on the handheld controller200to allow favorable finger control and easy user access to the sensors110as described herein. The touch pad204can include one region containing multiple sensors110, or the touch pad204can be partitioned into multiple regions, with each region having dedicated sensors110positioned within. In at least some embodiments, the sensors110are positioned on the touch pad204in a circular, triangular, or rectangular pattern, while in other embodiments, the pattern is composed of two regions using concentric hexagonal shapes. Other patterns are possible and are not limiting. The sensors110are arranged in known patterns to allow the processor116to accurately triangulate the user's finger position. In the present embodiment shown inFIG. 2, one such pattern uses two concentric outer and inner hexagons206,208respectfully, with6sensors deployed across each hexagon206,208and1sensor in the center of the hexagons206,208.

In the embodiment shown inFIG. 2, the touch pad204includes13sensors110, with sensor A210positioned at the center of the touch pad204, sensors B, C, D, E, F, and G,212,214,216,218,220, and222, respectfully, positioned along the inner hexagon208, and sensors H, I, J, K, L, and M,224,226,228,230,232, and234, respectfully, positioned along the outer hexagon206. Each sensor110can be of the capacitive or inductive type, a mechanical switch, or another type of sensor or switching mechanism. Other embodiments are possible, with the present embodiment just one example.

In at least some embodiments, the output from the sensors110can be combined to form aggregate sensor data which is sent to the processor116for further calculation by an algorithm. The algorithm, as described in more detail inFIG. 5, uses the aggregate sensor data to determine the X and Y coordinates of the user's finger or fingers over the sensors110. Combining the output of multiple sensors together enables the processor116to form an accurate estimate of the position of the hand and fingers around the sensors110in 3-D space. In some embodiments, output from the force/pressure sensors110are used to improve the accuracy of the capacitive sensors110by detecting when the user's finger is in contact with the capacitive sensor110or hovering just over it. Also in some embodiments, measurements from two or more separate regions can be analyzed to determine finger location and orientation.

In at least one embodiment, the sensors110on the touch pad204include hover, touch, and force sensors110, while the grip sensor238includes a force sensor110, and the trigger240incorporates the force sensor110. The sensors110can be employed as a hover sensor by combining the measurements from the outer and inner hexagons206,208respectfully and using the measurements to triangulate the location of the finger of the user. Other embodiments are possible, including any combination of hover, touch, and force sensors arranged individually or in patterns or regions on the touch pad204or grip202of the handheld controller200.

FIG. 3is a diagram of a capacitive sensor300to detect the presence of a finger320of the user when using the handheld controller200ofFIG. 2in accordance with some embodiments. The capacitive sensor300includes a printed circuit board structure made of a laminate layer302, an overlay layer304, and a ground plane306, with the ground plane306sandwiched between the laminate layer302and the overlay layer304. The ground plane306is electrically connected to earth ground. One or more base capacitors308are present within the ground plane306. The base capacitor308emits an electromagnetic (EM) field312that penetrates the overlay layer304and extends a distance beyond a surface of the overlay layer304. The finger320has an inherent touch capacitance322that is connected to a virtual ground324elsewhere in the body of the user. The virtual ground324is also electrically connected to earth ground. When the finger320is near the EM field312, it begins to influence the EM field312due to the touch capacitance322. As the finger320moves towards the base capacitor308, it introduces changes to the EM field312magnitude, and continued movement towards the base capacitor308has a stronger effect on the EM field312. The changes in the EM field312can be measured as changes in the current and voltage values sensed at the base capacitor308, with the current and voltage values sent to the processor116ofFIG. 1. In this manner, the presence and relative position of the finger320can be detected by the processor116or a processor located in the handheld controller200. The capacitive sensor modality is used in both the hover and touch sensors110.

In addition to measuring the changes to the base capacitor's308current and voltage values, in some embodiments the processor116tracks the current and voltage of the capacitive sensor300to establish a baseline voltage and current value. This occurs when the capacitive sensor300does not detect the presence of the finger320. This process is called a “baseline tracking procedure”. The baseline tracking procedure is used to monitor the capacitive sensor300for changes (drift) in ambient current and voltage values, and can thus detect variances to the baseline values. The baseline tracking procedure can also compensate for minor shifts in the baseline capacitance of the base capacitor308. Also, the processor116can perform a recalibration function to reset the baseline current and voltage values of the capacitive sensor300when the capacitive sensor300does not detect the presence of the finger320. These recalibration procedures can be performed periodically as defined by the processor116. When the finger320is near the capacitive sensor300, both the baseline tracking procedure and the periodic recalibration functions are disabled.

FIG. 4is a diagram of an inductive sensor400to detect the presence of the finger320of the user when using the handheld controller200ofFIG. 2in accordance with some embodiments. The inductive sensor400is located on a printed circuit board and includes a metal plane402parallel to, and physically separated from, a coil404a small distance, where the coil404is an inductor with an inductance value. When an external force406is applied to the metal plane402by an object, such as when the finger320presses on the inductive sensor400, the metal plane402undergoes a deflection408in relation to the coil404. Due to the proximity of the metal plane402to the coil404, the deflection408causes a change in the inductance of the coil404. Based on Lenz's Law, the closer the metal plane402is to the coil404, the lesser the inductance of the coil404. The change in inductance value is detected by the processor116in the form of changes to the current and voltage through the coil404. In this manner, the presence and force applied by the finger320can be detected by the processor116or a processor located in the handheld controller200. The inductive sensor400thus is used to measure the variable force or pressure applied by the finger320. The force/pressure sensor110employs an inductive sensor as described herein.

In some embodiments, the inductive sensor400also incorporates hysteresis functions as well as haptic feedback to the finger320of the user. Hysteresis, as applied to the present disclosure, introduces a lag or delay in response to a button or sensor being activated. Hysteresis prevents the unintentional rapid cycling or bouncing of a sensor in response to input such as the finger320. Hysteresis aids the user's experience in the VR/AR environment by smoothing out the response of the sensors110. Also, haptic feedback can recreate the sense of touch by applying forces, vibrations, or motion back to the user. This mechanical stimulation is used to aid in the creation and realism of virtual objects in a VR/AR environment.

FIG. 5is a flow diagram of a method500for calculating the position of the finger320ofFIG. 3of the user based on multiple sensors110as employed by the VR/AR system100ofFIG. 1in accordance with some embodiments. In the present embodiment, the method500uses capacitive sensors300, but other embodiments employing other types of sensors are possible. The method500is but one embodiment of an algorithm for calculating the position of the finger320in relation to the handheld controller200ofFIG. 2, and other embodiments are possible. In the present embodiment, the handheld controller200is used with the touch pad204having thirteen sensors110arranged in a concentric hexagonal pattern, and forming two distinct regions. The method500begins at block502where the VR/AR system100obtains measurements from the thirteen sensors via a wired or wireless communications path from the handheld controller200. Next, at block504, the method500determines the median, or 7thlargest, sensor measurement value based on the thirteen measurements. At block506, the VR/AR system100subtracts the median sensor measurement value from each of the set of thirteen measurements. Next, at block508, the 6 largest measurement values are used to perform a centroiding procedure to obtain the X- and Y-positions of the finger320. The centroiding procedure includes calculating the geometric center of all of the measurements, and is represented by calculating the arithmetic mean of the 6 measurements. At block510, the results of the centroiding procedure are passed through a filtering process via a Kalman filter to improve the resolution of the data and to minimize errors. Next, at block512, the results from the centroiding and filtering procedures are presented as X_Position and Y_Position values. The X_Position and Y_Position values represent to location of the finger in relation to the thirteen sensors110. Additionally, in some embodiments, movement data of the finger320over time can be calculated by the processor116by calculating the X_Velocity and Y_Velocity values for the finger320based on calculating and recording finger positions over time.

FIG. 6is a flow diagram of a method600for employing a plurality of sensors110located on the handheld controller200ofFIG. 2to sense the presence of one or more fingers of the user in the VR/AR system100ofFIG. 1in accordance with some embodiments. The method600includes a user's hand and finger movement using the handheld controller200at block602, where the handheld controller200employs sensors110arranged on the touch pad204in a geometric pattern as disclosed herein. Next, at decision block604, the processor116determines whether any of the hover, touch, or force sensors detects any movement of the fingers320. If one or more of the sensors110detects movement, the method600continues with collecting sensor measurement data at block606. If, however, no finger320movement is detected by the sensors110, the processor116begins the baseline tracking procedure at block612and the sensor recalibration function at block614. Afterwards, the method600returns to the decision block604to await the detection of finger movement by the sensors110.

Assuming the finger movement is detected at decision block604, the method600continues with the sensor data from the hover, touch, and force sensors110combined to form aggregate sensor data at block606. The data are sent to the processor116. At block608, the method600continues with the processor116applying an algorithm to the aggregate sensor data. The algorithm can be the method500ofFIG. 5or another process for analyzing the sensor measurement data. Next, at block610, the method600continues with the processor116calculating the estimated position of the finger320of the user using the X_Position and Y_Position variables that are derived from the aggregate sensor data. Additionally, the method600can also calculate the motion of the finger320by using current and historical data from previous measurements to determine the X_Velocity and Y_Velocity of the fingers320over time as described herein.