Patent ID: 8384665

Claim:
A system for a user to select a portion of a 3D virtual environment being displayed, the system comprising: a controller including a plurality of inertia sensors providing sensor signals sufficient to derive changes in motion of the controller in six degrees of freedom; a processing unit, receiving the sensor signals, configured to derive position and orientation of the controller from the sensor signals, and generate a ray originating from the controller to an interaction of a screen provided to display the 3D virtual environment, wherein the ray is further projected into the 3D virtual environment by a ray tracing technique; and a video capturing system including at least one camera, disposed in front or in vicinity of the user to capture the motion of the controller being held by the user, wherein the processing unit receives images from the video capturing system, the controller is used to calibrate a relationship with the screen used to display the 3D virtual environment to estimate size and location of the screen with respect to the camera so as to find absolute position and orientation of the controller in a physical world.