Patent ID: 7843470

Claim:
A system comprising: a first position/orientation acquisition unit configured to acquire a first position/orientation of a first viewpoint of a first observer in a world coordinate system; a first manipulation unit used by the first observer to manipulate a virtual object; a second manipulation unit used by a second observer to manipulate the virtual object; a first manipulation position acquisition unit configured to acquire a first position of said first manipulation unit in the world coordinate system; a second manipulation position acquisition unit configured to acquire a second position of said second manipulation unit, wherein the second position is a relative position of said second manipulation unit from a second viewpoint position of the second observer; an image generation unit configured (i) to transform the acquired second position of said second manipulation unit into a world coordinate position of said second manipulation unit using the first position/orientation of the first viewpoint of the first observer, so that a positional relationship between the world coordinate position of the second manipulation unit and the first viewpoint is equal to that between the acquired second position of the second manipulation unit and the second viewpoint position and (ii) to generate a virtual object image of the virtual object viewed from the first viewpoint based on the first position/orientation and object information managed by a management unit, and to generate a virtual pointer image of said second manipulation unit to be viewed from the first viewpoint based on the first position/orientation and the world coordinate position of said second manipulation unit; said management unit configured to manage object information of the virtual object on the basis of the first position, the world coordinate position of said second manipulation unit, and operation contents of said first and second manipulation units; an image acquisition unit configured to acquire an image of a physical space viewed from the first viewpoint; an image compositing unit configured (i) to combine the virtual object image, the virtual pointer image, and the image of the physical space into a combined image and (ii) to output the combined image to a first head mounted display worn by the first observer and a second head mounted display worn by the second observer, wherein manipulation results by said first manipulation unit and said second manipulation unit are reflected in the image of the virtual object.