Patent ID: 8098263

Claim:
An image processing apparatus for compositing an image of a virtual object and an image of a physical space to generate a mixed reality image and causing a HMD to display the mixed reality image, comprising: a database which holds 3D CG data and scene graph data, wherein the 3D CG rendering data contains data representing geographical shape and color of the virtual object, texture data, and data representing the position and orientation of the virtual object, and wherein the scene graph data is data in which the 3D CG rendering data is hierarchically managed on the basis of the parent-child relationship between components of the virtual object; an image capturing unit which is attached to the HMD and captures the image of the physical space; a first measurement unit which measures a position and orientation of the HMD; an object manipulation unit which is used by a user wearing the HMD in order to operate a position and orientation of the virtual object; a second measurement unit which measures a position and orientation of said object manipulation unit; an operation panel which can be operated by the user, arranged at a position in the physical space within a viewing field of said image capturing unit attached to the HMD, displays an operation panel image which is used for editing the virtual object and contains a region for hierarchically displaying information about each of the components in accordance with the scene graph data held in the database, and is capable of receiving a user instruction of editing the virtual object input by the user; an operation panel image generation unit which generates the operation panel image by using the data held in said database, and outputs the generated operation panel image to said operation panel; a rendering unit which updates the data held in said database according to the user instruction received via said operation panel and the measurement result of said second measurement unit, and renders, by using the updated data, the image of the virtual object according to the measurement results of said first and second measurement units; a composition unit which composites the image of the virtual object rendered by said rendering unit and the image of the physical space captured by said image capturing unit to generate the mixed reality image; and a HMD, to which said image capturing unit is attached, which displays the mixed reality image generated by said composition unit.