An apparatus comprising: a head mounted display configured to be worn by a user; a head position tracker configured to track a position of the user's head, and generate a corresponding head position signal, wherein the position of the user's head comprises head location and head orientation; a camera position controller configured to detect a change in the position of the user's head from a current head position to a new head position and, based at least in part on detecting the change, to communicate a command to a movable support for a camera to a next camera position, the next camera position being aligned with the new head position; a data storage configured to store a surface map, the surface map including a population of past views of respective portions of an environment and, for each past view, information identifying its viewing position; an image assembly module, coupled to the data storage, and configured to determine, based on the next camera position, a next camera viewing region, the next camera viewing region being a region of an environment that will be in a camera field of the camera when in the next camera position, receive a camera image frame from the camera, the camera image frame including an indication of camera position, determine, based at least in part on the indicated camera position, whether the image frame covers all of the next camera viewing region, and upon determining the camera image frame does not cover all of the next camera viewing region, generate an assembled image frame that encompasses the next camera viewing region, the assembled image frame including image data from at least one of the past views; and a rendering module configured to render a 3D image from the assembled image frame, the 3D image appearing as if viewed from the next camera position. a head mounted display configured to be worn by a user; a head position tracker configured to track a position of the user's head, and generate a corresponding head position signal, wherein the position of the user's head comprises head location and head orientation; a camera position controller configured to detect a change in the position of the user's head from a current head position to a new head position and, based at least in part on detecting the change, to communicate a command to a movable support for a camera to a next camera position, the next camera position being aligned with the new head position; a data storage configured to store a surface map, the surface map including a population of past views of respective portions of an environment and, for each past view, information identifying its viewing position; an image assembly module, coupled to the data storage, and configured to determine, based on the next camera position, a next camera viewing region, the next camera viewing region being a region of an environment that will be in a camera field of the camera when in the next camera position, receive a camera image frame from the camera, the camera image frame including an indication of camera position, determine, based at least in part on the indicated camera position, whether the image frame covers all of the next camera viewing region, and upon determining the camera image frame does not cover all of the next camera viewing region, generate an assembled image frame that encompasses the next camera viewing region, the assembled image frame including image data from at least one of the past views; and determine, based on the next camera position, a next camera viewing region, the next camera viewing region being a region of an environment that will be in a camera field of the camera when in the next camera position, receive a camera image frame from the camera, the camera image frame including an indication of camera position, determine, based at least in part on the indicated camera position, whether the image frame covers all of the next camera viewing region, and upon determining the camera image frame does not cover all of the next camera viewing region, generate an assembled image frame that encompasses the next camera viewing region, the assembled image frame including image data from at least one of the past views; and a rendering module configured to render a 3D image from the assembled image frame, the 3D image appearing as if viewed from the next camera position.