A system for utilizing augmented reality visualization in surgery, the system comprising: a plurality of fiducial markers comprising: a first subset of the plurality of fiducial markers configured to be placed in an operating room as fixed points whose position relative to a patient is known, and a second subset of the plurality of fiducial markers configured to be placed on a target area of a patient; an apparatus configured to generate a three dimensional reconstruction of an image stack representing the target area of the patient by performing image segmentation on all images of the image stack to identify structures within the image stack, and generating a three dimensional mesh defining boundaries of the identified structures, wherein the three dimensional reconstruction comprises the three dimensional mesh, wherein, when the fiducial markers have been placed, the apparatus is additionally configured to digitally place the corresponding locations of a digital representation of the plurality of fiducial markers into the three dimensional reconstruction of the image stack; a head-mounted display configured to superimpose a three dimensional projection of the three dimensional reconstruction onto a field of view of a user; and a camera in the head-mounted display configured to identify a location of each of the plurality of fiducial markers, and to match the plurality of fiducial markers with their digital representation, so that the apparatus is configured to maintain alignment between the projection and the user's actual view of the target area using the plurality of fiducial markers, wherein the camera is configured to detect a change in the location of the second subset of the plurality of fiducial markers indicating a change in location or shape of the target area, so that the apparatus is configured to compute a deformation of the three dimensional reconstruction, and the head-mounted display is configured to update the projection of the three dimensional reconstruction. a plurality of fiducial markers comprising: a first subset of the plurality of fiducial markers configured to be placed in an operating room as fixed points whose position relative to a patient is known, and a second subset of the plurality of fiducial markers configured to be placed on a target area of a patient; an apparatus configured to generate a three dimensional reconstruction of an image stack representing the target area of the patient by performing image segmentation on all images of the image stack to identify structures within the image stack, and generating a three dimensional mesh defining boundaries of the identified structures, wherein the three dimensional reconstruction comprises the three dimensional mesh, wherein, when the fiducial markers have been placed, the apparatus is additionally configured to digitally place the corresponding locations of a digital representation of the plurality of fiducial markers into the three dimensional reconstruction of the image stack; a head-mounted display configured to superimpose a three dimensional projection of the three dimensional reconstruction onto a field of view of a user; and a camera in the head-mounted display configured to identify a location of each of the plurality of fiducial markers, and to match the plurality of fiducial markers with their digital representation, so that the apparatus is configured to maintain alignment between the projection and the user's actual view of the target area using the plurality of fiducial markers, wherein the camera is configured to detect a change in the location of the second subset of the plurality of fiducial markers indicating a change in location or shape of the target area, so that the apparatus is configured to compute a deformation of the three dimensional reconstruction, and the head-mounted display is configured to update the projection of the three dimensional reconstruction. <EOS>