A computer-implemented process for constructing shared virtual environments, comprising: applying a computer to perform process actions for: generating a real-time environmental model of a real-world environment in which a shared immersive virtual environment is being presented to two or more users, wherein any view of the real-world environment is completely replaced with the shared immersive virtual environment; the real-time environmental model classifying geometry, positions, and motions of surfaces and objects in the real-world environment to represent a semantic understanding of the surfaces and the objects within the real-world environment; generating a unified real-time tracking model for each of the two or more users, wherein data from sensors placed at different locations in the real-world environment are combined to build each unified real-time tracking model; for each particular user, generating frames of the shared immersive virtual environment corresponding to a real-time field of view of the particular user; jointly constraining each frame of the shared immersive virtual environment for each particular user relative to both the real-time environmental model and each of the unified real-time tracking models; mapping and rendering one or more virtual elements of the shared immersive virtual environment to real objects in the real-world environment that can be touched and manipulated by two or more of the users to provide shared tactile interaction; and rendering one or more indicia of safety into the shared immersive virtual environment, wherein one or more of the indicia of safety are represented by a translucent rendering of a non-user person within the real-world environment that increases in solidity as any user approaches that translucent rendering. generating a real-time environmental model of a real-world environment in which a shared immersive virtual environment is being presented to two or more users, wherein any view of the real-world environment is completely replaced with the shared immersive virtual environment; the real-time environmental model classifying geometry, positions, and motions of surfaces and objects in the real-world environment to represent a semantic understanding of the surfaces and the objects within the real-world environment; generating a unified real-time tracking model for each of the two or more users, wherein data from sensors placed at different locations in the real-world environment are combined to build each unified real-time tracking model; for each particular user, generating frames of the shared immersive virtual environment corresponding to a real-time field of view of the particular user; jointly constraining each frame of the shared immersive virtual environment for each particular user relative to both the real-time environmental model and each of the unified real-time tracking models; mapping and rendering one or more virtual elements of the shared immersive virtual environment to real objects in the real-world environment that can be touched and manipulated by two or more of the users to provide shared tactile interaction; and rendering one or more indicia of safety into the shared immersive virtual environment, wherein one or more of the indicia of safety are represented by a translucent rendering of a non-user person within the real-world environment that increases in solidity as any user approaches that translucent rendering. <EOS>