A computer-assisted medical system comprising: a user device (108, 130-1, 130-2) wearable by an operator, the user device (108, 130-1, 130-2) comprising: a display device configured to present imagery overlaid in an environment, the environment physically containing a manipulator assembly, and a sensor configured to detect one or more landmarks physically in the environment; and a controller configured to execute instructions to perform operations, the operations comprising: receiving, from the sensor, position or orientation information for the one or more landmarks physically in the environment, directing a manual movement of a portion of the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information; and directing a manual movement of a portion of the medical system distinct from the manipulator assembly relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information, or directing a manual movement, of a patient relative to the manipulator assembly by causing the display device to present second imagery overlaid in the environment based on the received position or orientation information. a user device (108, 130-1, 130-2) wearable by an operator, the user device (108, 130-1, 130-2) comprising: a display device configured to present imagery overlaid in an environment, the environment physically containing a manipulator assembly, and a sensor configured to detect one or more landmarks physically in the environment; and a display device configured to present imagery overlaid in an environment, the environment physically containing a manipulator assembly, and a sensor configured to detect one or more landmarks physically in the environment; and a controller configured to execute instructions to perform operations, the operations comprising: receiving, from the sensor, position or orientation information for the one or more landmarks physically in the environment, directing a manual movement of a portion of the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information; and directing a manual movement of a portion of the medical system distinct from the manipulator assembly relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information, or directing a manual movement, of a patient relative to the manipulator assembly by causing the display device to present second imagery overlaid in the environment based on the received position or orientation information. receiving, from the sensor, position or orientation information for the one or more landmarks physically in the environment, directing a manual movement of a portion of the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information; and directing a manual movement of a portion of the medical system distinct from the manipulator assembly relative to the manipulator assembly by causing the display device to present the imagery overlaid in the environment based on the received position or orientation information, or directing a manual movement, of a patient relative to the manipulator assembly by causing the display device to present second imagery overlaid in the environment based on the received position or orientation information.