A system including a head-mounted display, comprising: a display device; an outward-facing depth camera; and a storage machine comprising instructions executable by a logic machine to: present one or more virtual objects on the display device so that the positioning of virtual objects is within an environment; receive information from the depth camera about the environment; determine a position of the head-mounted display within the environment; determine a position of the user's hand based at least on pixels of a depth map corresponding to the user as determined from the information received from the depth camera; infer a position within the environment of a joint of a user's arm that is not visible to the depth camera, wherein the joint of the user's arm is the user's elbow or shoulder, wherein the inferred position is based at least on the determined position of the head-mounted display and a prediction of the position of the user's shoulder and/or elbow based on the known position of the head-mounted display, ergonomics based metrics, calibration parameters as obtained from the user and the determined position of the user's hand; cast a ray from a portion of the user's hand based on the position of the joint of the user's arm and the position of the user's hand; and responsive to the ray intersecting with one or more control points of a virtual object, indicate to the user that the virtual object is being targeted. a display device; an outward-facing depth camera; and a storage machine comprising instructions executable by a logic machine to: present one or more virtual objects on the display device so that the positioning of virtual objects is within an environment; receive information from the depth camera about the environment; determine a position of the head-mounted display within the environment; determine a position of the user's hand based at least on pixels of a depth map corresponding to the user as determined from the information received from the depth camera; infer a position within the environment of a joint of a user's arm that is not visible to the depth camera, wherein the joint of the user's arm is the user's elbow or shoulder, wherein the inferred position is based at least on the determined position of the head-mounted display and a prediction of the position of the user's shoulder and/or elbow based on the known position of the head-mounted display, ergonomics based metrics, calibration parameters as obtained from the user and the determined position of the user's hand; cast a ray from a portion of the user's hand based on the position of the joint of the user's arm and the position of the user's hand; and responsive to the ray intersecting with one or more control points of a virtual object, indicate to the user that the virtual object is being targeted. present one or more virtual objects on the display device so that the positioning of virtual objects is within an environment; receive information from the depth camera about the environment; determine a position of the head-mounted display within the environment; determine a position of the user's hand based at least on pixels of a depth map corresponding to the user as determined from the information received from the depth camera; infer a position within the environment of a joint of a user's arm that is not visible to the depth camera, wherein the joint of the user's arm is the user's elbow or shoulder, wherein the inferred position is based at least on the determined position of the head-mounted display and a prediction of the position of the user's shoulder and/or elbow based on the known position of the head-mounted display, ergonomics based metrics, calibration parameters as obtained from the user and the determined position of the user's hand; cast a ray from a portion of the user's hand based on the position of the joint of the user's arm and the position of the user's hand; and responsive to the ray intersecting with one or more control points of a virtual object, indicate to the user that the virtual object is being targeted.