A method (600, 800, 1000, 1200) of performing localization of a handheld device with respect to a wearable device, the method comprising: obtaining, by an inertial measurement unit, IMU mounted to the handheld device, handheld data (632, 1032, 1232) indicative of movement of the handheld device, the handheld data (632, 1032, 1232) including linear accelerations and rotational velocities; obtaining, by an imaging device (310, 326) mounted to a first device, fiducial data (630, 1030, 1230) indicative of movement of the handheld device, wherein the first device is either the handheld device or the wearable device, and wherein obtaining the fiducial data (630, 1030, 1230) includes: capturing, by the imaging device (310, 326), a fiducial image containing a number of fiducials (306, 322) from at least three fiducials (306, 322) affixed to a second device different than the first device, wherein the second device is either the handheld device or the wearable device, the fiducial data (630, 1030, 1230) indicative of movement of the handheld device being determined based on the respective locations of the fiducials within the fiducial image and based on known physical relationships between the fiducials; determining the number of fiducials contained in the fiducial image; wherein in response to determining that the number of fiducials contained in the fiducial image is equal to or greater than three, updating a position and an orientation of the handheld device using solely the fiducial data (630, 1030, 1230); and in response to determining that the number of fiducials contained in the fiducial image is equal to one or two, updating the position and the orientation of the handheld device using both the fiducial data (630, 1030, 1230) and the handheld data (632, 1032, 1232). obtaining, by an inertial measurement unit, IMU mounted to the handheld device, handheld data (632, 1032, 1232) indicative of movement of the handheld device, the handheld data (632, 1032, 1232) including linear accelerations and rotational velocities; obtaining, by an imaging device (310, 326) mounted to a first device, fiducial data (630, 1030, 1230) indicative of movement of the handheld device, wherein the first device is either the handheld device or the wearable device, and wherein obtaining the fiducial data (630, 1030, 1230) includes: capturing, by the imaging device (310, 326), a fiducial image containing a number of fiducials (306, 322) from at least three fiducials (306, 322) affixed to a second device different than the first device, wherein the second device is either the handheld device or the wearable device, the fiducial data (630, 1030, 1230) indicative of movement of the handheld device being determined based on the respective locations of the fiducials within the fiducial image and based on known physical relationships between the fiducials; determining the number of fiducials contained in the fiducial image; wherein in response to determining that the number of fiducials contained in the fiducial image is equal to or greater than three, updating a position and an orientation of the handheld device using solely the fiducial data (630, 1030, 1230); and in response to determining that the number of fiducials contained in the fiducial image is equal to one or two, updating the position and the orientation of the handheld device using both the fiducial data (630, 1030, 1230) and the handheld data (632, 1032, 1232).