A wearable system comprising: a head-mounted display; a handheld user input device comprising an inertial measurement unit IMU; an electromagnetic EM tracking system comprising: an EM emitter (402, 602) disposed in or on the handheld user input device, the EM emitter configured to generate an EM field; and an EM sensor (404, 604) disposed in or on the head-mounted display, the EM sensor configured to sense the EM field, wherein the EM tracking system is configured to output an EM field matrix associated with an estimated pose of the EM sensor relative to the EM emitter; and a hardware processor programmed to iteratively, in real time during operation of the handheld user input device: access IMU data from the IMU, the IMU data representative of an estimated pose of the handheld user input device in a reference frame associated with the handheld user input device; access the EM field matrix from the EM tracking system; calculate a predicted EM field matrix representative of a predicted pose of the handheld user input device in a world reference frame associated with an environment of the wearable system; generate an error state based at least partly on the EM field matrix and the predicted EM field matrix, the error state representing at least one of: bias or noise in the IMU, or an offset between the reference frame of the handheld user input device and the world reference frame; apply a Kalman filter to the IMU data based on the error state; and determine, using the Kalman filter, a pose of the handheld user input device in the world reference frame. a head-mounted display; a handheld user input device comprising an inertial measurement unit IMU; an electromagnetic EM tracking system comprising: an EM emitter (402, 602) disposed in or on the handheld user input device, the EM emitter configured to generate an EM field; and an EM sensor (404, 604) disposed in or on the head-mounted display, the EM sensor configured to sense the EM field, wherein the EM tracking system is configured to output an EM field matrix associated with an estimated pose of the EM sensor relative to the EM emitter; and an EM emitter (402, 602) disposed in or on the handheld user input device, the EM emitter configured to generate an EM field; and an EM sensor (404, 604) disposed in or on the head-mounted display, the EM sensor configured to sense the EM field, wherein the EM tracking system is configured to output an EM field matrix associated with an estimated pose of the EM sensor relative to the EM emitter; and a hardware processor programmed to iteratively, in real time during operation of the handheld user input device: access IMU data from the IMU, the IMU data representative of an estimated pose of the handheld user input device in a reference frame associated with the handheld user input device; access the EM field matrix from the EM tracking system; calculate a predicted EM field matrix representative of a predicted pose of the handheld user input device in a world reference frame associated with an environment of the wearable system; generate an error state based at least partly on the EM field matrix and the predicted EM field matrix, the error state representing at least one of: bias or noise in the IMU, or an offset between the reference frame of the handheld user input device and the world reference frame; apply a Kalman filter to the IMU data based on the error state; and determine, using the Kalman filter, a pose of the handheld user input device in the world reference frame. access IMU data from the IMU, the IMU data representative of an estimated pose of the handheld user input device in a reference frame associated with the handheld user input device; access the EM field matrix from the EM tracking system; calculate a predicted EM field matrix representative of a predicted pose of the handheld user input device in a world reference frame associated with an environment of the wearable system; generate an error state based at least partly on the EM field matrix and the predicted EM field matrix, the error state representing at least one of: bias or noise in the IMU, or an offset between the reference frame of the handheld user input device and the world reference frame; apply a Kalman filter to the IMU data based on the error state; and determine, using the Kalman filter, a pose of the handheld user input device in the world reference frame.