A tool tracking system comprising: a first sensor; a second sensor; a camera; and a processor configured to: determine a sensor estimated tool state of a tool relative to a landmark for a point in time by using data that is received from the first sensor and that is indicative of the tool state at the point in time; determine a sensor estimated camera state of a camera relative to the landmark for the point in time by using data that is received from the second sensor and that is indicative of the camera state at the point in time; determine an image estimated tool state of the tool relative to the camera for the point in time using image data generated by the camera and indicative of the tool state at the point in time; and characterized by the processor further configured to: translate the sensor estimated tool state so as to be relative to the camera instead of the landmark; update an error transform that is a transform between the translated sensor estimated tool state and the image estimated tool state; determine a sensor estimated tool state of the tool relative to the landmark for a subsequent point in time by using data that is received from the first sensor and that is indicative of the tool state at the subsequent point in time; translate the sensor estimated tool state of the tool relative to the landmark for the subsequent point in time so as to be relative to the camera instead of the landmark to generate a translated sensor estimated tool state of the tool relative to the camera for the subsequent point in time; and determine an estimated tool state relative to the camera for the subsequent point in time by applying the updated error transform to the translated sensor estimated tool state of the tool relative to the camera for the subsequent point in time. a first sensor; a second sensor; a camera; and a processor configured to: determine a sensor estimated tool state of a tool relative to a landmark for a point in time by using data that is received from the first sensor and that is indicative of the tool state at the point in time; determine a sensor estimated camera state of a camera relative to the landmark for the point in time by using data that is received from the second sensor and that is indicative of the camera state at the point in time; determine an image estimated tool state of the tool relative to the camera for the point in time using image data generated by the camera and indicative of the tool state at the point in time; determine a sensor estimated tool state of a tool relative to a landmark for a point in time by using data that is received from the first sensor and that is indicative of the tool state at the point in time; determine a sensor estimated camera state of a camera relative to the landmark for the point in time by using data that is received from the second sensor and that is indicative of the camera state at the point in time; determine an image estimated tool state of the tool relative to the camera for the point in time using image data generated by the camera and indicative of the tool state at the point in time; and characterized by the processor further configured to: translate the sensor estimated tool state so as to be relative to the camera instead of the landmark; update an error transform that is a transform between the translated sensor estimated tool state and the image estimated tool state; determine a sensor estimated tool state of the tool relative to the landmark for a subsequent point in time by using data that is received from the first sensor and that is indicative of the tool state at the subsequent point in time; translate the sensor estimated tool state of the tool relative to the landmark for the subsequent point in time so as to be relative to the camera instead of the landmark to generate a translated sensor estimated tool state of the tool relative to the camera for the subsequent point in time; and determine an estimated tool state relative to the camera for the subsequent point in time by applying the updated error transform to the translated sensor estimated tool state of the tool relative to the camera for the subsequent point in time. translate the sensor estimated tool state so as to be relative to the camera instead of the landmark; update an error transform that is a transform between the translated sensor estimated tool state and the image estimated tool state; determine a sensor estimated tool state of the tool relative to the landmark for a subsequent point in time by using data that is received from the first sensor and that is indicative of the tool state at the subsequent point in time; translate the sensor estimated tool state of the tool relative to the landmark for the subsequent point in time so as to be relative to the camera instead of the landmark to generate a translated sensor estimated tool state of the tool relative to the camera for the subsequent point in time; and determine an estimated tool state relative to the camera for the subsequent point in time by applying the updated error transform to the translated sensor estimated tool state of the tool relative to the camera for the subsequent point in time.