A tool tracking system comprising: means for determining whether sensor data indicative of a tool state in a tool reference frame is available at a prior point in time, wherein the tool state includes a position and an orientation of a tool ; means for determining whether image data indicative of a tool state in a camera reference frame is available at the prior point in time; and characterized by : means for determining an error transformation between said tool states at the prior point in time by using both the sensor data and the image data when it is determined that both the sensor data and the image data are available at the same prior point in time; and when it is determined that the image data is unavailable at a current point in time, means for determining an updated tool state in the camera reference frame at the current point in time by determining an estimated tool state in the tool reference frame from the sensor data at the current point in time; and adjusting the estimated tool state in the tool reference frame with the error transformation to form the updated tool state in the camera reference frame at the current point in time. means for determining whether sensor data indicative of a tool state in a tool reference frame is available at a prior point in time, wherein the tool state includes a position and an orientation of a tool ; means for determining whether image data indicative of a tool state in a camera reference frame is available at the prior point in time; and characterized by : means for determining an error transformation between said tool states at the prior point in time by using both the sensor data and the image data when it is determined that both the sensor data and the image data are available at the same prior point in time; and when it is determined that the image data is unavailable at a current point in time, means for determining an updated tool state in the camera reference frame at the current point in time by determining an estimated tool state in the tool reference frame from the sensor data at the current point in time; and adjusting the estimated tool state in the tool reference frame with the error transformation to form the updated tool state in the camera reference frame at the current point in time. means for determining an error transformation between said tool states at the prior point in time by using both the sensor data and the image data when it is determined that both the sensor data and the image data are available at the same prior point in time; and when it is determined that the image data is unavailable at a current point in time, means for determining an updated tool state in the camera reference frame at the current point in time by determining an estimated tool state in the tool reference frame from the sensor data at the current point in time; and adjusting the estimated tool state in the tool reference frame with the error transformation to form the updated tool state in the camera reference frame at the current point in time. determining an estimated tool state in the tool reference frame from the sensor data at the current point in time; and adjusting the estimated tool state in the tool reference frame with the error transformation to form the updated tool state in the camera reference frame at the current point in time. <EOS>