A visual line estimating apparatus comprising: an image inputting section operable to take an image of a region including an eye on a face; a visual line measurement section operable to measure a direction of a visual line on the basis of the taken image; a visual line measuring result storing section operable to store therein visual line measuring results previously measured by the visual line measurement section; a representative value extracting section operable to extract a representative value from the visual line measuring results stored in the visual line measuring result storing section; and a visual line determining section operable to judge whether or not a difference between the representative value extracted by the representative value extracting section and the visual line measuring result measured by the visual line measurement section is lower than a predetermined threshold T to determine a visual line estimating result from the representative value and the visual line measuring result, the visual line estimating apparatus further comprising a threshold determining section operable to determine the threshold T, characterised in that that the threshold determining section is operable to determine the threshold T between a maximum eyeball rotation angle P and a minimum angle Q after setting the maximum eyeball rotation angle P indicating a possible travel angle of the visual line in a time period of a frame and setting the minimum angle Q indicating an angular difference D between the visual line at the time of defecting a true pupil and the visual line at the time of detecting a false pupil outside of a region of the true pupil. an image inputting section operable to take an image of a region including an eye on a face; a visual line measurement section operable to measure a direction of a visual line on the basis of the taken image; a visual line measuring result storing section operable to store therein visual line measuring results previously measured by the visual line measurement section; a representative value extracting section operable to extract a representative value from the visual line measuring results stored in the visual line measuring result storing section; and a visual line determining section operable to judge whether or not a difference between the representative value extracted by the representative value extracting section and the visual line measuring result measured by the visual line measurement section is lower than a predetermined threshold T to determine a visual line estimating result from the representative value and the visual line measuring result, the visual line estimating apparatus further comprising a threshold determining section operable to determine the threshold T, characterised in that that the threshold determining section is operable to determine the threshold T between a maximum eyeball rotation angle P and a minimum angle Q after setting the maximum eyeball rotation angle P indicating a possible travel angle of the visual line in a time period of a frame and setting the minimum angle Q indicating an angular difference D between the visual line at the time of defecting a true pupil and the visual line at the time of detecting a false pupil outside of a region of the true pupil.