A driver state estimation device for estimating a state of a driver driving a vehicle, comprising: an operation target value acquisition unit configured to acquire an operation target value of the driver; an actual movement amount acquisition unit configured to acquire an actual amount of movement of the vehicle; an actual operation amount acquisition unit configured to acquire an actual amount of operation of the driver; a driver model creation unit configured to create a driver model defining a relationship between a driver's input and a driver's output by using a difference between the operation target value and the actual amount of movement as the driver's input and the actual amount of operation as the driver's output; a driver model operation amount acquisition unit configured to acquire a driver model operation amount by inputting the difference between the operation target value and the actual amount of movement into the driver model; a residual calculation unit configured to calculate a reference model error which is obtained from a difference between the actual amount of operation and the driver model operation amount; an initial residual setup unit configured to set a residual to be a reference of the driver as an initial residual among residuals calculated by the residual calculation unit during an initialization period from timing of staring an operation of the vehicle as a starting point till a predetermined period of time elapses or from timing of a change in a traveling scene including a condition of a traveling road of the vehicle as a starting point till a predetermined period of time elapses; a driver state estimation unit configured to estimate a state of the driver based on the residual calculated by the residual calculation unit, wherein the driver state estimation unit estimates the state of the driver based on a result of comparing the initial residual set by the initial residual setup unit with the residual calculated by the residual calculation unit after the initialization period has elapsed, a residual storage unit configured to store the residual calculated by the residual calculation unit during the initialization period, wherein the initial residual setup unit sets the residual with a smallest value as the initial residual among the residuals stored in the residual storage unit, and a residual change rate computation unit configured to compute a change rate of the residual with respect to the initial residual, wherein the driver state estimation unit estimates the state of the driver based on the change rate of the residual with respect to the initial residual which is computed by the residual change rate computation unit. an operation target value acquisition unit configured to acquire an operation target value of the driver; an actual movement amount acquisition unit configured to acquire an actual amount of movement of the vehicle; an actual operation amount acquisition unit configured to acquire an actual amount of operation of the driver; a driver model creation unit configured to create a driver model defining a relationship between a driver's input and a driver's output by using a difference between the operation target value and the actual amount of movement as the driver's input and the actual amount of operation as the driver's output; a driver model operation amount acquisition unit configured to acquire a driver model operation amount by inputting the difference between the operation target value and the actual amount of movement into the driver model; a residual calculation unit configured to calculate a reference model error which is obtained from a difference between the actual amount of operation and the driver model operation amount; an initial residual setup unit configured to set a residual to be a reference of the driver as an initial residual among residuals calculated by the residual calculation unit during an initialization period from timing of staring an operation of the vehicle as a starting point till a predetermined period of time elapses or from timing of a change in a traveling scene including a condition of a traveling road of the vehicle as a starting point till a predetermined period of time elapses; a driver state estimation unit configured to estimate a state of the driver based on the residual calculated by the residual calculation unit, wherein the driver state estimation unit estimates the state of the driver based on a result of comparing the initial residual set by the initial residual setup unit with the residual calculated by the residual calculation unit after the initialization period has elapsed, a residual storage unit configured to store the residual calculated by the residual calculation unit during the initialization period, wherein the initial residual setup unit sets the residual with a smallest value as the initial residual among the residuals stored in the residual storage unit, and a residual change rate computation unit configured to compute a change rate of the residual with respect to the initial residual, wherein the driver state estimation unit estimates the state of the driver based on the change rate of the residual with respect to the initial residual which is computed by the residual change rate computation unit.