An autonomous driving system configured to control an autonomous driving vehicle that is a target of remote assistance, the autonomous driving system comprising: one or more storage devices configured to store specific position information indicating specific positions at which there is a possibility that the remote assistance is required, the specific position being a position other than an operational design domain, which is an area in which the autonomous driving is possible, and including a construction section, a traffic congestion section and an accident occurrence position; and one or more processors configured to: determine (S 100) presence or absence of an abnormality in a remote assistance system configured to provide the remote assistance to the autonomous driving vehicle; wherein the autonomous driving system is characterized in that the one or more processors is configured to: set, when the abnormality is detected in the remote assistance system, any one of the specific positions on a target route from a current position of the autonomous driving vehicle to a destination as a limit position based on the specific position information; set a target retracting position such that the target retracting position is included in the target route from the current position to the limit position; and control the autonomous driving vehicle such that the autonomous driving vehicle stops at the target retracting position; wherein the one or more processors are further configured to: acquire a first specific position (PS 1) closest to the current position on the target route based on the specific position information; set, when the abnormality in the remote assistance system is a functional failure, the first specific position (PS 1) as the limit position; determine, when the abnormality in the remote assistance system is performance degradation, whether the first specific position satisfies a permitted condition, the permitted condition including at least one of following conditions (i) and (ii): (i) a distance between the current position and the first specific position is equal to or greater than a distance threshold value; (ii) a vehicle control amount that is required to stop the autonomous driving vehicle at a position before the first specific position, is equal to or lower than a control amount threshold value; acquire, when the first specific position does not satisfy the permitted condition, a second specific position different from the first specific position on the target route based on the specific position information; and set the second specific position as the limit position. one or more storage devices configured to store specific position information indicating specific positions at which there is a possibility that the remote assistance is required, the specific position being a position other than an operational design domain, which is an area in which the autonomous driving is possible, and including a construction section, a traffic congestion section and an accident occurrence position; and one or more processors configured to: determine (S 100) presence or absence of an abnormality in a remote assistance system configured to provide the remote assistance to the autonomous driving vehicle; set, when the abnormality is detected in the remote assistance system, any one of the specific positions on a target route from a current position of the autonomous driving vehicle to a destination as a limit position based on the specific position information; set a target retracting position such that the target retracting position is included in the target route from the current position to the limit position; and control the autonomous driving vehicle such that the autonomous driving vehicle stops at the target retracting position; acquire a first specific position (PS 1) closest to the current position on the target route based on the specific position information; set, when the abnormality in the remote assistance system is a functional failure, the first specific position (PS 1) as the limit position; determine, when the abnormality in the remote assistance system is performance degradation, whether the first specific position satisfies a permitted condition, the permitted condition including at least one of following conditions (i) and (ii): (i) a distance between the current position and the first specific position is equal to or greater than a distance threshold value; (ii) a vehicle control amount that is required to stop the autonomous driving vehicle at a position before the first specific position, is equal to or lower than a control amount threshold value; (i) a distance between the current position and the first specific position is equal to or greater than a distance threshold value; (ii) a vehicle control amount that is required to stop the autonomous driving vehicle at a position before the first specific position, is equal to or lower than a control amount threshold value; acquire, when the first specific position does not satisfy the permitted condition, a second specific position different from the first specific position on the target route based on the specific position information; and set the second specific position as the limit position.