An apparatus comprising: a sensor (302, 304) configured to identify information about a first vehicle in a driving environment of a second vehicle having an autonomous driving mode; a computer-readable memory that stores: detailed map information for the driving environment, the detailed map information comprising information about a road on which the first vehicle travels; and first state information for the first vehicle, the first state information identifying at least one of position, speed, or direction of travel for the first vehicle; and one or more processors in communication with the computer-readable memory and the sensor, the one or more processors being configured to: use the detailed map information to control the second vehicle in the autonomous driving mode; receive sensor information from the sensor, the sensor information including information about the first vehicle in the driving environment for a predetermined period of time; determine the first state information based on the received sensor information; determine a first trajectory based on the first state information, the first trajectory corresponding to an actual path taken by the first vehicle for the predetermined period of time; determine an expected trajectory for a vehicle in the driving environment of the second vehicle based on the detailed map information; determine that the driving environment has changed from the detailed map information to the extent that the detailed map information is no longer accurate by comparing the determined expected trajectory with the determined first trajectory; and when it is determined that the driving environment has changed from the detailed map information to the extent that the detailed map information is no longer accurate, stop using the detailed map information to control the second vehicle in the autonomous driving mode. a sensor (302, 304) configured to identify information about a first vehicle in a driving environment of a second vehicle having an autonomous driving mode; a computer-readable memory that stores: detailed map information for the driving environment, the detailed map information comprising information about a road on which the first vehicle travels; and first state information for the first vehicle, the first state information identifying at least one of position, speed, or direction of travel for the first vehicle; and detailed map information for the driving environment, the detailed map information comprising information about a road on which the first vehicle travels; and first state information for the first vehicle, the first state information identifying at least one of position, speed, or direction of travel for the first vehicle; and one or more processors in communication with the computer-readable memory and the sensor, the one or more processors being configured to: use the detailed map information to control the second vehicle in the autonomous driving mode; receive sensor information from the sensor, the sensor information including information about the first vehicle in the driving environment for a predetermined period of time; determine the first state information based on the received sensor information; determine a first trajectory based on the first state information, the first trajectory corresponding to an actual path taken by the first vehicle for the predetermined period of time; determine an expected trajectory for a vehicle in the driving environment of the second vehicle based on the detailed map information; determine that the driving environment has changed from the detailed map information to the extent that the detailed map information is no longer accurate by comparing the determined expected trajectory with the determined first trajectory; and when it is determined that the driving environment has changed from the detailed map information to the extent that the detailed map information is no longer accurate, stop using the detailed map information to control the second vehicle in the autonomous driving mode. use the detailed map information to control the second vehicle in the autonomous driving mode; receive sensor information from the sensor, the sensor information including information about the first vehicle in the driving environment for a predetermined period of time; determine the first state information based on the received sensor information; determine a first trajectory based on the first state information, the first trajectory corresponding to an actual path taken by the first vehicle for the predetermined period of time; determine an expected trajectory for a vehicle in the driving environment of the second vehicle based on the detailed map information; determine that the driving environment has changed from the detailed map information to the extent that the detailed map information is no longer accurate by comparing the determined expected trajectory with the determined first trajectory; and when it is determined that the driving environment has changed from the detailed map information to the extent that the detailed map information is no longer accurate, stop using the detailed map information to control the second vehicle in the autonomous driving mode.