A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: receiving sensor data captured by a sensor in an environment; determining that the sensor data represents a first vehicle; determining a first velocity of the first vehicle; determining a first difference between the first velocity of the first vehicle and a speed limit at a location of the first vehicle in the environment, or determining a second difference between the first velocity of the first vehicle and a second velocity of a second vehicle in the environment; determining a feature associated with the first vehicle, the feature being of a feature type that indicates whether the first vehicle is associated with a reverse state; inputting the feature and one or more of the first difference or the second difference into a machine-learned model, the machine-learned model being trained to predict vehicle behavior based at least in part on the feature type and differences in vehicle speeds; receiving, from the machine-learned model, an indication that the first vehicle is associated with the reverse state, the reverse state indicating that the first vehicle is likely to reverse prior to a reverse motion being detected; predicting, based at least in part on the indication that the first vehicle is associated with the reverse state, a trajectory associated with the first vehicle; and determining, based at least in part on the reverse state, a maneuver for an autonomous vehicle to perform. one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: receiving sensor data captured by a sensor in an environment; determining that the sensor data represents a first vehicle; determining a first velocity of the first vehicle; determining a first difference between the first velocity of the first vehicle and a speed limit at a location of the first vehicle in the environment, or determining a second difference between the first velocity of the first vehicle and a second velocity of a second vehicle in the environment; determining a feature associated with the first vehicle, the feature being of a feature type that indicates whether the first vehicle is associated with a reverse state; inputting the feature and one or more of the first difference or the second difference into a machine-learned model, the machine-learned model being trained to predict vehicle behavior based at least in part on the feature type and differences in vehicle speeds; receiving, from the machine-learned model, an indication that the first vehicle is associated with the reverse state, the reverse state indicating that the first vehicle is likely to reverse prior to a reverse motion being detected; predicting, based at least in part on the indication that the first vehicle is associated with the reverse state, a trajectory associated with the first vehicle; and receiving sensor data captured by a sensor in an environment; determining that the sensor data represents a first vehicle; determining a first velocity of the first vehicle; determining a first difference between the first velocity of the first vehicle and a speed limit at a location of the first vehicle in the environment, or determining a second difference between the first velocity of the first vehicle and a second velocity of a second vehicle in the environment; determining a feature associated with the first vehicle, the feature being of a feature type that indicates whether the first vehicle is associated with a reverse state; inputting the feature and one or more of the first difference or the second difference into a machine-learned model, the machine-learned model being trained to predict vehicle behavior based at least in part on the feature type and differences in vehicle speeds; receiving, from the machine-learned model, an indication that the first vehicle is associated with the reverse state, the reverse state indicating that the first vehicle is likely to reverse prior to a reverse motion being detected; predicting, based at least in part on the indication that the first vehicle is associated with the reverse state, a trajectory associated with the first vehicle; and determining, based at least in part on the reverse state, a maneuver for an autonomous vehicle to perform.