A method comprising: receiving, by one or more processors, information identifying a pickup location for a passenger assigned to a vehicle and client device information for authenticating the assigned passenger; receiving, by the one or more processors, sensor data from a perception system of the vehicle identifying objects in an environment of the vehicle; when the vehicle is within a predetermined distance of the pickup location, attempting, by the one or more processors, to authenticate a client device (420, 430, 440) using the client device information; when the client device has been authenticated, using, by the one or more processors, the sensor data to determine whether a pedestrian is within a first threshold distance of the vehicle; when the pedestrian is determined to be within the first threshold distance of the vehicle, determining, by the one or more processors, a second threshold distance of the vehicle that is smaller than the first threshold distance of the vehicle based on an expected walking speed; using, by the one or more processors, the sensor data to determine whether the pedestrian is within the second threshold distance of the vehicle; when the pedestrian is determined to be within the second threshold distance of the vehicle, determining, by the one or more processors, that the pedestrian is moving towards the vehicle; when the pedestrian is determined to be moving towards the vehicle, causing the vehicle, by the one or more processors, prior to reaching the pickup location, to wait for the pedestrian to enter the vehicle while the vehicle is stopped; and after the pedestrian enters the vehicle, maneuvering, by the one or more processors, the vehicle to a destination with the pedestrian as an occupant of the vehicle. receiving, by one or more processors, information identifying a pickup location for a passenger assigned to a vehicle and client device information for authenticating the assigned passenger; receiving, by the one or more processors, sensor data from a perception system of the vehicle identifying objects in an environment of the vehicle; when the vehicle is within a predetermined distance of the pickup location, attempting, by the one or more processors, to authenticate a client device (420, 430, 440) using the client device information; when the client device has been authenticated, using, by the one or more processors, the sensor data to determine whether a pedestrian is within a first threshold distance of the vehicle; when the pedestrian is determined to be within the first threshold distance of the vehicle, determining, by the one or more processors, a second threshold distance of the vehicle that is smaller than the first threshold distance of the vehicle based on an expected walking speed; using, by the one or more processors, the sensor data to determine whether the pedestrian is within the second threshold distance of the vehicle; when the pedestrian is determined to be within the second threshold distance of the vehicle, determining, by the one or more processors, that the pedestrian is moving towards the vehicle; when the pedestrian is determined to be moving towards the vehicle, causing the vehicle, by the one or more processors, prior to reaching the pickup location, to wait for the pedestrian to enter the vehicle while the vehicle is stopped; and after the pedestrian enters the vehicle, maneuvering, by the one or more processors, the vehicle to a destination with the pedestrian as an occupant of the vehicle.