Only add an exit time if the semaphore was actually acquired to

prevent excessive semaphore releases in the maintenance thread.