Make sure to exit the runnable once we resolve a promise