Speed up inotify tests by only sleeping as long as necessary