Add `wait_if_ongoing` option to _flush requests

This commit adds the ability to force blocking on the flush operaition
to make sure all files have been written and synced to disk. Without
this option a flush might be executing at the same time causing the
current flush to fail and return before all files being synced.
Move this option to the top, as it is the only non-internal one