Try to exit gracefully in parallel