Set up the monitor only on await for tasks

Closes #2345