Fix race condition around stopping observe

The race condition should not have had any adverse effects other than
logging an exception.

Fixes #2070.