Only remove observer if we're observing

We’re only observing this object if we have a delegate for it. Since a background task might come back when we create a new AFURLSessionManager.

Closes #1763