Cleanup observer cleanup code. Also, add testing that asserts that observers are properly closed.

R=sorvell@google.com, sorvell
BUG=

Review URL: https://codereview.appspot.com/103260043