Use weak references in KVO observation

Hopefully fixes #1206 and #1548.
How did we behave before?