Use weak references in KVO observation

Hopefully fixes #1206 and #1548.