Only observe weak properties for deallocation

Fixes #672.

Unsurprisingly, this also improves performance.