Properly remove a component from a previous owner when adding.

Fixes #81.