Correctly update owners in the tree specializer

Closes #41
Confirmation test