Remove all child nodes try 2

Compared to the previous patch this now ensures that we clear/update the internal pointers in all cases.