Clean up invalidation code due to add or removal of children

We only need to invalidate the renderer if we are adding or removing a direct child of a host or a shadow root.

Other cases are handled by overriding `invalidateShadowRenderer`

This cleanup was part of trying to fix #216