Avoid adding non-existent children to our internal list.

This really only affects testing where we might be returning nil after a
push_single expectation has been met