Only adds children if the parent exists first, which allows us to filter the tree by attributes on the leafs