Handle traversing back up the tree

Refactored some of the code to handle these cases.