Simplify the logic around parents/children_depth

This makes the code a bit more compact and easier to read plus that we
explicitly check for -1 instead of allowing any negative value.