improve layer ordering so layers with similar ancestries end up next to each other