Remove hole in the middle

There is additional cleanup that will come with the `DirPath` merge but at least the hole-in-the-middle is gone now.
This is a behaviour change by the looks of it, it is no longer hard failing if it can't infer priority.