Properly gather the branches to delete