Branches are not generally evil. In this case 1 branch miss is to be expected. I think keeping the buffer logic separate is worth it.