Navigation: Factor out transition determination logic, reuse for dialog

Fxies gh-4951
Navigation: Test correct transition calculation for dialog