Don't use fixed remote ref for refs

Instead depends on what the currently selected remote is