Preserve # when descending too

Forgot to guard descent with <CR> against changes to the alternate file
name. Now <C-^> goes back to the last open file buffer, regardless of
how many times one may have navigated up and down the directory tree.