Vim only lets you set cursor position on the current buffer.