Handle different buffers

if the buffer was changed when jumping to declaration,
make sure we change it back when going back, otherwise
it would move to a seemingly arbitrary position in
the current buffer.