Use the right autocommands

Silly me, BufEnter is run every time you switch to a buffer.