Don't remove fileinfo on BufUnload

If 'hidden' is not set then BufUnload is called every time a buffer
would get hidden, removing the fileinfo. This causes unnecessary ctags
processing of files.