Give CtrlPBuffer a try as an alternative to Buffet.vim.
Scoping files in ctrlp doesn't work when using ag without an .agignore.