I don't think we are really disagreeing that much. I specialized in biographical research with Stephen Oates at the University of Massachusetts. History is about perception.
Memoirs are one piece of the puzzle but an important piece.
The best way to verify them, and it's imperfect at best, is to look for independent verification. However, if you're writing history, which is all about human endeavor, and you are unwilling to use memoirs as a serious source, then I think you're missing something.