POP is very simple, so it's a huge win that it's text.
IMAP is a more interesting case as it has a number of complex datastructures which use a simple S-expression style syntax. I don't know of any significant problems that have been caused by these being text encoded. There are certain traditions which can make text protocols painful as they get complicated. Thankfully most of the IMAP server implementors didn't make this mistake so the IMAP protocol is one of the most syntacticly interoperable text protocols. It does become harder to precisely specify a text protocol as it becomes more complex.