Ingore some pytest related cache dirs
Only allow cPO and fPO on CPython

Since we are using ctypes, let's - for now at least - only allow those
parsers on CPython

To allows this on PyPy we'd need to reimplement the interface using
CFFI.