Revert cffi==0.4 pinning on non-pypy interpreters

Apparently 0.4 has some issues on CPython on some platforms or at least
the one travis is using.