libffi: Move it to the base system

We are not interested in doing CI for it right now.