Improved build matrix.

PyPy wasn't added yet since the tests need adjustments.