Fix build matrix to be as minimal as possible.