Trim down the build matrix a bit.

We probably only need one or two builds per minor version, except where there are BC breaks (e.g. 1.3.4).