Apparently the system python3-numpy still won't work on Python 3.3 probably due to a binary incompatibility. I think this can be resolved by just installing Numpy with pip via a wheel as we do on Astropy, but in the meantime I'm just leaving 3.3 disabled.