Try using the apt-get version of matplotlib too, in travis build.

This should speed up and simplify the build, if it works.