The point of `apt-get` installing those packages is not to get the Python packages, but all the dependencies, e.g. a fortran compiler that is needed for scipy.
I think leaving those is in results in a more robust travis-ci build when they upgrade their VMs.