Travis: remove NumPy if necessary

... ignoring failures with the assumption that failing to remove is
always caused by NumPy not being present in the first place.

Signed-off-by: Giel van Schijndel <me@mortis.eu>