BLD: remove numpy from apt if full deps were installed