Remove Additional path for Caffe: numpy conflict

Going to use a venv for caffe