Incorporate caffe into network