Ok, so not installing requirements every time is the wrong way to optimize this.  

You could just as easily set PIP_DOWNLOAD_CACHE or use pip wheel and still have a 1) new image every time 2) fast installations.