Cleaned up deps to avoid pip compiling code on minimal installs

Change-Id: Iccb8faf10f49b7c0b2081993a352cd951a9daaa4