Actually use the cache when building Plans

This fixes a major performance regression on device backends.  We were previously computing the plan on every par_loop call.