Drastically speed up infer by only inferring what's output.