Remove thread-poll from caching call sites

* The IR builder adds thread polls at all required places.