Only update holes where the context has changed