Be more patient if grab_pointer() doesn't succeed immediately