fix numerical issues in frozen cell list test