Limit the number of transmutes in path finding

They more dangerous than ordinary casting. We can't avoid them entirely because
of the fat pointers to closure objects, let's not use them where they aren't
necessary.