Avoid double lookups of commits during walking operations.

Fix a performance regression since the grafts code.