But Japanese Prime Minister Junichiro Koizumi says he visits the shrine to pray for peace .