Japan 's relations with its two neighbors have deteriorated because of Mr. Koizumi 's annual visits to the shrine most recently last month .