Mr. Aso said Japan did not have to worry about being isolated because of Prime Minister Junichiro Koizumi 's visits to the shrine since the only countries complaining were South Korea and China .