During talks Saturday , Seoul 's Foreign Minister Ban Ki-moon told his Japanese counterpart , Taro Aso , that visits by Japan 's prime minister to a controversial war shrine are disrupting bilateral relations .