However, in the dispute over who would succeed Tokugawa Iesada in 1858, Seisai supported the Hitotsubashi faction. When the Iemochi faction won, they purged the losing faction, Seisai was rusticated to Kōfu. He was pardoned in 1862, and in 1865 assigned to guard the "Second Moat" perimeter of Edo Castle. In 1866, during the Second Chōshū expedition, he was detailed to ensure the cooperation of samurai from the Kokura Domain, but instead they burned down Kokura Castle. The Bakufu system was falling apart.