In 1491, Hōjō Sōun intervened in the conflict between the Ashikaga shogunate and the Ashikaga of the Horigoe Gosho and seized control of Izu Province. He then relocated to Niirayama Castle in central Izu, returning Kōkuku Castle back to the Imagawa. The castle later changed hands between the Takeda clan and the Tokugawa clan after the fall of the Imagawa. After the Battle of Sekigahara, Tokugawa Ieyasu assigned the castle to Amano Yasukage, one of his old retainers. Amano was later disposed after an uprising in his territory and the castle was abolished in 1607.