齋藤道三
It is unclear when the castle was demolished, but it is known that Saitō Dōsan chose to use Inabayama Castle as his base when he gained control of the area.