Toki clan
After Saitō Dōsan usurped power from the Toki clan in 1530, he moved the area's power into Inabayama Castle atop Mount Kinka. However, he occasionally took refuge in Sagiyama Castle because of its ease of access. Dōsan gave possession of Sagiyama Castle to his son, Saitō Yoshitatsu, in 1548 after some repairs. Originally, Dōsan was planning on naming Yoshitatsu as his successor, but when Yoshitatsu heard that Dōsan might choose another son, Yoshitatsu killed his two brothers. The murders led to the Battle of Nagaragawa between Dōsan and Yoshitatsu in 1556, which resulted in Dōsan's death and the destruction of the castle.