Hosokawa clan
Sumimoto was one of the few sons of Hosokawa Yoshiharu and an adopted son of Hosokawa Masamoto, who was the Kanrei of the Ashikaga shogunate. His roots was the Hosokawa clan at Awa Province. His childhood name was Rokuro (六郎).