Even though Toji forgot his son's name and even sold him to the Zenin family, he loved Megumi more than anything. The love for his son is unquestionable, and he was experiencing difficulties in expressing that. And well, this is what makes many people say he wasn't a good father.