As he grew older ( and following Kasturba 's death ) he was to have more women around him and would oblige women to sleep with him whom -- according to his segregated ashram rules -- were forbidden to sleep with their own husbands .