Torgoff was born in New York City and grew up in Glen Cove, on Long Island. He is the son of Bess Kagan and Irving Torgoff, a two time All American basketball player at LIU who later became a notable player in the NBA as the first “sixth man” in pro basketball history when he played for Red Auerbach's Washington Capitols. Torgoff grew up playing sports before being swept up in the turbulence of the late 1960s as a teenager, becoming one of the leaders of the student strike at his high school after the shootings at Kent State in May 1970. He attended SUNY Cortland and the University of Neuchatel, Switzerland, receiving BAs in both History and French.