Segal was born in Saratov, Russia and moved to Palestine as a toddler. He survived the 1929 Hebron massacre. He was the founding rabbi of the Young Israel of Newark, and served there for thirty-three years, from 1945 to 1978. During this period, Segal also served a three-year term as president of the Rabbinical Council of America, from 1968 to 1971.