Farrell was born in Brooklyn, New York in 1927. His grandfather, Patrick Farrell, owned an auto parts company which his father later ran. Following the stock market crash in 1929 Farrell's father committed suicide. His mother placed 4-year-old Bob and his sister in an orphanage for five years, taking them back when she remarried in 1934.