Bradlee was the second of three children; his siblings were older brother Frederick, a writer and Broadway stage actor, and younger sister Constance. The children grew up in a wealthy family with domestic staff. They learned French from governesses, took piano and riding lessons, and went to the symphony and the opera; but the stock market crash of 1929 cost Bradlee's father his job, and he took on whatever work he could find to support his family, from selling deodorants to supervising janitors at the Boston Museum of Fine Arts.