Born in 1896 in New Rochelle, New York, Robert was a son of Arthur Murray Sherwood, a rich stockbroker, and his wife, the former Rosina Emmet, a highly accomplished illustrator and portrait painter known as Rosina E. Sherwood. His paternal grandmother was Mary Elizabeth Wilson Sherwood, the author and social leader. He was a great-great-grandson of the former New York State Attorney General Thomas Addis Emmet and a great-grandnephew of the Irish nationalist Robert Emmet, who was executed for high treason after leading the Irish rebellion of 1803, one of a series of attempts to dislodge British rule in Ireland, in 1803. His relatives also included three other notable American portrait artists: his aunts, Lydia Field Emmet and Jane Emmet de Glehn, and his first cousin, once removed, Ellen Emmet Rand.