Woolsey was born in Leelanau County, Michigan, on August 29, 1894, the son of Byron Woolsey and Sarah Hall. Woolsey's grandfather, Chauncey Woolsey, died in 1864 at the Battle of Totopotomoy Creek in Hanover County, Virginia during the Civil War. His great-grandfather, Adolphus Woolsey, served in the War of 1812. Woolsey was a descendant of George (Joris) Woolsey, 17th Century settler of New Amsterdam and progenitor of the Woolsey family in America.