Tudor was born in East Windsor, in the Colony of Connecticut, in 1769, son of Samuel Tudor, III, and Naomi Diggens. His father was called to the Lexington alarm as a lieutenant in the Connecticut militia, while his uncle, Dr. Elihu Tudor, was a loyalist and intimate friend of loyalist Governor William Franklin of New Jersey. Elihu was a preeminent surgeon who attended to British General James Wolfe at the Battle of Quebec. Tudor’s grandfather, Samuel, Jr, was a Yale-educated Presbyterian minister. The family descended from Owen Tudor, a founder of Hartford who was in Connecticut by 1645 and who made unsubstantiated claims to be related to the British monarchs.