He is a graduate of Wesleyan University, he received a M.A. from the University of York (UK), and a M.T.S. from the Harvard Divinity School. At Wesleyan, Spencer took a class in writing verse with Pulitzer Prize-winning writer Annie Dillard (Tinker at Pilgrim Creek), whom he describes as "an early encourager," along with James Merrill, the Stonington poet with whom Spencer shared a correspondence.