Graham was born in Washington, D.C., the son of Colonel James Duncan Graham (1799–1865) and Charlotte Meade (sister of George Gordon Meade). His uncle and namesake, William Montrose Graham (1798–1847), was killed during the Mexican–American War while commanding the 11th U.S. Infantry at Molino del Rey.