He was born in Madrid, the eldest son of Gregorio José de Toro and of his wife Josefa Dumont y Michel. He became Count at the death of his father in 1816, at the young age of 18. He fought with the Spanish Army against independence, and went into exile in Perú with the remains of the Royalist army after the defeat at the Battle of Chacabuco in 1817, while the pro-independence government confiscated all his properties.