Fernando was born in Piedrahíta, Province of Ávila, on 29 October 1507. He was the son of García Álvarez de Toledo y Zúñiga, heir of Fadrique Álvarez de Toledo and Enríquez de Quiñones, II Duke of Alba de Tormes, and of Beatriz Pimentel, daughter of Rodrigo Alonso Pimentel, IV Count - I Duke of Benavente and his wife, María Pacheco. Fernando was orphaned at age three when his father, García, died during a campaign on the island of Djerba in Africa in 1510. At the age of six, Fernando accompanied his grandfather, the second duke of Alba on a military mission to capture Navarre.