James 's paternal grandfather was Walter Butler , 11th Earl of Ormond .
His mother took him to England where his grandfather ( Walter Butler , 11th Earl of Ormond ) made him go to a Catholic school .