His family held extensive estates in both Yorkshire and Lancashire, as did one of the biggest noble families of the region, the Nevilles. Between the late 1450s and 1460s the Harringtons had a close relationship with them: "The Harrington brothers [were] feed by Warwick, and their father had been feed by Salisbury," as Rosemary Horrox has pointed out. He was a retainer and councillor to Richard Neville, 16th Earl of Warwick, and was granted duchy of Lancaster leases by him, with his father, Sir Thomas. He was in the Earl of Salisbury's army when it was ambushed by the Lancastrian Lord Audley near Blore Heath, in Shropshire, on 23 September 1459. Although a Yorkist victory, Sir James and his father, along with Warwick's brother John Neville were captured after the battle, at Acton Bridge, and sent to Chester Castle. Harrington was pardoned on 29 March 1460, but Neville was not to be released until the Yorkist victory at the Battle of Northampton in July 1460. However, with James's elder brother John, Sir Thomas later died with Richard, Duke of York at the Battle of Wakefield on 30 December 1460. Following the victory of Edward IV at the Battle of Towton on Palm Sunday, 29 March the next year, Sir James was made the King's escheator for Yorkshire. Following his death, Sir Thomas's feoffees transferred his estates, that his daughters were due to inherit, to James in 1463, a decision that would consume the next eleven years and ultimately have grave consequences for James and his family. On 13 July 1465 he assisted with the capture of the by-now fugitive Henry VI by being the instrument of persuasion that induced Sir Thomas Talbot of Bashall, and Sir Richard Tempest of Bracewell, who were sheltering the King, to betray him. For this he received one hundred marks (£66) for expenses, and a reward of one hundred pounds.