Fürstentum Tarent
James belonged to the noble House of Baux, specifically the branch settled in the Kingdom of Naples. He was the son of Francis, Duke of Andria, and Margaret, daughter of Prince Philip I of Taranto and his second wife, the Empress Catherine II, Princess of Achaea. His mother's brother was Prince Philip II of Taranto. In 1373, the childless Philip declared his nephew his universal heir. He thus stood to inherit the Principality of Taranto in the Neapolitan kingdom, and the Principality of Achaea in Greece, as well as a claim on the Latin Empire.