The second son of William the Hardy, Lord of Douglas, William Wallace's companion in arms, and Eleanor Ferrers. Hugh's elder brother was Sir James Douglas, a hero of the Wars of Independence, and his younger was Sir Archibald Douglas, Guardian of the realm, and Scots commander at the Battle of Halidon Hill.