When the Welfs left in 1180 , the Duchy of Saxony lost a lot of territory .
With the final removal of the Welfs in 1180 , the Duchy of Saxony was sharply reduced in territory .