Georg Olivier's ancestor was Richard Wallis of Carrickmines, County Dublin, who became one of the first Irish officers in imperial service in 1632. He died later that year after being wounded in the Battle of Lützen. Richard's eldest son Theobald returned to Ireland, with this branch of the family from then on taking the name Walsh, whilst his youngest son Olivier remained in the imperial Austrian army and became the founder of the Austrian Wallis branch before dying in 1667 as a major general in Hungary. Theobald's son, Feldzeugmeister Ernst Georg Wallis (died 1689) was the father of George Olivier and his elder brother (died 1737).