Oliver Remigius was born on 1 October 1742 into an Irish emigrant family living in the Habsburg Monarchy. During the 17th century, laws introduced in Ireland limited, and eventually removed, authority from the Catholic aristocracy, preventing Catholics from inheriting land, sitting in Parliament, and holding office. Many immigrated to Central Europe and sought service in the Habsburg military. One ancestor, Richard Wallis, or Walsh, as he had been known in Ireland, emigrated with his family in 1612, and was a colonel in the Habsburg military. He was killed at the Battle of Lützen in 1632. Another ancestor of Oliver Remigius, George Olivier, Count von Wallis (son of Richard), also served in the Habsburg military during the Thirty Years War under Wallenstein.