Surah Maryam Ayah 18
I seek refuge to God from YOU. And this was possible if that “man” was God conscious or have fear of God. By hearing the name of God he would desist from doing anything indecent. One possibility was that God Himself gave her a special protection from the evil of man and the other was that man desist himself from doing anything bad for God’s sake. This is also a refuge God has provided to people under His name. Oath too is of that kind. We ask people to regard their oaths and covenant in the name of God.
Language doesn’t follow logical reasoning. It is a sudden expression and has its own implications