Unfortunately, the world is filled with individuals who are self-righteous and adamant that anything that is different or a mystery is wrong. We know a person who tells his spouse that he's going to work and instead, goes to the riverboats to gamble. He is lying about where he is and what he is doing. He isn't honoring his commitment to be truthful. The difference lies in perception.
People define infidelity based on their own moral views.
Of course, the biggest, most common moral issue is sex. Because of this, society prioritizes sexual conduct as a higher moral issue than most other types of conduct. In essence, this lifestyle offends the very code of moral ethics most people believe in. Personally, my moral priorities are very different than this. I believe that most couples involved in this lifestyle hold their commitment to be sacred and their perception is that the breach of trust to a relationship has less to do with sex than with honesty.