The ex-believer may feel doubly misunderstood and isolated