They have disobeyed God. It's true that he hasn't given them the grace that he gave the elect, so he determines who is going to be left in their sins. But they do choose to disobey. Thus it is just to condemn them. He chooses people from all nations, all walks of life, etc.
Election is a completely unmerited gift.
Of course we observe that there are people who are Christians for a while and then fall away permanently. There are two possibilities. Some of them probably weren't really saved in the first place. They didn't actually put their trust in Christ for salvation, but simply went along with what looked like all the Christian rules. But others really did have faith.