It's a matter of being able to click. This does not mean that there is something wrong with you. At the outset, relationships are based on physical attraction and also basic nature of the person and last but not the least the instincts. This opinion is no certificate of the other being a bad person. It just means that the other is not 'my' type.
Lets see now when the situation is reversed.
Someone might want to have a relation with you and you reject the thought. This is a situation everyone needs to learn to deal with. Everyone needs to learn to over-come his fears. Fear of being rejected should not stop from asking. You need to try and accept the outcome.