Walking away may be a difficult decision, but it could be necessary if you are unable to overcome your anxiety and attraction. Prioritize your mental and emotional well-being. However, before making this decision, consider seeking guidance from a trusted friend or a professional counselor who can provide perspective and support in navigating the situation. Remember that it's essential to be true to yourself and make choices that align with your values and happiness.