You have to look at both to get good, long-lasting results. It is a proven fact that diet alone will not give you long term weight loss and in fact can leads to yo-yo weight where you end up heavier than you were before. The only proven way to change your body's own self-image so that it does not regain the weight, plus a cushion, is to exercise. Do whatever you can first. Take easy steps.
Every calorie does count.
You have to change the way you eat forever. Do it gradually and ease into it so that it lasts. That's how many calories you should eat a day. Don't go under that. If you eat too few calories you actually can gain fat and lose muscle and that's going in the wrong direction.