If you limit the number of calories you eat but get most of them from refined carbs, sugars, and saturated fats, your body might not get all the nutrients it needs, especially protein, to keep lean tissue strong. Because of this, you may be losing weight in the form of lean muscle rather than fat, so your love handles may stay the same.