Most of the annoyance related to fitness apps falls within improper usability and user experience; namely, users are often annoyed with:
Lack of automation
Battery discharge
Lack of in-app community and social features
GPS errors
Monotonous gamification