FIX: Not possible to withdraw invitation of inactive users

Fixes https://www.openproject.org/work_packages/1565

Old pull request (for conversation) was https://github.com/finnlabs/openproject-meeting/pull/23