why not just check for `admin group` permissions at first and only then check for ability to remove owner.
Separating authorization from other logic will prevent security issues in future. 
So keep this before filter. And check for removing wner inside `user_group` without abilities logic