Improve handling of complicated voting permissions