Reviewer: Mike - Minor performance improvement

Since we never walk the authorized AoR set, we can use an unordered set for better insert/retrieve performance.