Cache merged permissions to prevent recalculating each time