Memoize user serializer permission check