Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a scalable mean-field game (MFG) approach to the problem of cloud resource management. It introduces an online natural actor-critic (NAC) algorithm for MFGs, which uses function approximation and lets the mean-field state naturally evolve as the agents learn. The authors establish finite-time convergence of NAC with linear function approximation and softmax parameterization, but also implement neural-net based function approximation. Experimental results on open-source serverless platform OpenWhisk with real-world workloads from production traces demonstrate that NAC is scalable to a large number of agents and significantly outperforms other common baselines.
strengths_and_weaknesses: Strengths
1. The paper is well written. Even the theory parts that are more involved are not hard to follow and the authors provide adequate refences and clarifications to explain the different concepts. I found this very helpful, because the NAC algorithm contains a number of steps such as policy evaluation, gradient estimation, mean-field update and policy update.
2. The problem that motivates this work is an important real-world problem. Furthermore, a MFG approach is a quite reasonable design choice for this problem, in particular for large numbers of users where collective user behavior can be summarized by a population distribution, and a common policy can be applied to all agents.
3. I was unable to check the proofs in detail, but the proposed algorithm and the ingredients therein make sense, as they follow faithfully various prior works.
4. The related work is concise but seems to cover a lot of important and relevant prior works.
5. The experiments show that the proposed NAC does not suffer from the zigzag fluctuations of the double-loop version of NAC that uses a fixed point iteration. Furthermore, experiments with real-world workloads from production traces show that NAC significantly outperforms other common baselines such as ENSURE and OpenWhisk's original resource manager. 

Weaknesses
1. The novelty is not strong. Convergence of entropy-regularized natural policy gradient with linear function approximation has been already studied by (Cayci, He, and Srikant, 2021). And indeed this work builds very heavily upon this specific work (which, incidentally, is a pre-print and has not been accepted to peer-reviewed venues yet). The main difference is that NAC is now applied to a MFG setting, where the main solution concept is that of a Nash Equilibrium. Still, due to the mean-field approximation, the agent essentially faces a single-agent policy optimization problem, and the game-theoretic setting roughly becomes equivalent to a single-agent Markov decision process. That said, I do not claim that the two works are identical, since the deal with a different setting; however, they do overlap significantly.
2. In the experimental evaluation, NAC is only compared to two other heuristic-based baselines. It would have made much more sense to also compare against other RL-based schemes. As the authors point out in the related work, there are various RL-based frameworks for scheduling or resource management (e.g., FIRM, FaaSRank, etc.). I feel that the authors would have made a much stronger point is they were able to show that the NAC algorithm can outperform other RL-based competitors (even by focusing on a simpler objective such as minimizing latency only). Even if some approaches are single-agent, the authors could also experiment multi-agent RL-based baselines to showcase the benefits of their framework (even in a basic setup to demonstrate the scalability challenges with traditional multi-agent approaches).
questions: 1. In terms of theory, what are the main differences of this work with the NAC setting with linear function approximation by (Cayci, He, and Srikant, 2021)? I understand that the setting is a bit different, because this work focuses on mean-field approximation for a multi-agent setting, whereas the cited work is purely single-agent. On the other hand, the mean-field approach effectively transforms the problem into a single-agent one. What are the additional innovations/challenges (particularly in terms of theory and proofs) of this work compared to the work by (Cayci, He, and Srikant, 2021)?
2. Why have the authors not included any RL-based baseline (except for the double-loop version of NAC)? Multi-agent RL-based approaches without the mean-field assumption may suffer from limited scalability, so the authors could show the benefit of MFG NAC in a rather basic setting. Is it also possible to compare single-agent RL-based approaches (e.g., FIRM) to the proposed framework, e.g., by looking at the problem from the perspective of the single agent?
3. In Equation (1), right in the middle do the authors perhaps mean V^{\pi,\lambda}_{\mu^*}(\rho) instead of V^{\pi}_{\mu^*}(\rho)? Shouldn't the \lambda appear somewhere?

---AFTER REBUTTAL---
The authors addressed my concerns; I am therefore willing to raise my recommendation to Weak Accept.
limitations: The authors have addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The competition and scheduling of cloud service resources is a significant research problem with economic value. Due to the emergence of new cloud service paradigms such as serverless computing, we consider the aid of multi-agent reinforcement learning (MARL) to address new management challenges. To this end, the authors design a mean-field game (MFG) approach that can efficiently manage large-scale cloud users and applications. They propose an online natural actor-critic algorithm for approximating functions in mean-field games and demonstrate the finite-time convergence of the algorithm theoretically. The study evaluates the solution's effectiveness on OpenWhisk using workloads captured from real production systems, showing advantages in scalability, latency, and resource utilization.
strengths_and_weaknesses: Strengths:
- The cloud resource competition and scheduling problem has real research significance and economic value
- Comprehensive theoretical basis and proof of properties
- The efficiency advantage of the proposed solution makes it adaptable to real system scenarios

Weaknesses：
- The background and problem formalization are insufficient, resulting in the disjointed problem background and methodology
- Validation in large-scale system scenarios/cases
questions: Overall, this paper constructs a comprehensive multi-agent reinforcement learning system and demonstrates the properties of online natural actor-critic algorithms in detail. Still, the most prominent weakness is the lack of necessary problem formulation (i.e., cloud resource management). Therefore, at the very beginning of the paper, it is suggested to supplement the problem case study and formalization to formalize the resource scheduling problem into a multi-agent reinforcement learning process in the context of a serverless computing scenario. This issue involves a lot of real details and needs to be explained. For example, cloud users do not need to manage the computing resources in a serverless computing scenario, so how can self-interested users compete for resources.
limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a mean field game approach to the many selfish agent problem. The paper contextualises this around the resource management problem for FaaS before moving on to a theoretical evaluation of the problem and finishes with experimental results.
strengths_and_weaknesses: The paper presents a clear introduction to the FaaS problem along with a detailed mathematical analysis of the MFG solution. 

However, the paper is very dense and feels like two papers rammed into one. The introduction is all about the Cloud and the FaaS problem. The next section of the paper is a dense mathematical work on the MFG problem - with no mention of the FaaS problem. The FaaS only returns as one of the examples in the results section. It would have probably been better to have couched this paper as a fully theoretical paper and just keep the FaaS as one of the results examples. This would have given more space for covering the 'middle paper'. This could have helped in making it clear what the 'NeurIPS' element of this work was.
questions: The paper is in general well written. No small questions.
limitations: This was not discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to solve the problem of improving latency and resource utilization in serverless platforms. The paper proposes a mean-field game approach where they design a natural actor-critic (NAC) learning paradigm for MFGs with function approximation.  The evaluation shows the convergence of their approach, and shows some latency and resource efficiency improvement on the OpenWhisk platform.

Note: I am not an expert on the mean-field game theory side so I am not sure about the theoretical contribution, but I am confident in my evaluation of the serverless system side. I appreciate the authors’ effort to try to formalize the resource allocation problem in serverless.

strengths_and_weaknesses: Strengths
1. Interesting approach to map resource allocation (or autoscaling) problem in serverless to MFGs, and the authors try to solve the system challenge in a theoretica way.
2. The paper applies their approach to a real system (OpenWhisk).

Weaknesses
1. The mapping between the idealized theoretical setup and the real system is a bit vague. E.g., the major part considers homogeneous agents and only the OpenWhisk experiments consider heterogeneous agents.
2. The paper does not evaluate or prove the scalability of the proposed algorithm.

questions: 1. The majority of the paper assumes homogeneous agents, and the theorems and proofs are all based on this assumption. However, the evaluation on OpenWhisk assumes heterogeneous agents. Could you provide more details on how did you adapt the algorithms for real deployment? And are those proofs still valid for heterogeneous agents? Please clarify.
2. It is unclear in the paper why this MFG approach is uniquely tailored for serverless. The theorems did not include any assumptions from the serverless system.
3. This paper only considers a very coarse-grained resource allocation type – vertically adding CPUs or memory, or horizontally adding containers. How does it scale to many different resource allocation knobs such as CPU frequency, memory bandwidth, I/O bandwidth, etc? 
4. The reward function in Line 332 seems to be constructed naively by averaging CPU and memory utilizations, could you clarify that?
5. How does the resource allocation part interact with the scheduler? Do you use round-robin or least loaded worker? How do you decide which functions are co-located in one worker (function placement issue)? Those are unclear in the paper.
6. Usually resource allocation needs some time (seconds to minutes) to take effect, so the reward would only be observed after a certain delay. Therefore, resource allocation needs to be proactive, predicting what will happen next. However, the algorithm does not seem to consider such delay.
7. In prior work such as FIRM, they consider a workflow of serverless functions and incorporate dependencies between functions. How does MFG work considering dependencies?
8. Though the authors claim that scalability is the main issue of resource allocation in the cloud the paper does not provide proof or evaluate the scalability of their approach. In the evaluation, only 50 functions are tested which indicates only 50 agents are running concurrently. But the production environment usually serves thousands of different functions simultaneously.

limitations: The authors discussed some limitations and potential social network impacts in the paper. I found some limitations after reading the paper (mentioned in the above Questions section).
review_title: null

Meta-review: metareview: I agree with the reviewers that this is a well-written paper on an interesting application of mean-field games. The paper is a nice blend of theoretical developments and experimental evaluations. I believe that it will be well-received by the NeurIPS community and recommend acceptance. 
confidence: Certain
Decision: Accept