<figure><img src="image_1.png" /></figure>

HAL
open science

# Reinforcement Learning Agent Design and Optimization with Bandwidth Allocation Model

Rafael F. Reale, Joberto S B Martins

► To cite this version:

Rafael F. Reale, Joberto S B Martins. Reinforcement Learning Agent Design and Optimization with Bandwidth Allocation Model. International Conference on Computational Science and Computational Intelligence (CSCI), American Council on Science and Education, Dec 2022, Las Vegas, United States. 10.48550/arXiv.2211.12987 . hal-03999836

HAL Id: hal-03999836

https://hal.science/hal-03999836

Submitted on 21 Feb 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Reinforcement Learning Agent Design and Optimization with Bandwidth Allocation Model

Rafael F. Reale

Federal Institute of Bahia (IFBA - Valença), Brazil

https://orcid.org/0000-0003-4464-4548

Joberto S. B. Martins

Salvador University (UNIFACS), Brazil

https://orcid.org/0000-0003-1310-9366

**Abstract**—Reinforcement learning (RL) is currently used in various real-life applications. RL-based solutions have the potential to generically address problems, including the ones that are difficult to solve with heuristics and meta-heuristics and, in addition, the set of problems and issues where some intelligent or cognitive approach is required. However, reinforcement learning agents require a not straightforward design and have important design issues. RL agent design issues include the target problem modeling, state-space explosion, the training process, and agent efficiency. Research currently addresses these issues aiming to foster RL dissemination. A BAM model, in summary, allocates and shares resources with users. There are three basic BAM models and several hybrids that differ in how they allocate and share resources among users. This paper addresses the issue of an RL agent design and efficiency. The RL agent's objective is to allocate and share resources among users. The paper investigates how a BAM model can contribute to the RL agent design and efficiency. The AllocTC-Sharing (ATCS) model is analytically described and simulated to evaluate how it mimics the RL agent operation and how the ATCS can offload computational tasks from the RL agent. The essential argument researched is whether algorithms integrated with the RL agent design and operation have the potential to facilitate agent design and optimize its execution. The ATCS analytical model and simulation presented demonstrate that a BAM model offloads agent tasks and assists the agent's design and optimization.

**Index Terms**—reinforcement learning, agent design, agent optimization, bandwidth allocation model, RL task offloading.

## I. Introduction

Reinforcement learning [1] is being extensively adopted for the solution and optimization of problems for real-life applications like 5G and beyond (5G&B), the internet of things (IoT), internet of vehicles (IoV), and network slicing [2] [3] [4] [5].

The fostering of machine learning use in various areas depends, at least in part, on having agent modeling and design approaches or methods that make ML agents competitive concerning heuristics and meta-heuristics. In effect, it is expected that ML agents with formal, framework-supported, or speedy-like approaches could not only solve the unsolved problem using heuristics but also address the current problems that require intelligence or knowledge acquisition.

Research supported by ANIMA Institute.

The design of machine learning agents is a fundamental issue currently addressed by research. As such, the design of machine learning agents needs frameworks, methods, or approaches to facilitate and optimize agent design.

Bandwidth allocation models (BAM), in summary, share resources among users based on resource priority [6]. The BAM models have attributes that allow controlling the allocation and sharing of resources with different characteristics (behaviors).

This paper addresses the issue of using a bandwidth allocation model (BAM) as part of the reinforcement learning agent deployment. It investigates how the BAM model can support an RL agent design and efficiency. The BAM capability to offload computational tasks from the RL agent is evaluated with the objective to simplify the agent modeling task and allow the agent operation optimization aiming to facilitate and improve agent design and operation.

The remainder of the paper is structured as follows. Section II presents the related work and Sections III and IV are summary background sections about the RL agent design issues and bandwidth allocation models operation and management. Sections V presents an analytical model for the BAM ATCS. Section VI illustrates the ATCS operation dynamics with a use case simulation and Section VII discusses the ATCS support for RL agent design and optimization. Finally, section VIII concludes with an overview of the main highlights, contributions, and future work.

## II. Related Work

Reinforcement learning agents are being developed for various application areas and the issues and problems related to these developments are mentioned in the literature and are the focus of attention [7].

The state-space explosion issue impacts substantially the RL agent and there are approaches that are used to mitigate this problem that limits agent state-space [8], or use auxiliary processing to do so [9] [10]. Most of the time, these approaches are presented as part of the agent definition and deployment not having specific regard or more extensive discussions about it.

The agent design and optimization approach proposed in this paper is aligned with the strategy of using auxiliary
<figure><img src="image_2.png" /><figcaption>Fig. 1. BAM Resource Allocation and Resource Sharing.</figcaption></figure>

algorithms, methods, or software to reduce the working state-space of the agent [11] [12].

The proposal uses the bandwidth allocation models (BAMs) to mitigate the state-space explosion issue in an RL agent and, to the limit of our knowledge, this specific approach is new in the literature.

Machine learning agents in general are adopting new development and deployment approaches like distributing agents (multi-agent RL) [13] [14] and sharing knowledge among agents (federated learning) [15] [16]. The pushing of machine learning agents' design into new development strategies is, to some extent, motivated by the need to enforce efficiency.

### III. Reinforcement Learning Agent Modeling and Training

Reinforcement learning agents are being used extensively for a broad range of problems with multiple input variables and multiple objectives. In such scenarios, heuristics or meta-heuristics are difficult to develop and apply and, consequently, knowledge acquisition with machine learning becomes of paramount importance.

However, machine learning agents need to be modeled to the target problem aiming to solve it. Like any heuristic or meta-heuristic algorithm design, the agent modeling task or agent design requires minimal expertise concerning the addressed problem and a set of development and experimentation steps.

There are frameworks and methods for agent design like the design of experiments (DoE) methodology [17], the ANOVA statistical method for data analysis [18], and, obviously, the ad-hoc method in which a trial-and-error design is used.

The steps involved in the reinforcement learning agent design for a target problem are as follows [17]:

* Definition and model of the relevant agent input variables (factors) and output variables (variables of interest);

* Experiment design and test runs; and

* Statistical analysis of data.

Some of the reinforcement problems and issues concerning agent modeling, training, and efficiency include:

* The agent designing process;

* The agent state-space explosion; and

* RL parameters definition like the reward matrix and RL learning parameters tuning.

Agent design, state-space explosion control, and agent parameters definition and tuning are large and complex problems. This paper aims to contribute to facilitating the design process and reducing the state-space explosion for an RL agent.

### A. The Proposed Approach to Facilitate Agent's Design and Contribute to Agent's Efficiency

The essential aspect investigated is the inclusion of a BAM model software in an RL agent deployment to evaluate to what extent the BAM model can offload computational tasks that are, primarily, executed by the RL agent software.

In following this approach, it is necessary to understand the essential aspects of BAM model operation and to what extent it can mimic the behavior and operation of an RL agent.

### IV. Bandwidth Allocation Model Summary

The bandwidth allocation models basically allocate and share resources among users considering their priority [6]. There are three basic BAM models and several hybrids
that basically differ in how they allocate and share resources among users.

The three basic bandwidth allocation models that are considered in this discussion are:

* The Maximum Allocation Model (MAM) [19];

* The Russian Dolls Model (RDM) [20]; and

* The AllocTCSharing (ATCS) model [21].

All BAM models group resources in resource classes with priorities and each of the BAM mentioned above has resource allocation and sharing characteristics as follows:

* In the MAM model, resources are grouped into isolated classes that never share their resources with other resource classes;

* In the RDM model, resources belonging to high-priority classes can be shared with lower-priority resource classes, and

* In the AllocTCSharing (ATCS) model, resources are shared among all resource classes regardless of their priority.

Figure 1 illustrates the BAM resource allocation and sharing operations commonly used by the different BAM models.

It is essential to highlight that BAM resource allocation is meaningful only when there are limited resource constraints. In other words, some resource sharing is required to comply with users' demands dynamically.

## A. BAM Parameters for Resource Allocation

The bandwidth allocation model parameters used to model the target problem and to manage BAM operation are as follows (Figure 2):

* Class resource constraint ($R\_Ct_y$); and

* Class priorities ($P\_Cl_j$);

The class resource constraint is the number or amount of resources available to the class. Users request resources from classes.

<figure><img src="image_3.png" /><figcaption>Fig. 2. BAM resource class ($R\_Cl_y$) parameters setup: resource constraint ($R\_Ct_x$) and priority ($P\_Cl_j$) by class.</figcaption></figure>

Although the application of BAM models has focused initially on sharing the resource bandwidth (link bandwidth sharing problem), it is worth highlighting that BAM

resources may be generalized to allocate and share various discrete units used by computational systems like slots for elastic optical networks (EON) [22] and slice resource management [23], among others.

## V. The ATCS Analytical Model

The BAM models (MAM, RDM, and ATCS) essentially share resources among clients based on their priority. A formal model describing their operation is necessary to detail how they allocate and share resources. The BAM model focused on in this paper is the ATCS since it has a flexible approach to allocating and sharing resources and, as such, it potentially shows a closer behavior to an RL agent.

The AllocTC-Sharing (ATCS) analytical model presents the resource allocation and sharing behavior executed by the model. It aims to identify to what extent the ATCS mimic the behavior of a reinforcement learning agent allocating and sharing resources among user with similar objectives as defined for the ATCS model. The ATCS model is the target of the analytical presentation because it provides the best possible method to allocate and share resources among clients as discussed in [24]. The ATCS choice is also motivated by the fact that ATCS has more capabilities in terms of resource sharing and consequently may potentially be closer to an RL agent's behavior.

The network infrastructure for ATCS is modeled as a bidirectional graph $N^q = (N^q, L^q)$, where $N^q = \{n_1, n_2, n_3, ..., n_n\}$ is the set of nodes in the network $q$ and $L^q = \{l_{ij}\}$, with $i$ and $j = 1, 2, 3, ..., n$, is the set of connections or links, with link $l_{ij}$ connecting $n_i$ to $n_j$.

The node connectivity matrix (NCM) is defined by $C = [c_{ij}]$ with $i$ and $j = 1, 2, 3, ..., n$, and $c_{ij} = 1$ if $n_i$ is connected to $n_j$; 0 otherwise and $c_{ii} = 0$ $\forall i$ by definition.

The node resource matrix (NRM) for the bidirectional network is defined by $NR = [NR_{ij}]$ with $i$ and $j = 1, 2, 3, ..., n$, where $NR_{ij}$ is the total amount of resources of node $n_{ij}$ managed by the ATCS to allocate all incoming resource requests arriving at $n_i$ to $n_j$.

This modeling approach allows a generalization for the type of resources in a node that is available for another connected node for both directions in terms of the node connectivity ($n_{ij}$ and $n_{ji}$).

The ATCS defines for each connected node $n_{ij}$ (node $n_i$ connected to $n_j$) a set of resource classes ($R\_CL^k$), with $k = 0, 1, ..., n$. Each resource class $R\_CL^k$ belonging to the node $n_{ij}$ has a resource constraint $R\_CT^k$ (Resource Class Constraint), such that:

$$NRC_{ij} \geq \sum_{i=1}^{n} \sum_{j=1}^{n} \sum_{k=1}^{n} R\_CT_{ij}^{k} \quad (1)$$

Where:

* $NRC_i$ is the total node resource capacity available at $n_j$ to be allocated to connected node $n_i$, and

* Each connected node has a set of classes with a limited amount of resources ($R\_CT^k$).
<figure><img src="image_4.png" /><figcaption>Fig. 3. ATCS Resource Allocation and Resource Sharing among Resource Classes.</figcaption></figure>

Each node can consume resources from the next node in the path (bidirectional path). As such, resources might be allocated in a sequence of nodes like, for instance, packets (traffic) flowing through a series of nodes allocating resources (bandwidth) in the path. Again, it is essential to highlight that with this generalization approach, resources, independently of their type, can be allocated at network nodes.

In the ATCS operation by node, $R_{ij}R_{Cl}^kN^j$ is the incoming resource request from node $i$, with $i = 1, 2, ..., n$, belonging to resource class $R_{CL}^k$, with $k = 0, 1, ..., n$ at node $n_j$.

Resources are dynamically allocated by class and the ATCS defines the resource class ($R_{Cl}$) used or allocated resources ($R_{Cl\_UR}$). The $R_{Cl\_UR}$ matrix is defined by $R_{Cl\_UR} = [R_{Cl}^kUR_{ij}]$, with $k = 0, 1, ..., n$, and $i$ and $j = 1, 2, ..., n$.

The bandwidth constraint for the resource classes is:

$$R_{Cl}^kUR_{ij} \le R_{Cl}^k \forall k, i, j \quad (2)$$

* Resource classes ($R_{Cl}^s$) have a priority;

* Resource sharing is allowed between all resource classes; and

* The resources shared are limited by the amount of resources defined for the resource class ($R_{CT}$).

In addition to the resource class capacity and allocation control, the ATCS model configuration and operation includes a resource class priority for all links $l_{ij}$ with $i, j = 1, 2, ..., n$ as follows:

$$R_{Cl}^0 > R_{Cl}^1 > R_{Cl}^2 > \dots > R_{Cl}^n \quad (3)$$

In the ATCS convention, resource class 0 priority ($R_{CL}^0$) has the highest value, and resource class n priority ($R_{CL}^n$) has the lowest value.

<table><caption>TABLE I
ATCS dynamics allocating and sharing resources</caption><thead><tr><th></th><th>Constraint</th><th>Request</th><th>Resource Class</th><th>Resource</th></tr></thead><tbody><tr><td>R<sub>CT</sub><sup>2</sup><sub>ij</sub></td><td>20 Mbps</td><td>Request 3</td><td>R<sub>Cl</sub><sup>2</sup></td><td>10 Mbps</td></tr><tr><td></td><td></td><td>Request 5</td><td>R<sub>Cl</sub><sup>2</sup></td><td>20 Mbps</td></tr><tr><td>R<sub>CT</sub><sup>1</sup><sub>ij</sub></td><td>50 Mbps</td><td>Request 2</td><td>R<sub>Cl</sub><sup>1</sup></td><td>30 Mbps</td></tr><tr><td></td><td></td><td>Request 4</td><td>R<sub>Cl</sub><sup>0</sup></td><td>30 Mbps</td></tr><tr><td>R<sub>CT</sub><sup>0</sup><sub>ij</sub></td><td>30 Mbps</td><td>Request 1</td><td>R<sub>Cl</sub><sup>0</sup></td><td>20 Mbps</td></tr></tbody></table>

The resource class priority is used by the ATCS to:

* Allocate unused resources from other resource classes; and

* Preempt or return shared resources previously allocated from other resource classes.

ATCS policy allows each resource class $R_{CL}^k$ to have two resource partitions: private and public. The private resource partition or pool is, by definition, exclusively allocated for $R_{CL}^k$ users, and the public resources may be allocated by other resource classes when not used by $R_{CL}^k$ users. As such for all $l_{ij}$ with $i, j = 1, 2, ..., n$:

$$R_{Cl}^k R_{ij} = RCl_{pri}^k R_{ij} + RCl_{pub}^k R_{ij} \quad (4)$$

Where:

* $R_{Cl}^k R_{ij}$ is the amount of resources available for $R_{Cl}^k$ at node $n_j$ for node $n_i$

In case private and public resource pools are considered in the ATCS operation, the resource class used resources ($R_{Cl\_UR}$) matrix must then be rewritten as follows for all $l_{ij}$ with $i, j = 1, 2, ..., n$:

$$R_{Cl}^kUR_{ij} = [R_{Cl}^k_{pri}UR_{ij} + R_{Cl}^k_{pub}UR_{ij} + \sum_{z=1}^{n} R_{Cl}^k_{pub}UR_{ij}] \forall z \ne k \quad (5)$$
Equation 5 describes the ATCS operation when allocating and sharing resources among resource classes. In summary, the ATCS model does the following:

* It allocates private resources to users belonging to a resource class $R\_Cl^k$;

* It shares unused public resources belonging to this class ($R\_Cl^k$) with users belonging to other resource classes independently of the resource class priority, and

* It allocates public available resources from other resource classes ($R\_Cl^z$ with $z \neq k$).

Figure 3 summarizes and illustrates the ATCS operation for private and public resource allocation and all resource classes.

ATCS users are mapped to resource classes $\sum_{k=1}^{n} R\_Cl^k$ (Figure 2). As such, for all nodes $n_j$ connected to node $n_i$ through links $l_{ij}$ with $i, j = 1, 2, ..., n$, there are sets of users $\sum_{1=1}^{n} U^i$ mapped or grouped in resource classes $R\_Cl^k$.

In terms of the ATCS operation in a graph path with a set of nodes for each resource class $R\_Cl^k$, there is a User Allocated Resource matrix (UALR) with allocated resources requested by nodes $n_i$ to node $n_j$ through links $l_{ij}$ with $i, j = 1, 2, ..., n$ (Equation 6)

$$ UALR = \left[ \sum_{y=1}^{m} U^y \sum_{x=1}^{n} ALR^x \right] \forall R\_Cl^k, k = 0, 1, \dots, n. \quad (6) $$

The ATCS model, in summary, allocates and shares resources among the users $U^y$ belonging to a resource class $R\_Cl^k$. The operation is such that unused resources for that class are shared until they are exhausted. This operational characteristic of the ATCS model is analytically modeled by equations 5 and 6 and illustrated by Figure 3.

## VI. ATCS Dynamics Use Case Simulation

The dynamics of the ATCS model operation (equations 5 and 6 and Figure 3) may also be perceived by a case of use simulation scenario in which the resources available for a class are allocated until exhaustion.

The case of use simulation scenario is for the allocation of the resource bandwidth in a node, and the ATCS model configuration parameters are as follows:

* ATCS model operating at node $n_j$ receiving requests from node $n_i$ (Figure 3);

* Total amount of resources allocated and shared $NRC_{ij}$ is 100 Mbps;

* Three resource classes ($R\_Cl^0$, $R\_Cl^1$, and $R\_Cl^2$);

* Resource class constraints are: $R\_CT_{ij}^2 = 20$ Mbps, $R\_CT_{ij}^1 = 50$ Mbps, and $R\_CT_{ij}^0 = 30$ Mbps;

In this didactic simulation scenario, users of different resource classes request resources sequentially from node $n_i$ to node $n_j$ as follows:

* Request 1 - $R\_Cl^0$ - $R_{ij}R\_Cl^0N^j = 20$ Mbps;

* Request 2 - $R\_Cl^1$ - $R_{ij}R\_Cl^1N^j = 30$ Mbps;

* Request 3 - $R\_Cl^2$ - $R_{ij}R\_Cl^2N^j = 10$ Mbps;

* Request 4 - $R\_Cl^0$ - $R_{ij}R\_Cl^0N^j = 30$ Mbps;

* Request 5 - $R\_Cl^2$ - $R_{ij}R\_Cl^2N^j = 20$ Mbps;

* Request 6 - $R\_Cl^1$ - $R_{ij}R\_Cl^1N^j = 20$ Mbps;

* Request 7 - $R\_Cl^0$ - $R_{ij}R\_Cl^0N^j = 10$ Mbps;

* Request 8 - $R\_Cl^2$ - $R_{ij}R\_Cl^2N^j = 10$ Mbps;

* Request 9 - $R\_Cl^n$ - $R_{ij}R\_Cl^nN^j = \text{any amount of resources}$;

As illustrated in Table I, ATCS grants request 1, 2 and 3, allocating resources available at their respective resource classes (green highlight).

Request 4, belonging to resource class 0, extrapolates the class 0 capacity, and ATCS shares unused resources from resource class 1 (orange highlight). ATCS executes a similar operation with request 5 belonging to resource class 2 and shares unused resources from class 1 to grant the user request.

All following requests (6, 7, and 8) can not be granted anymore by ATCS using private or unused resources from any class. In effect, private and unused resources are exhausted at this point.

This stage of the ATCS model execution is relevant concerning utilizing BAM models to contribute to reinforcement learning agent design and optimization.

There are two alternatives to consider at this execution point of the ATCS model in terms of the objectives and constraints of the resource allocation problem:

* The resource allocation objective to be attained has a constraint not allowing allocated resources to be revoked from users (preemption or resource devolution); or

* The resource allocation objective allows allocated resources to be revoked from users.

If the first alternative prevails, the ATCS model must be managed, and additional resources are configured for the resource classes. This typically requires some management knowledge about the system's overall behavior. This task can be executed by an intelligent reinforcement learning agent or human manager.

If the second alternative prevails, the ATCS model preemption or devolution of resources is executed according to the resource class priority [21] [6].

As an illustrative example of ATCS dynamic operation, request 6 from resource class 1 would imply preempting and returning resources used by request 4 and request 5 since they use resources that were shared because were unused but effectively belong to resource class 1. The following requests, 7 and 8, would be granted since resource classes 0 and 2 have unused resources at this phase of the resource allocation process.

As a final consideration, the arrival of request 9 from any class requesting any amount of resources would be denied since the maximum amount of resources for all classes is exhausted. There are no more resources to allocate or
<figure><img src="image_5.png" /><figcaption>Fig. 4. ATCS Offloading the RL Agent Resource Allocation Task.</figcaption></figure>

share whatsoever, and, as such, granting new resources to users imply in reconfiguring the ATCS parameters.

## VII. ATCS Assisting the Design and Contributing to Optimizing the Reinforcement Learning Agent Operation

A reinforcement learning agent design involves agent modeling and agent training. The RL agent training and performance depend significantly on the state-space model of the agent.

The issue addressed in this paper is whether the ATCS model contributes to the design and operational efficiency of an RL agent whose objective is to allocate resources among users.

We first argue that the ATCS model mimics an RL agent operation with similar allocating and sharing resources objectives. Consequently, the RL agent state-space is reduced since the BAM model does nearly all resource and sharing allocations until resources are exhausted. BAM operation, as such, facilitates the agent design and contributes to the optimization of its operation.

The ATCS model operation (including preemption and devolution) allocating and sharing resources is similar to an RL agent allocating resources when the following objectives are defined for both software:

* ATCS and RL agent allocate resources to users based on resource priority;

* ATCS and RL agent allocate unused resources to all users; and

* ATCS and RL agent preempt allocated resources upon requests from high-priority resources or resource owners.

Equations 5 and 6 of the analytical model demonstrate and validate the ATCS model mimicking the operation of an RL agent to allocate and share resources. The ATCS mimics the RL agent until:

* Resources are exhausted for all ATCS resource classes; and

* Unused resources among classes are exhausted, and preemption and devolution are not allowed in the problem definition and requirements.

Based on this conclusion, the ATCS offloads and assists the RL agent computation of the resource allocation process, as proposed in Figure 4.

In the Figure 4 setup, all resource-allocating computation is offloaded from the RL agent, and the agent acts on behalf of the ATCS model exclusively to manage its configuration. It is worth mentioning that the ATCS model management is a high-level task that requires knowledge acquisition and operation expertise in alignment with the purpose of an RL agent.

As such, the RL agent is invoked to manage the ATCS configuration parameters in a resource-exhausted condition. The RL agent is called because the ATCS model has no possibility whatsoever to allocate resources to users anymore when this condition is reached.

The RL agent design with ATCS assistance has the following benefits:

The RL agent allocating resources task offload inherently reduces the state-space model for the agent. That is so because it is invoked only to reconfigure ATCS parameters, and the agent no longer executes all computation and state analysis concerning user resource requests. The same reasoning applies to the agent modeling and training phases.

## A. The Computational Tradeoff between Agent Invocation and ATCS Algorithm Execution

The reinforcement learning agent state-space reduction has a cost that is the deployment and processing overhead of the BAM algorithm (ATCS or any other).

In terms of computational complexity, the BAM software is inherently less complex than the RL agent that executes extensive state analysis and matrix processing consuming significant computational resources.
## VIII. Final Considerations

Reinforcement learning agents require advancements in terms of their design and efficiency. This paper proposes to use the ATCS BAM model integrated as part of an RL agent deployment to facilitate its design and contribute to the agent's efficiency.

The analytical model and the use case simulation presented demonstrate that an ATCS software offloads part of the computational tasks of a reinforcement learning agent aiming to allocate and share prioritized resources for users.

The RL agent task offloading is achieved by the ATCS model executing most of the allocation and sharing of resources. This approach results in the RL agent acting as a manager, adjusting the ATCS configuration parameters when resources are exhausted and need an intelligent management decision.

The proposed approach facilitates the agent design process, reduces the state-space of the agent, and, consequently, the training process involved.

Future work will include evaluating numerically the deployment and processing cost of the BAM algorithm against the cost reduction in modeling and training the RL agent in a reduced state-space.

## Acknowledgment

The authors thank the ANIMA Institute, Salvador University (UNIFACS), and the Federal Institute of Bahia (IFBA) for supporting this work.

## References

[1] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.

[2] Wen Wu, Conghao Zhou, Mushu Li, Huaqing Wu, Haibo Zhou, Ning Zhang, Xuemin Sherman Shen, and Weihua Zhuang. AI-Native Network Slicing for 6G Networks. IEEE Wireless Communications, 29(1):96-103, February 2022.

[3] Rakesh Kumar, Mayank Swarnkar, Gaurav Singal, and Neeraj Kumar. IoT Network Traffic Classification Using Machine Learning Algorithms: An Experimental Analysis. IEEE Internet of Things Journal, 9(2):989-1008, January 2022.

[4] Mahbuba Afrin, Jiong Jin, Akhlaqur Rahman, Ashfaqur Rahman, Jiafu Wan, and Ekram Hossain. Resource Allocation and Service Provisioning in Multi-Agent Cloud Robotics: A Comprehensive Survey. IEEE Communications Surveys Tutorials, 23(2):842-870, 2021. Conference Name: IEEE Communications Surveys Tutorials.

[5] Baofeng Ji, Xueru Zhang, Shahid Mumtaz, Congzheng Han, Chunguo Li, Hong Wen, and Dan Wang. Survey on the Internet of Vehicles: Network Architectures and Applications. IEEE Communications Standards Magazine, 4(1):34-41, March 2020.

[6] Joberto S. B. Martins, Romildo Bezerra, Rafael Reale, and Gilvan Durães. Uma Visão Tutorial dos Modelos de Alocação de Banda como Mecanismo de Provisionamento de Recursos em Redes IP/MPLS. Revista de Sistemas e Computação, 5(2):144-155, December 2015.

[7] André L. C. Ottoni, Erivelton G. Nepomuceno, Marcos S. de Oliveira, and Daniela C. R. de Oliveira. Tuning of Reinforcement Learning Parameters Applied to SOP Using the Scott-Knott Method. Soft Computing, 24(6):4441-4453, March 2020.

[8] Esther Wong. State-space decomposition for Reinforcement Learning. project, Imperial College London, London, 2021.

[9] Meng Zhao and Hajime Nobuhara. A Fuzzy Logic Based State-space Reducing Method in Soccer Agent Reinforcement Learning. Scis & Isis, 2010:76-81, 2010.

[10] Hao Jiang, Renjie Gui, Zhen Chen, Liang Wu, Jian Dang, and Jie Zhou. An Improved Sarsa (λ) Reinforcement Learning Algorithm for Wireless Communication Systems. IEEE Access, 7:115418-115427, 2019.

[11] David Ha. Reinforcement Learning for Improving Agent Design. Artificial Life, 25(4):352-365, November 2019.

[12] Thanh Thi Nguyen, Ngoc Duy Nguyen, Peter Vamplew, Saeid Nahavandi, Richard Dazeley, and Chee Peng Lim. A Multi-Objective Deep Reinforcement Learning Framework. Engineering Applications of Artificial Intelligence, 96:103915, November 2020.

[13] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. In Kyriakos G. Vamvoudakis, Yan Wan, Frank L. Lewis, and Derya Cansever, editors, Handbook of Reinforcement Learning and Control, Studies in Systems, Decision and Control, pages 321-384. Springer International Publishing, Cham, 2021.

[14] Shariq Iqbal and Fei Sha. Actor-Attention-Critic for Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, pages 2961-2970. PMLR, May 2019. ISSN: 2640-3498.

[15] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A Survey on Federated Learning. Knowledge-Based Systems, 216:106775, March 2021.

[16] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated Learning in Mobile Edge Networks: A Comprehensive Survey. IEEE Communications Surveys & Tutorials, 22(3):2031-2063, 2020. Conference Name: IEEE Communications Surveys & Tutorials.

[17] Douglas C. Montgomery. Design and Analysis of Experiments. Wiley, 9th edition edition, March 2017.

[18] Lars Stahle and Svante Wold. Analysis of Variance (ANOVA). Chemometrics and Intelligent Laboratory Systems, 6(4):259-272, November 1989.

[19] F L Faucher and W Lai. Maximum Allocation Bandwidth Constraints Model for DiffServ-aware MPLS Traffic Engineering. Request for Comments RFC 4125, Internet Engineering Task Force - IETF, June 2005.

[20] Walter da Costa Pinto Neto and Joberto S. B. Martins. Adapt-RDM - A Bandwidth Management Algorithm Suitable for DiffServ Services Aware Traffic Engineering. In IEEE Network Operations and Management Symposium (NOMS), pages 975-978, April 2008. ISSN: 2374-9709.

[21] Rafael F. Reale, Walter da C. P. Neto, and Joberto S. B. Martins. AllocTC-sharing: A new bandwidth allocation model for DS-TE networks. In 2011 7th Latin American Network Operations and Management Symposium, pages 1-4, October 2011.

[22] Gilvan Durães, Rafael Reale, Romildo Bezerra, Alexandre Fontinele, André Soares, and Joberto S. B. Martins. Evaluating the Applicability of Bandwidth Allocation Models for EON Slot Allocation. pages 1-6, December 2017.

[23] Ahmed El-mekkawi, Xavier Hesselbach, and Jose Ramon Piney. Squatting and Kicking Model Evaluation for Prioritized Sliced Resource Management. Computer Networks, 167:107006, February 2020.

[24] E. M. Oliveira, R. F. Reale, and J. S. B. Martins. Cognitive Management of Bandwidth Allocation Models with Case-Based Reasoning - Evidences Towards Dynamic BAM Reconfiguration. In 2018 IEEE Symposium on Computers and Communications (ISCC), pages 00397-00403, June 2018. ISSN: 1530-1346.