## 5 Proposed mMTC Device Grouping Algorithms
### Reinforcement Learning-Based Device Grouping Algorithm
#### 5.2.1 Constrained Optimization with Policy Gradients  
The Reinforcement Learning Framework  
As previously mentioned, the problem is reformulated as a constrained Markov decision process (MDP), where the interaction between the agent and the environment is defined as follows:  
**Environment:** This represents the network configuration with the total number of devices, their specific features, and their weight (how many tasks they have to perform and what these tasks are). It also holds the device's status (previously grouped devices) and the \"yet to be grouped\" devices. These devices can perform an RACH if no grouping action is intended. So, each MTD behaves independently during the initial time, and no group exists.  
**Agent**: For each instance \(t\) until the time horizon H, it generates the corresponding grouping vector (_action_), indicating in which group \(G\) the device \(n\) should be placed. The environment then evaluates the grouping decision and, using (1), computes the quality of the action (_reward_). The whole process proceeds iteratively until all possible combinations are explored. The overall reward value determines the best grouping strategy. The time frame is defined as a set of time steps \(t\) with \(t\in[1,\) H\(]\).  
**State:** The state contains the status of the MTDs, applications, and groups, such as the grouping status of a device \(\hat{\mathbf{N}}\), the situation of the groups \(\hat{\mathbf{G}}\), the specific situation of an application \(\hat{\mathbf{M}}\) (arrival time, time constraints), the task status for each application \(\hat{\mathbf{K}_{\mathbf{M}}}\), the task status for each device \(\hat{\mathbf{K}_{N}}\), and the application status for each device \(\hat{\mathbf{M}_{\mathbf{N}}}\). \(S  
In summary, the expression \(s_{t}=[\hat{N},\hat{G},\hat{M},K_{M},K_{N},\hat{M}_{N}]\) represents a sequence of elements within the state vector \(s_{t}\) of the model. Each element in the sequence corresponds to a specific variable or parameter within the state vector, defining the relation between the elements.  
**Action:** As seen in Figure 4, the action set At is defined as the set grouping decision in which an MTD is assigned to a group. Hence, it determines the assignment action of MTD \(n\) to group \(G\) at time step \(t\). During eachstate, a grouping action is feasible if one or more groups have enough space to fit. Now, the decision \(d\) involves choosing among all the possibilities to decide which group to assign the device to because each device may have different task/application features and different space/size requirements. Keep in mind that our goal is to optimize the device placement decision. Accordingly, the action at time t involves a set of grouping decisions with \(z\) as the total number of possibilities:  
\[\mathbf{a}_{t}=[d_{1},d_{t},d_{3},...,d_{z}]. \tag{25}\]  
**Reward:** The reward set \(R_{t}\) determines the set of feedbacks following the actions \(A_{t}\). It aims to incentivize a good action. Hence, \(rt\) is the incentive awarded to the agent for performing the action that consists of assigning the \(MTD_{n}\) to a group G in the best possible way [51] with less free space in the group, as per our objective).  
RNN Architecture  
The proposed neural architecture is a sequence-to-sequence (seq2 seq) model based on an encoder-decoder structure. The decoder is built as an attention-based LTSM model, which outputs the group to which the device considered as an encoder input will be assigned.  
The model trains a multi-stacked LSTM cell to act as a recurrent neural network (RNN) agent capable of embedding information from the environment and variable-length sequences batched from the entire combination's input space. The rewards are used to optimize the parameter for future grouping actions using a stochastic gradient descent policy.