# Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
## 3 Methodology
### FRL Applications of IoT in the Vehicular Industry
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Number** & **Work** & **Domain** & **Technology** & **Goal** & **Year** \\ \hline
1 & [PERSON] et al. [109] & \begin{tabular}{c} Multi-energy \\ systems (MES) \\ \end{tabular} & \
\begin{tabular}{c} Deep deterministic policy gradient \\ and abstract critic network \\ \end{tabular} &
\begin{tabular}{c} Method to address the joint P2P energy and \\ carbon trading (JPC) problem in a local community \\ \end{tabular} & 2023 \\ \hline
2 & [PERSON] et al. [110] & \
\begin{tabular}{c} IoT traffic \\ management \\ \end{tabular} & Advantage-actor-critic &
\begin{tabular}{c} Propose an edge-based backhaul selection to improve \\ traffic delivery based on multi-objective feedback \\ \end{tabular} & 2022 \\ \hline
3 & [PERSON] et al. [111] & \begin{tabular}{c} Offloading \\ optimization \\ \end{tabular} & \
\begin{tabular}{c} Proximal policy \\ optimization (PPO) \\ \end{tabular} &
\begin{tabular}{c} Adaptive offloading FL framework \\ to tackle efficiency challenges \\ \end{tabular} & 2022 \\ \hline
4 & [PERSON] et al. [112] & \begin{tabular}{c} Offloading \\ optimization \\ \end{tabular} & \
\begin{tabular}{c} Federated DRL-based online \\ task offloading and resource allocation (FDOR) \\ \end{tabular} &
\begin{tabular}{c} Online task offloading and resource \\ allocation in WPC-MEC Networks \\ \end{tabular} & 2022 \\ \hline
5 & [PERSON] et al. [113] & \begin{tabular}{c} Offloading \\ optimization \\ \end{tabular} & \
\begin{tabular}{c} Double deep Q-learning \\ (DDQN) \\ \end{tabular} &
\begin{tabular}{c} FL and DRL to optimize \\ IoT computation offloading \\ \end{tabular} & 2019 \\ \hline
6 & [PERSON] et al. [114] & \begin{tabular}{c} Offloading \\ optimization \\ \end{tabular} & \
\begin{tabular}{c} Deep deterministic \\ policy gradient (DDPG) \\ \end{tabular} &
\begin{tabular}{c} Task offloading \\ and resource allocation \\ \end{tabular} & 2022 \\ \hline
7 & [PERSON] et al. [116] & \
\begin{tabular}{c} Resource \\ allocation \\ \end{tabular} & Deep Q-network (DQN) &
\begin{tabular}{c} FRL-based resource allocation \\ in D2D-enabled 6G \\ \end{tabular} & 2022 \\ \hline
8 & [PERSON] et al. [117] & \
\begin{tabular}{c} Resource \\ allocation \\ \end{tabular} & Deep Q-network (DQN) &
\begin{tabular}{c} Resource allocation \\ in IoT edge computing \\ \end{tabular} & 2022 \\ \hline
9 & [PERSON] et al. [118] & \
\begin{tabular}{c} Resource \\ allocation \\ and double deep Q-network (DDQN) \\ \end{tabular} &
\begin{tabular}{c} Resource allocation in \\ mobility-aware networks \\ \end{tabular} & 2020 \\ \hline
10 & [PERSON] et al. [115] & \begin{tabular}{c} Offloading \\ optimization \\ \end{tabular} & \
\begin{tabular}{c} Double deep Q-network \\ (DDQN) \\ \end{tabular} &
\begin{tabular}{c} Delay and energy minimization \\ in IoT networks \\ \end{tabular} & 2021 \\ \hline
11 & [PERSON] et al. [119] & \begin{tabular}{c} Energy \\ management \\ \end{tabular} & \
\begin{tabular}{c} Adapter multi-armed bandit \\ (MAB) algorithm \\ \end{tabular} &
\begin{tabular}{c} Device scheduling for renewable \\ energy-powered federated learning \\ \end{tabular} & 2022 \\ \hline
12 & [PERSON] et al. [120] & \begin{tabular}{c} Energy \\ management \\ \end{tabular} & \
\begin{tabular}{c} Personalized deep federated \\ reinforcement (PDRL) \\ \end{tabular} &
\begin{tabular}{c} Residential energy management system to tackle the \\ standby energy reduction in residential buildings \\ \end{tabular} & 2022 \\ \hline
13 & [PERSON] et al. [121] & \begin{tabular}{c} Caching \\ \end{tabular} & \
\begin{tabular}{c} Double deep Q-network \\ (DDQN) \\ \end{tabular} &
\begin{tabular}{c} Federated DRL-based cooperative edge caching (FADE) \\ framework to enable collaborative learning of \\ shared predictive models \\ \end{tabular} & 2020 \\ \hline
14 & [PERSON] et al. [122] & \
\begin{tabular}{c} Hierarchical federated \\ deep reinforcement learning \\ (HPDRL) \\ \end{tabular} &
\begin{tabular}{c} Prediction of user’s future requests to determine \\ appropriate content replacement strategies \\ \end{tabular} & 2021 \\ \hline
15 & [PERSON] et al. [123] & \begin{tabular}{c} Service \\ provision \\ \end{tabular} & \
\begin{tabular}{c} Actor-critic (AC) \\ reinforcement learning (RL) \\ \end{tabular} &
\begin{tabular}{c} Propose an RL-based request \\ service provisioning system \\ \end{tabular} & 2022 \\ \hline \hline \end{tabular}
\end{table}