# Decision-Time Learning and Planning Integrated Control for the Mild Hyperbaric Chamber
## 3 Model Learning
### Decision-Time Learning with State Context Representation  
We consider the state transition of the system to be generated by the following parametric MDP model:  
\[s_{t+1}=d(s_{t},a_{t},o)+\epsilon_{t} \tag{1}\]  
where \(s_{t}\in\mathcal{S},a_{t}\in\mathcal{A}\), and \(\epsilon_{t}\sim\mathcal{N}(0,Q_{t})\). Here, \(Q_{t}\) is the diagonal covariance matrix and \(o\) represents unobserved system parameters that actually influence the system state. The process can be treated as a Partially Observable Markov Decision Process (POMDP) [21]. We define \(\mathcal{T}=(s_{t},a_{t},s_{t+1},\dots,\)\(a_{t+T-1},s_{t+T})\) as trajectory segments of length \(T\), and \(\mathcal{C}=\{(s,a,s^{+})\}\)as a set of system state representations, where \(s^{+}=d(s,a,o)+\epsilon\). The objective of model learning is to determine a probabilistic context representation model \(d_{f}(s_{t})\)\(s_{0:t-1},a_{0:t-1},\mathcal{C})\), which maximizes the expected log-likelihood function:\[\max_{\theta}\mathbb{E}\log p_{\theta}(\mathcal{T}\mid\mathcal{C}) \tag{2}\]  
where \(\theta\) denotes the parameters of the neural network. As we cannot directly observe \(o\), we introduce a latent variable \(u\) to encode the context representation through learning:  
\[\log p_{\theta}(\mathcal{T}\mid\mathcal{C})=\log\int p_{\theta}(\mathcal{T} \mid u)p_{\theta}(u\mid\mathcal{C})du \tag{3}\]  
We use a neural network model \(d_{f}(s_{t+H}\mid\hat{s}_{t},a_{t:t+H-1},u)\) to model \(p_{\theta}(\mathcal{T}\mid u)\) and apply a context representation encoder \(d_{c}(u\mid\mathcal{C})\) to model \(p_{\theta}(u\mid\mathcal{C})\). We construct an encoder to process the input sequence. Given a series of states \(s_{0:T}\), actions \(a_{0:T-1}\), and the latent context representation variable \(u\), the likelihood estimation of the predictive model \(d_{f}\) is as follows:  
\[ \begin{split} p_{\theta}(\mathcal{T}\mid u)\approx\prod_{t=0}^{T }d_{f}(\hat{s}_{t}=s_{t}\mid s_{t})\cdot\prod_{t=1}^{T}d_{f}\Big{(}\hat{s}_{i \mid t-1}=s_{t}\mid s_{t-1},a_{t-1},u\Big{)}\\ \cdot\prod_{t=2}^{T}d_{f}\Big{(}\hat{s}_{i\mid 0}=s_{t}\mid s_{0}, a_{0:t-1},u\Big{)}\end{split}  \tag{4}\]  
The three terms in the above equation correspond to state reconstruction, single-step prediction, and multi-step prediction likelihood functions, respectively. The transition model \(d_{f}(\hat{s}_{t+H}\mid\hat{s}_{t},a_{t:t+H-1},u)\) is computed using the following equation:  
\[\hat{s}_{t+H}=h(s_{t},a_{t},\ldots,a_{t+H-1},u)+\epsilon \tag{5}\]  
where \(\epsilon\) is additive Gaussian noise with a mean of zero. A Recurrent Neural Network (RNN) is a neural network architecture specifically designed to process sequential data, that features parameter sharing along the time dimension [22]. We use Gated Recurrent Units (GRUs) as a special type of RNN to construct the latent environment model, where \(h_{s},h_{a}\), and \(h_{u}\) encode the state \(s_{t}\), action \(a_{t}\), and latent context representation vector \(u\) into the RNN loop:  
\[ \begin{split} z_{t}=h_{s}(s_{t})\\ z_{t+1}=h_{\text{CRU}}(z_{t},[h_{a}(a_{t}),h_{u}(u)])\end{split}  \tag{6}\]  
where \([\cdot,\cdot]\) denotes vector concatenation.