Identifying Latent State-Transition Processes for Individualized Reinforcement Learning
Authors: Yuewen Sun
Biwei Huang
Yu Yao
Donghuo Zeng
Xinshuai Dong
Songyao Jin
Boyang Sun
Roberto Legaspi
Kazushi Ikeda
Peter Spirtes
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Identifiability
Individualization
Primary area: reinforcement_learning

Abstract: The application of reinforcement learning (RL) involving interactions with individuals has grown significantly in recent years. These interactions, influenced by factors such as personal preferences and physiological differences, causally influence state transitions, ranging from health conditions in healthcare to learning progress in education. As a result, different individuals may exhibit different state-transition processes. Understanding individualized state-transition processes is essential for optimizing individualized policies. In practice, however, identifying these state-transition processes is challenging, as individual-specific factors often remain latent. In this paper, we establish the identifiability of these latent factors and introduce a practical method that effectively learns these processes from observed state-action trajectories. Experiments on various datasets show that the proposed method can effectively identify latent state-transition processes and facilitate the learning of individualized RL policies.

Identifying Latent State-Transite Processes for Individualized Reinforcement Learning
Yuewen Sun 1.2, Biwei Huang Yu Yao", Donghuo Zeng", Xinshuai Dong?, Songyao Jin
Boyang Sun', Roberto Legaspi", Kazushi Ikeda,, Peter Spirtes", Kun Zhang Mohamed bin Zayed University of Artificial Intelligence, Carnegie Mellon University, University of California San Diego, "The University of Sydney, KDDD Research
Abstract
The application of reinforcement learning (RL) involving interactions with indi- viduals has grown significantly in recent years. These interactions, influenced by factors such as personal preferences and physiological differences, causally influence state transitions, ranging from health conditions healtheare learning progress in education. As a result, different individuals may exhibit different transitioii processes. Understanding individualized state-transition processes is essential for optimizing individualized policies. In practice, however, identify- factors ing these trae-trannition processes is challenging. as individual-speciin often remain latent. In this paper, we establish the identifiability of these latent factors and introduce a practical method that effectively learns these processes from observed state-action trajectories. Experiments on various datasets show that the proposed method can effectively identify latent state- transition processes and facilitate the learning of individualized RL policies.
Introduction
Reinforcement Learning (RL) 44
involves training agents to make decisions by interacting with the state with
environment. The agent observes its current state, takes action, and transitions a reward. Such a sequence of moving from one state to another is known as a state-transition process.
Individualized RL focuses adapting the policy for each individual. It has recently seen increas- ing application in various sectors, including healthcare [86] [5 19 education [66 12), and e-commerce [51 884 I.. The setividuualsppecfff. factors (58), which capture the unique characteristics of each individual, play an important role in causally influencing the transitions between states. These factors cover range of aspects, including individual preferences, past experiences, and physiolog- ical differences. For example, in education, different learning styles can influence how students with the same prior knowledge benefit from a tutorial. In healthcare, genetic differences can affect how patients with hypertension respond to the same treatments. Understanding individual-specine factors is essential for designing better RL systems that provide more individualized and effective decisions [39] 26 3 (63]. By understanding learning styles in education, RL agents can recom- mend personalized tutorials, such as animated content for visual learners or hands-on exercises for kinesthetic learners. Similarly, in healthcare, knowledge of genetic makeup can help agents suggest treatment plans tailored to individual needs, leading to improved health outcomes.
However, these individual-see -peciffc factors are not always observable, making it challenging to understand the Latent Individualized Stane-Transition (LIST) processes, illustrated in Figure ((a) These latent factors, such as learning styles and genetic makeup, are unique to each individual and have time-invariant influence the state-transition process. This raises the question: can the identifiability of these latent factors be guaranteed?
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

6.
9
(a) IMDP for individual m.
(b) Latent causal processes.
(c) Factored nonstationary MDP.
Figure Comparisons of different transitinnition processes. Latent variables are colored grey.
Such identifiability is easier achieve when the observations are either i.i.d., i.i.d. given side information (e.g., domain index, time index, etc.), by exploiting sparsity [94]. variability [45]. or functional complexity [41] To the best of our knowledge, only a few studies have explored the identifiability of latent factors from temporal observations, These methods primarily focus on the vime-narying latent factors, which differs from our work on lime-invariant latent factors. Specifically, assume time-varying latent variables without considering the influence of actions the generative process (see Figure T(b). Factored MDPs [3] incorporate actions into the process but still assume that the latent factors change over time (see Figure I
existing works 83 82 6
Thus, the findings from these studies cannot be applied to our setting. Intuitively, this is because time-invariant latent factors cannot provide the variability that many current methods rely on to achieve identifiability. Therefore, remains unclear how to derive the identifiability the latent individual-peecficc factors and the corresponding latent state-transe tnte-ttransition processes from observed states and actions.
Recent advances in finite mixture models 74 65 have demonstrated strong identifiability results by exploiting group information in nonparametric settings. By assuming that observations within the same group are known to come from the same component, the mixture of probability measures can be uniquely identified under appropriate assumptions. Inspired by these works, we establish the identifiability of latent factors by leveraging group information from the data, making it easier to distinguish different underlying components. We propose both finite latent, nonparametric setting and an infinite latent, parametric setting, and develop a theoretically grounded framework that effectively learns these processes from observed state-action trajectories. Our contributions are summarized as follows:
We introduce Individualized Markov Decision Processes (iMDPs), novel framework that incorpo- rates latent individual-speciff factors к into mate-transition processes. We treat such latent factors that influence each state in the decision process and vary across individuals.
Our work provides theoretical guarantees and new insights into learning state- tnte-trransiion processes with latent factors. For finite K, we consider two scenarios to establish identifiability in nonpara- metric settings. For infinite best our knowledge, this is the first work to provide a theoretical guarantee for identifying latent individual-specific factors from observed transitions.
demonstrate identifiability in the post-non post-nonlinear case. To the
We propose practical generative method that can effectively estimate the latent individual-speciff factors. Empirical results on various datasets demonstrate the effectiveness of the method both in inferring latent factors and learning individualized policies.
2 Related Work
Individualized Leabinng Applications Recently, machine learning has created highly individualized solutions across various domains. healthcare, algorithms support individualized treatment for physical activity, weight loss, and diabetes management 86
In finance, it provides accurate stock predictions for stock market activities [54]. Education is benefiting
35 16 15]
from individualized ICT systems that address the individual learning needs of students [14] 38
Furthermore, transportation has seen the development of individualized car-following strategies that improve driving safety and efficiency. Meanwhile, entertainment platforms such as YouTube and Tok are using it to provide individualized video recommendations [5] 3T.
Reinforcement Learning with Latent State- Transition Processes In the field of RL, various models explore the state transition dynamies with latent variables. One such approach Partially Observable Markov Decision Processes (POMDPs) (62). where the full information about the state is

---

unknown. In POMDPs, observations are generated from the latent states, which do not match our individual latent setting. For example, block MDPs 89 93 assume that there a fixed and unknown mapping from observations to the latent states. Factored MDPs 33 13 which provide the partial identifiability of latent factors, assume that the latent factors evolve over time following a Markov process. On the other hand, there exists piece of work focusing on estimating state transitions with time-invariant latent factors. Models such as contextual MDPs [27 58 63,, latent MDPs 44 48) and multitask RL [71 (22] consider similar scenarios with our latent andividual-specifii factors. However, these works lack theoretical guarantees on the identifiability of the latent factors thus it is hard for them to guarantee individualized decision-making
3 Problem Formulation
Consider a population with M individuals that can be divided into G groups, where the exact group memberships are unknown. We introduce iMDPs to model individualized decision- processes, where observed individual uniqueness captured by u, and latent group-level properties are encoded by K. Specifically, each individual is assigned unique value of u, with the cardinality of u being M. Meanwhile, individuals within the same group share the same value of x, which differs across groups, and the cardinality of x is G. For each individual, the value of is predetermined and has time-invariant influence on the state- sateetrrnsition process. Furthermore, all individuals are assumed to share the same state and action spaces. The iMDP is defined as follows.
1M Definition 3.1 (iMDP). An iMDP consists of a tuple (S,A, R. {sō MM {um 1m=1: {Tm 1M I==1), where M is the number of individuals; S and A are the state and action spaces, respectively; R E R is the immediate reward received after transitioning from current state s to new state s' via action i.e., r = Ra (s, s') for E S, s' E S and A. To model individualized dacision-making each individual is associated with unique index and an individualized state-transition process. In the with MDP, umn is the unique index identifying each individual, and 8 is the individualized initial state. The individualized state-transition probability is denoted by Tm := Pm (s') [8, O, km)) E RR|Ssx A|x|si, where Kmm is the latent individual-spen -spccific factor with a cardinality of G. Thus, the joint distribution of any adjacent state cate-action pairs (s, a, s') can be fully characterized by u and k as:
P(s, 0, 8 8'(u) = P(s'(s (s, a, 1)) P)) alw).
Data Generation Process
Here we introduce the LIST processes based on iMDP framework. For individual m. the observed states s are generated according to the following generation process:
8 m = [( mi a Ki, Ft m)
where s (s0, -0, 2 E Rd. denotes the dimdinensional state at time and a." (a dg. independent noise, while is characterizes the group-level properties within the population. The transition function f is identical across individuals, governs the dynamics and is consistent with Eq. [T] During interaction with the environment, the trajectory Tm = {s" al as sequence of observed state-action tuples, where T denotes the length of the trajectory.
m e) E Rd denotes the dy-dimensional action at time t. The term e," represents
sT) is recorded
Objectives In this work, we investigate RL with focus on capturing individualized latent state- transition dynamics. Our objectives are twofold: 1) to identify the latent individual-spedt factors from observed trajectories, and 2) to learn individualized policies for each agent, facilitating policy adaptation for newcomers. Consider the example of hypertension diagnosis in healthcare. Treating all patients identically may lead to different outcomes due to the dynamics of state transitions, which are influenced by latent k. Therefore, accurate identification of к from the population provides crucial dynamic insights. Once x is uncovered, we can categorize patients into different groups and tailor individualized treatments for each, which motivates our second goal.
4 Identifiability Analysis
We establish the identifiability of latent individualized state-transition processes under two conditions: (1) finite latent condition under group determinacy assumption (see Theorem 4.1 and Theorem 4.2) and (2) infinite latent condition under functional constraint (see Theorem The corresponding identifiability results are presented below.
3

---

Finite Latent Condition Suppose the value of R. is finite; we first provide the definition of group- wise identifiability. For the detailed assumptions discussion and proofs, please see Appendix Definition 4.1 (Group-wise Identifiability). Let [Tm]m=
m=1 1 be sequences of observed states and actions collected from G groups under a fixed policy, following the true latent individualized state- transition processes described in Eq. (2) learned generative model (f, R. è) is observational equivalent to (f, Ki, <) if the joint distribution i f.a.e (8, 8') matches IP f. (8, a, a,8') everywhere. We say that the latent individualized trae-trannition processes are group- wise identifiable if observational equivalence can always lead to the identifiability of latent individual -spettt factors across the population up to the invertible transformation g:
P Pj a, s') = ( -....e(s,a,85) - h== h-
(3)
Assumption 4.1 (Group Determinacy). Consider a finite mixture model Eg=1 To (1)), where Ry
represents mixing proportions with Eg=1 Fy = 1, and 8n, is the Dirac function centered at Kig. Each unique value of k. corresponds to specific group in the population, with 5x, (k) = 1 if K = Ki and O otherwise, and the number of individuals per group is greater than 2G -
Assumption 4.1 guarantees that identifiability can be derived from the finite mixture model per- spective using group information. We consider two scenarios to establish identifiability under finite latent conditions. Theorem 4.1 considers finite samples, establishing identifiability under specific assumptions about the initial states {sS) =m Theorem 4.2 guarantees asymptotic identifiability for sufficiently long trajectories without imposing constraints on the initial states.
Theorem 4.1. Assume the LIST processes described in Eq. 2 Suppose the distributions of initial states are the same for all individuals within the same groups, and the trajectory length finite. Under Assumption 4.1] the identifiaiiity of the individual-ppecte factor is guaranteed.
Theorem 4.2. Assume the LIST processes described in Eq. (2). Suppose the distribution of the initial state varies across individuals, and the trajectory length is sufficiently long, i.e., there exist two different individuals within the same group have overlap condition (Pss, uu) IP(s, a u,),
where i + j. Under Assumption 4.1 the identifiabiliyy of k. is asymptotically guaranteed.
Infinite Latent Condition Theorem 4.3 demonstrates that under specific functional constraints, the identifiability of latent individual-specinn variables can be extended to multiple and infinite latent factors. Specifically, we consider the post-nonlinear temporal model [90] and allow multiple instances of k to influence the decided by the rank conditions of specific covariance submatrices derived from the observed data. Furthermore, empirical results in Section 6 indicate that even when multiple latent factors with infinite cardinality are present, our estimation framework (see Section [5) still encourages the identification of latent factors. For a detailed proof, please see Appendix C
dynamics. The identifiability and cardinality of latent factors
Theorem 4.3. Consider trajectory collected from the post-nonlinear temporal model (Defini- ,....,T T. Let m latent factors Kj, j I,...,,.. m. have direct causal influence on all states, and S, = {s1,t, $2,t:..., represent the set of all state variables at time t. These latent factors, as well as the state-transition process, + 2..... T (m + 1), there exist pairs of minimal rank sets (Definition C.2 (A;, Bi), denoted as A₁ = R2 and B; = Ri where i <i<it,
tion C.7) with dg dy-dimensional observed states over time = I,...,
can be identified if and only for every
that satisfy:
(Rank Deficiency for Identification) In addition to S; shared by A; and B,, each subset should include randomly selected set of m additional state variables. If the covariance matrices ΣA,,, exhibit a consistent rank r (where r > d,) across all distinet indices i, this consistency implies the existence of latent factors within the system.
(Quantification of Latent Factors) Once identification is established, the number of latent factors m can be inferred from the rank deficiency of E,,, relative to the dimensionality of the observed variables, specifically given by m vank(2A,,,B) da.
5 Estimation and Policy Learning Framework
Overview We propose a two-stage approach that addresses two key objectives: (1) developing an estimation framework to recover the latent factors x from individual trajectories, and (2) implementing

---

individualized policy learning to facilitate policy adaptation for new individuals. The estimation framework is designed to meet the conditions outlined in the identifiability theorems. According to Definition 4.1 identifiability is achieved if and only if observational equivalence implies latent factor equivalence. This motivates the use of generative model estimate latent factors and ensure that the reconstructed distribution aligns closely with the true observed distribution.
To fulfill this requirement, we make a modification the variational autoencoder (VAE) architec- ture 43 which enables the unsupervised estimation of latent factors. Theorems 4.2 guarantee the asymptotic accuracy of this alignment. As illustrated in Figure 2 individual trajectories are encoded in discrete embedding space, which is consistent with the assumptions in the theorems. Detailed pseudocode for the proposed approach is provided in Appendix [] and a comprehensive description of each component is provided in Appendix
5.1 Latent Estimation Framework
Temporal Encoding and Latent Factor Quantization The group determinacy assumption sug- gests the identifiability of the latent aadividual-speciine factor K. Given that y is lime-invariant and influences each state in the transition process, we begin by employing a sequential encoder to capture the high-level representation, denoted Zm. based on the input from all states across each trajectory. We then utilize a vector quantization layer [73] to discretize the latent space and estimate the latent factor Am. This quantization step ensures that the learned representation aligns with the group-level characteristics of the latent factors, as assumed in our framework, thereby supporting our objectives.
Specifically, to capture temporal dependency from sequential observations, we use sequential neural networks such as Conv 1D [50] or Long Short-Term, Memory (LSTM) [29] as encoders. The encoder function, denoted 9. maps the input trajectory {s"" ..... ST) to continuous high-level representa- tion 2m = ( th ST ). where Conv ID extracts local temporal patterns from subsequences, and LSTM aggregates information over time by sequentially updating its hidden and cell states. After processing the whole trajectory sequentially, the final hidden state of the LSTM and the final output of the ConviD layer serve the representation Zy--
However, continuous representations 2m are incompatible with our framework's requirements. To address this, we apply vector quantization layer to discretize the latent space and approximate the latent factor. This layer maps 2m to the nearest vector a predefined embedding dictionary E = fei, c eg), ) where each vector e; represents a distinct group in the discrete embedding space. The assignment is realized by finding the nearest neighbor in the dictionary as &m = arg min,, l|am eill2, where Rm represents the quantized vector e; closest to the continuous representation 2m
Latent Factor Estimation via Conditional Reconstruction Reconstruction plays a critical role in unsupervised latent factor estimation by ensuring that the reconstructed distribution closely matches the true observed distribution. As stated in Definition 4.1 this alignment allows the estimated latent factors to approximate the true latent factors. Given the nature of the transition processes, we design a conditional decoder that uses the state-action pairs a: as conditions to guide the reconstruction of 87" These conditions, together with the estimated latent factors K.. serve as inputs to the decoder. The reconstruction accuracy is evaluated through the reconstruction likeli- km)) where PReson denotes the reconstruction distribution. It provides a probabilistic measure of how well s/" approximates st" This likelihood quantitatively evaluates the performance of the decoder and the ability of the model to accurately reconstruct observed states.
hood PRecon |s a!" |$
Training Objectives
The parameters are optimized according to the following ELBO objective:
LELBO LRecon + aL + 3.ommit
where a and are weights for the corresponding loss components. Specifically, (1) Reconstraction loss LRecon = E, ||s De ), s 1, a." 2/. This measures the discrepancy between the reconstructed state 8.Th and the original state s." where En and De are the encoder and decoder, respectively. (2) Quantization loss Lqum E. ||sg im]] em. 12 This evaluates the discrepancy between the encoder output ²m and its discretized representation c. Since the quantization step is undifferentiable, we use the stop-gradient operation sg(:) to update the dictionary vectors without affecting the encoder parameters. (3) Commimment loss LCommit = Σ; ||=mm - sg :ll2. This term minimizes the discrepancy between Zm and em, encouraging 2m to align closely with the embedding
5

---

Policy Learning
Traeccury Collection
Factoe Group
(a) Latent Estimation Franscwork
8:
bnblcking Didiemme
New Indivitual
Sex
Condition
Polley Adeptation
En
Dveance Calcalation
Indee fhom Didioraay
Vettor Quenlization Layer
Figure 2: (a) Latent estimation framework takes each trajectory So:r as input and processes it through quantized encoder to estimate the latent factor R. A conditional decoder then reconstructs š,, using (Sr-1, ar-1) 1) as condition and R as input. (b) Once the estimated latent factors are assigned to each trajectory, the policy learning framework integrates these latents as augmented labels optimize the RL policy. For new individuals, the initial policy is adapted based on their group affiliation, enabling indivicualized policy adaptation for newcomers.
space. By applying the stop gradient to c,,i, gradients from this loss do not change the dictionary vectors, but instead optimize the encoder parameters.
5.2 Policy Learning Framework
The estimation network is pre-trained offline. When new individual arrives, we estimate k and adapt the policy simultancously through new interactions. Specifically,
Latent-based Policy Individualization The estimated latent individual-speciff factors R, together with the offline trajectories from all individuals, used to learn the individualized policy #x- We view the estimated factors as an augmented component of the policy input, and the policy training objective is adjusted match the unique characteristics of each individual.
Take Q-learning (56) as an example. the individualized process, the latent factor is augmented as policy input as Mr (Sy; 0²)) - nT (s"", Rmm 0*), where OHr represents the parameters of the policy network. This incorporation allows the policy to adapt effectively to individual-
dynamics.
The training objective is updated accordingly
J [(0)) E [E 7'Q (st, μ4 (s?" Rm: O²)) 89)] where 7 is the discount factor, and Q represents the Q value. Such individualization improves policy adaptability across varying environments, and our framework is general enough to be seamlessly integrated with various RL. algorithms.
Policy Adaptation for New Individual Policy adaptation for a new individual involves two steps: initializing the policy from the individualized policy Tt and fine-tuning it through new interactions. For a new individual from group In. the group factor Rn is first estimated. The policy network Nnow is then initialized by directly transferring parameters from T Specifically, Tnew arg max, Es a D.D. (Ra, (S:)) is fine-tuned based on the new individual's s trajectory D, by maximizing the expected reward. The dataset D,, is incrementally augmented with new observations (st, a, R,,5544 $1+1) collected under Taew- This process refines the policy to better fit the specific characteristics of the new individual. Additionally, the updated D, helps to accurately estimate the latent factor of the new individual, further improving the adaptation process.
Experiment
Evaluation Metries To measure the latent identification, we quantify the correlation between the estimated and true latent factors using: (1) Pearson Correlation Coefficient (PCC) for single latent, which quantifies the linear correlation between the estimated and true factors, and (2) Kernel Canoni- cal Correlation Analysis (KCCA) for multiple latents, which evaluates the correlation between sets of estimated and true factors. An absolute value close indicates a strong correlation and better latent recovery. To evaluate the control performance, we measure the adaptation performance using: (3) jumpstart, which captures the improvement initial performance when learning agent leverages knowledge from source tasks; (4) accumulative reward, which indicates the learning quality over the

---

learning process; and (5) initial and final reward, which measures the initial performance benefiting from policy adaptation and the performance after the full training process.
Baselines For estimation evaluation, we compared with: (1) Disentangled Sequential Autoen-
coder [85] which disentangles latent representations into static and dynamic parts. However,
does not capture the global influence of k. limiting its ability to model individual-specific factors. (2) Population-level component, which embeds latent factors based on population data instead of -ndividual-specific information. For policy evaluation, our baselines include: (3) Aligned Latent Models 20] which jointly optimize a latent-space model and policy maximize returns. (4) Soft Actor-Crite (SAC) [24), which incorporates entropy into the objective function to encourage ex- ploration and improve robustness. (5) Deep Deterministic Policy Gradient (DDPG) [56), which combines deterministic policy gradients with deep neural networks for continuous action spaces. (6) Dueling Double Deep Q-Network (D3QN) [78,, which introduces dueling architecture for value function estimation to improve value estimation. (7) Rainbow DQN [28], which integrates prioritized experience replay and dueling network architectures to improve performance and learning stability.
6.1 Evaluation on Latent Estimation Framework
Synthetic Experiments We first conduct experiments the synthetic dataset to evaluate the effectiveness of the estimation framework. The dataset manually generated based on the post- nonlinear model. We design three types of latent factors к, each either satisfying violating the required assumptions. Case 1: ń. is finite latent factor following the categorical distribution Cat(0.1. 0.2, 0.3, 0.4) with cardinality of 4. Case 2:
three- dirre-dimensional finite latent factors, each following a categorical distribution Cat(0.2, 0.8), Cat(O.2, 0.3, 0.5). Cat(O.1, 0.2, 0.3, 0.4), with cardinality equal to 2, 3, 4, respectively. Case 3: are three-dimensional infinite latent factors, and each factor follows a Gaussian distribution (0, 1), uniform distribution Uniform(O, 1), and exponential distribution Exp(1), respectively. We synthetically generate 40 unique trajectories, each representing an individual, with a maximum trajectory length of 20.
KOLA
KOCA
Thans To Epedt
Carcriaal Naribli For The Laferb
Concriad Varible fro Trae Loto
Thkning Eyeoh
(a)
(b)
(c)
(d)
Figure 3: Synthetic results. (a) Comparisons of PCC trajectories in Case 1. (b-c) Scatterplot of the canonical variables in Cases 2 and 3. (d) Identifiability performance responses of the sample size.
For Case 1. we use PCC to evaluate estimation performance and report the training curve in Figure where shaded regions indicate the standard deviation. The comparative results show that our method can recover the true latent factors, outperforming other baselines. Specifically, the population-level component fails to identify the andividual-specifii factor, as overlooks the underlying differences between groups. Additionally, although the disentangled sequential autoencoder shows compromised identifiability in the carly training stage by capturing the static part of the latent space, fails to achieve full identifiability due to its inability to model individualized transition processes, leading to worse recovery performance over time. For Cases 2 and 3, we use KCCA to quality the correlation and visualize the results through scatter plots in Figure 3(b) and Figure 3(C) These results demonstrate identifiability in both finite and infinite latent factor scenarios and support the theoretical claims.
Moreover, we slightly violate the restriction on the number of individuals per group specified in Assumption 4.1 I and analyze its impact on the training curve under varying population sizes, as shown by the dashed lines in Figure 3(d) The result shows that satisfying the sample sufficiency condition necessary to recover the latent factors. addition, we evaluate the effect of the trajectory length outlined in Theorem 4.2 The findings, shown by the solid lines, demonstrate that increasing the
7

---

trajectory length would significantly improve the identifiability performance. This observation aligns with the theoretical guarantees provided by the proposed theorem.
Ablation Study The contributions of the differ- ent components in the latent estimation framework are reported in Table [ We build the autoen- coder framework with a quantization layer and add each component sequentially to the previous mod-
Table I: Contribution of each module.
Module
PCC (j)
Bias (1 o)
Quantized Encoding
0.646 +: 3.1e-0
0.099 = 2.3-04
ule. Incorporating a sequential encoder significantly improves the identifiability, which is important for
Sequuenial Encoder
0.910 :: 1.3e-04
0.077 +: 5.7e-06
Noise Estimater
0.942 = 4.0c-05
0.072 = 3.0c-07
the accurate recovery of latent factors. In the imple- mentation,
noise estimator during optimization to minimize bias and improve identifiability. The results suggest that the added components help fine-tuning the overall performance of the model, allowing for more accurate and reliable recovery of latent individual-specift factors.
FersuasionForGood Corpus We further evaluate our framework on the real-world dataset, Persua- sionForGood corpus [77). which widely used for analyzing persuasion strategies [64 7 87 The dataset contains 1017 person-to-person dialogues and 32 personality traits for each participant. In each dialogue, the persuader attempts to convince the persuadee to donate a charity, In the context of iMDP, the state represents the persuadee's 's response, the action corresponds to the persuader's utterance, and the reward is defined as the final donation. Since this offline dataset does not include real-time interactions required for control performance evaluation, we focus on identifying the latent personality traits of each individual. We use BERT ( as the backbone to embed each utterance into 7 feature representation. These features are processed through an LSTM encoder, followed by quantization layer, to recover the latent personality traits. The canonical correlation results under different latent dimensions are shown in Figure 4(a] The results demonstrate that our method is effective and feasible for real-world applications, particularly when the latent dimensions are fine-tuned appropriately.
Wovord
Popase Hatt
Laterd Prodks Indiduaal Tondte
Fanboo CON Graii inciddul Yootte
Epcood
Epsode
Ep0odd
(a)
(d)
Figure 4: (a) Canonical correlation with respect to the latent dimensions in the PersuasionForGood corpus. (b-d) Accumulative reward curves in Pendulum, HeartPole, and Half Cheetah, respectively.
6.2 Evaluation on Policy Learning Framework
Pendulum Pendulum 4 is continuous control task for RL study with the goal of swinging up and stabilizing in upright position. The states are the x-y coordinates and the angular velocity, and the action the torque applied to the pendulum. For simplicity, we choose DDPG as the basic optimization algorithm and manually create 20 individualized environments. In these environments, the gravity g is randomly drawn from a categorical distribution over the set (3.....12), The performance of the policy adaptation is evaluated on new individual with g 10.
We compare our method against several baselines: (1) SAC; (2) DDPG without prior knowledge; (3) aligned latent models; (4) pre-trained DDPG incorporating knowledge from given individuals, termed individual transfer; (5) individualized policy incorporating randomly defined group embedding. termed random group embedding. The training curves over accumulative reward are reported in Figure 4()), showing that the proposed method outperforms other baselines both jumpstart and accumulative reward. Specifically, methods that benefit from population knowledge (our method and cross- innd--iidiiiuull transfer) outperform non-transfer methods, indicating that the pre-trained policy would accelerate the learning process. However, since cross-ine cross-individual transfer ignores the individual-
8

---

specific information, such mixed policy knowledge yields worse initial performance compared to the individualized policies derived from our method.
HeartPole HeartPole [57] is a discrete healthcare environment that explores the long-term health outcomes of short-term decisions. The six-dimensional states represent different health conditions, including alertness, hypertension, intoxication, time since sleep, time elapsed, and work done. Actions be chosen from work, coffee, alcohol, and sleep. We create 100 individualized scenarios and assign each patient with individual characteristics, such as coffee tolerance, hypertension risk, and alcohol tolerance, according to categorical distribution over the set {0.6, 0.8, 1.2). The adaptation performance is evaluated on a new individual with all indices set to 1.
We compare our method against the following baselines: (1) D3QN without prior knowledge, (2) Rainbow DQN, (3) cross- -oos-iindiidual transfer with D3QN, and (4) random group embedding. The training curves over accumulative reward are shown in Figure 4{c] and our method outperforms other baselines in both jumpstart and accumulative reward. Interestingly, while inappropriate source domain knowledge result from random group embedding indicates that incorporating group embedding knowledge can enhance generalization. The group structure, together with properly estimated group information, jointly enables our method to converge better and faster than other baselines.
negatively impact control performance (see tross-individus transfer), the
Half Cheetah Half Cheetah 72 is a Mujoco- Mujoco-based task aiming to control 2D bipedal robot. The agent consists of 9 links and 8 joints, and the goal to apply torque to the joints to make the cheetah run forward as fast as possible. We introduced 50 individualized settings with the gravity following a categorical distribution with probabilities p = 0.2 and corresponding g values from {8, 8.5,...... 10). The adaptation performance is evaluated on new individual with g = 9.8. We compare our method with (1) DDPG without prior knowledge, (2) aligned latent models, (3) cross-individual transfer with DDPG, and (4) random group embedding. The comparative results, shown in Figure 4(d) indicate that our method outperforms all baselines terms of convergence speed and efficiency. While inappropriate source domain data can degrade performance (see coss-individuul transfer), the inclusion group embedding facilitates generalization, enabling more effective adaptation.
Conclusion and Limitations
Our work focuses on learning latent state-transition processes from observed state-action trajectories, guaranteeing identifiability in the presence of latent -ndividual-specifie factors. To the best of our knowledge, this study provides novel identifiability guarantees for several settings that have not been previously addressed. Despite these contributions, our approach has three major limitations. (1) It currently does not account for instantaneous causal dependencies within st. This limitation could be addressed by adjusting the temporal resolution of the data and explicitly modeling these dependencies. Integrating causal graphical models or advanced inference techniques could further enhance the framework's 's ability handle instantaneous causal relationships. (2) While empirical results suggest the framework may adapt to scenarios with continuous latent factors, a formal nonparametric proof for these settings remains absent. Developing such proof is an important avenue for future work. (3) The model assumes latent factors are time-invariant. Establishing such theoretical identifiability for time-varying latent factors highly non-trivial and would require additional theoretical constraints to establish identifiability. These limitations highlight key directions for future research.
Furthermore, practical concerns such as privacy, robustness, and reliability are essential for real- world applications. To address privacy risks, techniques like de-identification (c.g., removing direct identifiers, data perturbation, pseudonymization) and differential privacy approaches should be explored. Ensuring privacy and security will enhance the framework's 's applicability practice.
Acknowledgement
This material based upon work supported by NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract 1515L1 and grants from Salesforce, Apple Inc., Quris AI, and Florin Court Capital. BH supported by NSF DMS- -428058

---

References
[1] M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey, ACM Computing Surveys, 55(7):1-38, 2022.
[2] Jonathan Bassen, Bharathan Balaji, Michael Schaarschmidt, Candace Thille, Jay Painter, Dawn Zimmaro, Alex Games, Ethan Fast, and John C Mitchell. Reinforeement learning for the adaptive scheduling of educational activities. In Proceedings of the 2020 CHI Conference on Human Factors Computing Systems, pages 1-12, 2020.
[3] Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off-policy evaluation in infinite- horizon reinforcement learning with latent confounders. In International Conference on Artifi- cial Intelligence and Statistics, pages 1999-2007. PMLR, 2021.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[5] Qingpeng Cai, Ruohan Zhan, Chi Zhang, Jie Zheng, Guangwei Ding, Pinghua Gong, Dong Zheng, and Peng Jiang. Constrained reinforcement learning for short video recommendation. arXiv preprint arXiv:2205. 13248, 2022.
[6] Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process. arXiv preprint arXiv:2401. 14535, 2024.
[7] Maximillian Chen, Weiyan Shi, Feifan Yan, Ryan Hou, Jingwen Zhang, Saurav Sahay, and Zhou
Yu. Seamlessly integrating factual information and social content with persuasive dialogue. arXiv preprint arXiv:2203.00 0.6657, 2022.
[8] Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing, pages 1-4, 2009.
[9] Jacob Devlin, Ming-Wac Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18/0 04000, 2018.
[10] Xinshuai Dong, Biwei Huang, Ignavier Ng, Xiangehen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. A versatile causal discovery framework to allow causally- hidden variables. arXiv preprint arXiv:23/2.11001 2023.
[11] F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. ArXiv Preprint ArXiv:1710.05266 2017.
[12] Bisni Fahad Mon, Asma Wasfi, Mohammad Hayajneh, Ahmad Slim, and Najah Abu Ali. Reinforcement learning in education: A literature review. In Informatics, volume 10, page 74. MDPI, 2023.
[13] Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non- stationary reinforcement learning. Advances in Neural Information Processing Systems, 35:31957-31971, 2022.
[14] Apple WP Fok, Hau-San Wong, and YS Chen. Hidden markov model based characterization of content access patterns in an e-learning environment. In 2005 ieee international conference on multimedia and expo, pages 201 204. IEEE, 2005.
[15] Evan M Forman, Michael P Berry, Meghan I. Butryn, Charlotte J Hagerman, Zhuoran Huang, Adrienne S Juarascio, Erica M LaFata, Santiago Ontañón, Mick Tilford, and Fengqing Zhang. Using artificial intelligence to optimize delivery of weight loss treatment: Protocol for an eificacy and cost-effectiveness trial. Contemporary Clinical Trials, 124:107029, 2023.
[16] Evan M Forman, Stephanie Kerrigan, Meghan L Butryn, Adrienne S Juarascio, Stephanie M Manasse, Santiago Ontañón, Diane H Dallal, Rebecca J Crochiere, and Danielle Moskow. Can the artificial intelligence technique of reinforcement learning use continuously-monice digital data to optimize treatment for weight loss? Journal of behavioral medicine, 42:276-290 29776,900000 2019.
10

---

[17] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning (ICML), 2019.
[18] D. Ghosh, A. Gupta, and S. Levine. Learning actionable representations with goal conditioned policies. ICLR, 2019.
[19] Susobhan Ghosh, Raphael Kim, Prasidh Chhabria, Raaz Dwivedi, Predrag Klasjna, Peng Liao, Kelly Zhang, and Susan Murphy, Did we personalize? assessing personalization by an online reinforcement learning algorithm using resampling. arXiv preprint arXiv:2304.055000 2023.
[20] Raj Chugare, Homanga Bharadhwaj. Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhut- dinov. Simplifying model- rl: learning representations, latent-space models, and policies with one objective. arXiv preprint arXiv:2209. 284666, 2022.
[21] K. Gregor, G. Papamakarios, F. Besse, L. Buesing, and T. Weber. Temporal difference variational 2018.
auto-encoder. arXiv preprint 1806.
[22] Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Bran-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In International Conference on Machine Learning, pages 3875-3888 3886. PMLR, 2020.
[23] D. Ha and J. Schmidhuber. World models. In Advances in Neural Information Processing Systems, 2018.
[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[25] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.0455 2018.
[26] D. Hafner, T. Lillicrap. M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[27] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: arxiv:/50..002.0 022..0 2015.
[28] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
[29] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):1735 1997.
[30] Jesse Hoey, Pascal Poupart, Craig Boutilier, and Alex Mihailidis. Pomdp models for assistive technology. In Assistive Technologies: Concepts, Methodologies, Tools, and Applications, pages 120-140. IGI Global, 2014.
[31] William Hoiles, Vikram Krishnamurthy, and Kunal Pattanayak. Rationally inattentive inverse reinforcement learning explains youtube commenting behavior. The Journal of Machine Learning Research, 21(():6879-66 6917, 2020.
[32] Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics: methodology and distribution, pages 162-190. Springer, 1992.
[33] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021.
[34] Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. Advances in Neural Information Processing Systems, 35:5549-5566 2022.

---

[35] Biwei Huang, Kun Zhang, Pengtao Xie, Mingming Gong, Eric P Xing, and Clark Glymour. Specific and shared causal relation modeling and mechanism-based clustering, Advances in Neural Information Processing Systems, 32, 2019.
[36] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by cone-contrattive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[37] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[38] Xiaoyuan Ji, Hu Ye, Jianxin Zhou, Yajun Yin, and Xu Shen. An improved based optimization algorithm and its application to a combinatorial optimization problem in foundry industry. Applied Soft Computing, 57:504-516 -16004---6,,0 2017.
[39] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, Czechowski, ... and H. Michalewski. Model-based reinforcement learning for Atari. arXiv preprint arXiv: /003. 003774 2019.
[40] M. Karl, M. Soelch, J. Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint
2016.
[41] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[42] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arKiv: 1312.6114, 2013.
[43] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[44] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[45] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial identifiability for domain adaptation. arXiv preprint arXiv:2306.06510, 2023.
[46] R.G. Krishnan, U. Shalit, and D. Sontag. Deep kalman filters. arXiv preprint 15.51115 2015.
[47] T. D. Kulkarni, A. Saeedi, S. Gautam, and S. J. Gershman. Deep successor reinforcement learning, arXiv preprint 16Xiv:1606.0233 02396, 2016.
[48] Jeongyeol Kwon, Yonathan Efroni, Constantine Caramanis, and Shie Mannor. RI for latent mdps: Regret guarantees and lower bound. Advances in Neural Information Processing Systems, 34:24523-24534. 2021.
[49] Jeongyeol Kwon, Yonathan Efroni, Shie Mannor, and Constantine Caramanis. Prospective side information for latent mdps. arXiv preprint 10X00:2310.07556, 2023.
[50] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural compulation, 1(4):541-551, 1989.
[51] Yu Lei and Wenjie Li. Interactive recommendation with user-specific deep reinforcement learning, ACM Transactions on Knowledge Discovery from Data (TKDD), 13(6):1-15, 2019.
[52] Lesort, N. Dfaz-Rodrísuez, J. F. Goudou, and Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392, 2018.
[53] Minne Li, Mengyue Yang, Furui Liu, Xu Chen, Zhitang Chen, and Jun Wang. Causal world models by unsupervised deconfounding of physical dynamics. arXiv preprint 20x1x 2020.
12

---

[54] Zhige Li. Derek Yang, Li Zhao, Jiang Bian, Tao Qin, and Tie- Yan Liu. Individualized indicator for all: Stock-wiss technical indicator optimization with stock embedding. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
pages
[55] Peng Liao, Kristjan Greenewald, Predrag Klasnja, and Susan Murphy. Personalized heartsteps: A reinforcement learning algorithm for optimizing physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-22, 2020.
[56] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 150.00291, 2015.
[57] Vadim Liventsev, Aki Härmä, and Milan Petković. Towards effective patient simulators. Frontiers in artificial intelligence, 4:798659, 2021.
[58] Chaochao Lu, Bernhard Schölkopf, and José Miguel Hemmáneez-oobtoo. Deconfounding reinforcement learning in observational settings. arXiv preprint
10576, 2018.
[59] Zhiyao Luo, Mingcheng Zhu, Fenglin Liu, Jiali Li, Yangchen Pan, Jiandong Zhou, and Tingting Zhu. Dtr-bench: An silico environment and benchmark platform for reinforcement learning based dynamic treatment regime. arXiv preprint arXiv:24055 18610, 2024.
[60] S. Mahadevan and M.. Maggioni. Proto-value functions: laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research (JMLR),
2007.
[61] Geoffrey J McLachlan, Sharon X Lee, and Suren I Rathnayake. Finite mixture models. Anmual review of statistics and its application, 6:355-377 378, 2019.
[62] Kevin P Murphy. A survey of pomdp solution techniques. environment, 2(10), 2000.
[63] Alizée Pace, Hugo Yèche, Bernhard Schölkopf, Gunnar Rätsch, and Guy Tennenholtz. Del- phic offline reinforcement learning under nonidentifiable hidden confounding. arXiv preprint arXivv2306.01157. 2023.
[64] Wei Peng. Yue Hu, Luxi Xing, Yuqiang Xie, and Yajing Sun. Do you know my emotion? emotion-aware strategy recognition towards a persuasive dialogue system. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 724-739. Springer, 2022.
[65] Alexander Ritchie, Robert A Vandermeulen, and Clayton Scott. Consistent estimation of identifiable nonparametric mixture models from grouped observations. Advances in Neural Information Processing Systems, 33:11676-11686 2020.
[66] Doaa Shawky and Ashraf Badawi. Towards personalized learning experience using rein- forcement learning. Machine learning paradigms: Theory and application, pages 169-187, 2019.
[67] Dongjian Song, Bing Zhu, Jian Zhao, Jiayi Han, and Zhicheng Chen. Personalized car-folloe folloogg control based hybrid of reinforcement learning and supervised learning. IEEE Transactions on Intelligent Transportation Systems, 2023.
[68] Seth Sullivant, Kelli Talaska, and Jan Draisma. Trek separation for gaussian graphical models. 2010.
[69] Yuewen Sun, Erli Wang, Biwei Huang, Chaochao Lu, Lu Feng, Changyin Sun, and Kun Zhang Acamda: Improving data efficiency in reinforcement learning through guided counterfactual data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15193-15201 2024.
[70] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
13

---

[71] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning, Advances in neural information processing systems, 30, 2017.
[72] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[73] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[74] Robert Vandermeulen and Clayton D Scott. On the identifiability of mixture models from grouped samples. arXiv preprini arXiv: 152.06644, 2015.
[75] Thanh Vinh Vo, Pengfei Wei, Wicher Bergsma, and Tze Yun Leong. Causal modeling with stochastic confounders. In International Conference on Artificial Intelligence and Statistics, pages 3025-3033 PMLR, 2021.
[76] Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems, 34:21 164-21175, 2021.
[77] Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5635-5666 5549, Florence, Italy, July 2019. Association for Computational Linguistics.
[78] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. International conference on machine learning, pages 1995-2003. PMLR, 2016.
[79] M. Watter, Springenberg, Boedecker, and M. Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. NeurIPS, 2015.
[80] L. Wiskott and T. J. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, (4((4)715-777) 770, 2002.
[81] Xinghao Yang, Weifeng Liu, Wei Liu, and Dacheng Tao. A survey on canonical correlation analysis. IEEE Transactions on Knowledge and Data Engineering, 33(6):339-2368,, 2019.
[82] Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. Advances Neural Information Processing Systems, 35:26492 265003 2022.
[83] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint 10.0v:21 2021.
[84] Chunli Yin and Jinglong Han. Dynamic pricing model of e-commerce platforms based on deep reinforcement learning. CMES-Compune Modeling in Engineering & Sciences, 127(1), 2021.
[85] Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference Machine Learning, pages 5670-5679. PMLR, 2018.
[86] Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of medical Internet research, 19(10),338, 2017.
[87] Donghuo Zeng, Roberto S Legaspi, Yuewen Sun, Xinshuai Dong, Kazushi Ikeda, Peter Spirtes, and Kun Zhang. Counterfactual reasoning using predicted latent personality dimensions for optimizing persuasion outcome. In International Conference on Persuasive Technology, pages 287-300. Springer, 2024.
[88] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and Levine. Learning invariant representations for reinforcement learning without reconstruction. ICLR, 2021.
14

---

[89] Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. International Conference Machine Learning, pages 11214-11224, PMLR, 2020.
[90] Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint 1 2012.
[91] M. Zhang, S. Vikram, L. Smith, Abbeel, M. Johnson, and S. Levine. Self-supervised visual planning with temporal skip connections. ICML, 2019.
[92] M. Zhang, Vikram, L. Smith, Abbeel, M. Johnson, and S. Levine. Solar: deep structured representations for model-basee reinforcement learning. arXiv preprint arXiv:1805 (9 09105, 2018.
[93] Xuezhou Zhang, Yuda Song, Masatoshi Uehara, Mengdi Wang. Alekh Agarwal, and Wen Sun. Efficient reinforcement learning in block mdps: A model-free representation learning approach. In International Conference on Machine Learning, pages 26517-26547. PMLR, 2022.
[94] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances in Neural Information Processing Systems, 35:164111 2022.
15

---

Supplementary Materials for "Identifying Latent State- Taat-Trrnsstton Processes for Individualized Reinforcement Learning"
A
Notation and Terminology
We summarize the notations used throughout the paper in the following table.
Index
Trajectory
Time index
T
Total length of time series
G
Number of groups
M
Number of individuals
m
Index for a specific individual
i,j
Variable element index
a.8
Weights of ELBO objective
GG
[1,2 ...,...... G)
Sequence of integers from G inclusive
Variable
E
i..d. noise term for at time State transition function
f
St. S
Observed reconstructed states at time f State, action, and latent factor from individual d.-dimensional observed states
s
a m Rm
881 82,-
8d
a
(11, U12 ad.
dz-dimensional observed actions
K
K1, R2:
........ Kd.
d.-dimensional latent individual-specifc factors
B Identifiability Theory
Given the identifiability theorems, we first provide intuitive explanations for each assumption and discuss their relevance to real-world applications. Then, we provide the proof. Finally, we introduce some preliminaries related our theorems, which are essential for the proof.
B.1
Preliminaries for Theorem 4.1 and 4.2
B.1.1 Markoy Property
The first-order Markov property implies that the transition probability to the next state depends only on the current state, uninfluenced by the sequence of previous states. Specifically,
Definition B.1 (First-order Markov Property (70).. A stochastic process (X, : i N3 has the first- order Markov property if. for each set of times f, t 1- O and corresponding state It, T2-1,.... Fo in the state space, the following conditional independence property holds:
P(X: Iz X X1-1,X Xx 2t-2, Xo .o) = P(X, Fz X 21-1)
(5)
The first-order Markov property implies that the transition probability to the next state depends only on the current state, uninfluenced by the previous states. In the context of the state transition process, it possesses the first-order Markov property. Mathematically: can be represented as:
P(s.|s:11 at-1, 8t a Sp, ao) = P(s (t a1-1),
where (s, (S,]|s,- 18 81-1) is the transition probability from (s:-1, a 1) to the state St.
B.1.2 Finite Mixture Model
finite mixture model used for modeling total population that comprises unobserved hidden groups. Each of these groups is assumed to follow its own distinct probability distribution. In this context, the overall population model is expressed as a weighted sum of these individual distributions [61. Specifically.
16

---

Definition B.2 (Finite Mixture Models 744). A finite mixture model is probability law based on a finite number of probability measures, Pi..... Hmm Hmm and a discrete distribution Wi..... Wm- realization of a mixture model is generated by generating a component random k, S k: S m, and then drawing from pk P. Then, the mixture measure P defined as weighted sum of probability measures #4 with weights wi Specifically,
p
widppr
B.2 Discussions on Assumptions
Group Determinacy The distinct values of the latent factor y categorize the population into separate groups, with each group characterized by its unique probability distribution and denoted as Eg Tgōx,(()) The mixture formulation implies that the latent factor k serves as a categorical variable, with each unique value explicitly specifying a distinct group within the population. Such a formulation facilitates the identification and analysis of heterogeneous subpopulations within the finite mixture model.
The idea of group determinacy is important in real-world applications. Take personalized education example. For each student m, st represents their current knowledge state, an denotes their personalized learning action, and the function / determines the unique educational trajectory for each student. The latent factor x influences how student's 's learning progresses over time. It can be based on factors such as learning style preferences that help to logically group students. Specifically, one group might consist of visual learners who excel in interactive, craphically-orientee while another group might include students who prefer textual information and excel in reading and writing-intenee
subjects,
subjects. Each group exhibits its own set of learning outcomes and patterns, allowing educators to personalize teaching methods and materials to effectively meet the different needs of each group.
Sample Sufficiency In finite mixture model with G groups, each group requires sufficient observations to identify the latent group factor к. This assumption provides a minimum number of observation samples, which is 2G - observations each group. Such a threshold ensures that we have enough information and variability in the observed data to distinguish the characteristics of each group. This assumption helps to identify the unique characteristics each individual, which critical for identifiability.
Sample sufficiency indicates that sufficient data are needed to achieve identifiability, which is fundamental assumption in many analytical models. For example, in the context of nonlinear ICA using auxiliary variables (37,, it is necessary to have least 2n values for the auxiliary variables to ensure sufficient variability and guarantee identifiabiliyy. Similarly, for successful disentanglement with minimal change [45), least 2n + domain embeddings are required to ensure identifiability. Intuitively, without sufficient data to provide us with relevant information about the parameters, it is impossible determine the values of these parameters.
Asymptotic Identifiability Asymptotic identifiability refers to the property that a model becomes identifiable the sample size goes to infinity. In practical terms, this means that given an infinite amount of data, one would be able to consistently estimate the parameters of the model. In the context of finite mixture models, if there enough samples for each individual, then the corresponding components can be identified directly from each individual 35
The overlapping condition requires the existence of at least two different individuals within the same group of the population, who have identical conditional probabilities. This assumption is crucial as ensures that the model accounts for overlapping behavioral responses between different individuals, which is a common phenomenon in heterogeneous populations. Consider personalized education as an example, where students come from different academic backgrounds and have different levels of prior knowledge. Despite this initial heterogeneity, is possible for two students in the same learning group to have the same probability of successfully completing a task.
17

---

B.3 Proof of Theorem 4.1 and Theorem 4.2
We first show that the individualized transition processes can be viewed as a finite mixture model with grouped samples and then derive the identifiability under two scenarios.
Assumptions in Theorem 4.1 and Eq. (2) ensure that individuals within the same group share identical joint distributions. Suppose the observations can be grouped into G finite components, then the joint distribution can be factorized as:
Proof on Theorem 4.
/ / P(x))(s', al
P(,,0,8')=
(8)
This formulation asserts that the joint distribution of P(s, s') for the entire population can be modeled mixture model governed by the respective N, values.
The following Lemma B.I addresses the identifiability of mixture models from grouped samples.
Lemma B.1 (Identifiability of Mixture Models from Grouped Samples (74).. Suppose have observations from a mixture model and that they are grouped such that observations the same group known to be drawn from the same componens. Denote by G the number of groups. If there are at least 2G - observations per group, any mixture of G probability measures can be uniquely identified.
implies that with sufficient data per group, each component of the mixture model can be determined without ambiguity from the observed data. Specifically, suppose we have mixture model consisting of different probability distributions, that is, G components Cı ......, CG. Each component c;
corresponds to unique probability density function (.). These components mixed together, with each component having a mixing weight TT, satisfying n 2 and Σ ,ि
Now suppose we have G observation groups 92.92..... ................................ 9G. with each group 9i containing obser- vations that are independent and identically distributed drawn from the same component c, Since sample sufficiency ensures that the sample size of observations within each group is greater than 2G 1, then the identifiability of is guaranteed by Lemma B.1
Under the assumptions in Theorem if each individual's trajectory is sufficiently long, the latent individual-specifee factors become asymptotically identifiable. Consider an extreme scenario where each individual is treated as a distinct group. Some individuals may share the same latent factor к and can be grouped together. In that case, is necessary measure the similarity between individuals and merge those with the same into a single group.
Proof on Theorem 4.2
An intuitive criterion for merging is follows. Asymptotically, for a particular state-action pair (s, a) = (s", "'s",a"""" a²), the probability of given a") will be identical across individuals in the same group. Define t the time of the j-th occurrence of (s", a²). For each j. let X = {s t 1....). representing the collection of states observed at time + given the fixed state-action pair (s*,a" a*) time /. In this way, X can be considered as samples drawn from a particular group j. Identifiability is then guaranteed by Lemma B.1 since the distributions of X allow the grouping of individuals based shared latent factors.
Remark B.1. Prior work (35] used a Gaussian mixture model as a prior on the coefficients, while the latent confounder variable, denoted as Z, was constrained to a binary state, thereby indicating group membership for a given individual. We extend this foundation by generalizing the latent confounder to set of discrete values and considering a nonparametri model for broader applications.
C Further Discussion on Identifiability Theorem
Recent work 10 provides the necessary and sufficient conditions for the identifiability of certain latent structural patterns, but it rules out the case of triangle structure involving latent variables. In
18

---

this paper, we extend their work to the temporal case and provide the identifiability of the latent group factor K, where can be either continuous or discrete. Furthermore, we allow multiple instances of to influence the state transition dynamics.
C.1 Problem Setting
In Theorem 4.3
we aim to identify the latent group factors based on a post-nonlinear temporal causal model, as shown in Figure 5 We assume the existence of learnable and invertible embedding mapping f. which able to preserve the causal structure intrinsic to the state 8. Specifically,
Definition C.1 (Post-nonlineat Temporal Causal Models), Consider scenario where d, observed states from the k-th individual are denoted as st = (sf......,, sd.r)) which is direct observation of embedded representation h(s), alongside unobserved group factors [K] :.m.).)))))))): ) The state transition dynamics satisfy
$1,41 -h
Σ aigh((sf,)) JEP;
+ E Eiga Jat jEC.
+
(10)
3=1
fori=1,...,""" nh, Here, Osi and Bij represent causal coefficients that quantify the influence of the state h(53)) and the action aj, on 1((51,41),, respectively. P, and Li denote the sets of direct state and action that influences s (or his h(s(,141)) Actions are considered to be stochastic. The coefficients >g are individual-oe -specffcc and show variation across individuals. The random noise term c2,441 is independent of 832 and az for all j E N to account for unmeasured influences.
K2
Km
an
a
a
h(Sa)
A(ss)
h($p-1)
A(S))
h(G2e2)
Figure 5: Post-nonlinear temporal causal model.
Definition C.2 (Minimal Rank Set). Let S. = {s1,t. $2,1 sd,,a represent the set of all state variables in the system at any time t = ==,...,,T? and ler L {K1, [K1, K2..... Rem) represent the set of
latent variables, where m and dg are the numbers of latent and state dimensions, respectively. A subset R2 C S, U (or R2,, ( with cardinality is called minimal rank set if satisfies the following conditions:
(i) The bottleneck set, defined as B LU for any given time t, can t-separate (see Definition C.5 any pair of minimal rank sets (Re, Re, where t- < tt.
(ii) There does not exist subset R2 C R2 with (R,, x+ that can satisfy condition (i).
The set R2 is considered minimal in the sense that it is the smallest cardinality subset of observed state variables that includes bottleneck set and disjoint state variables, capable of representing the essential separation status within the system. An illustrative example of a minimal rank set (see Figure G is shown the yellow area, and a bottleneck set is depicted in the green area.
C.2 Proof of Theorem
The underlying intuition of Theorem 4.3 is that, in the absence of latent variables, rank information should align with what Conditional Independence (CT) skeleton (see Definition C.6) provides; if not, then there must exist at least one latent variable.
19

---

C.2.1
Necessary Lemmas
ΣA,B The following lemma indicates that the rank of the covariance matrix (see Definition C33
between any two sets of variables A and B is less than or equal to the sum of cardinalities of any trek-separating (see Definition C.5) sets Ca and CB. The equality holds for generic covariance matrices consistent with the graph G.
Lemma C.1 (Trek Separation for Directed Graphical Models [68]. The submatrix AA.B has rank less than or equal to r for all covariance matrices consistent with the graph G if and only if there exist subsets CA. CB C V(g) with |Ca + CB S r such that Ca. CB I-separates A from B.
Consequently,
rank({A..)) S min + [CB] (Ca. CB) I-separates A from B}
(11)
and equality holds for generic covariance matrices consistent with G.
Lemma C.2 (Identifiability of Linear Regression Models). Consider linear regression model with a response variable Y and p predictors X1, X Xp. The lineur relationship is defined as: Y Po + 31 X1 + B2X² +...t 8p +
(12)
where So, 1u.3...... Sp are the regression coefficients and E is the error term. The matrix representation can be expressed as:
(13) where Y is the response vector, X is the design matrix including predictors, B is the vector of regression coefficients, and e is the vector of error terms. For the regression coefficients 3 to be identifiable, the design matrix X must have full column rank, meaning no predictor is a perfect linear combination of the others. This ensures that the matrix X² X is invertible, allowing for the unique estimation of B3 through:
Y = x
i x
(14)
Lemma C.3 (Identifiability of Factor Analysis). Consider a factor analysis model with p observations for each of n individuals and common factors (k < p). The relationship is defined by the factor loading matrix L Roxk and the factor matrix F E IR Specifically,
X =F +
(15)
I the observation matrix and E p_ the error term matrix. The factor loading matrix L and the factor matrix F are unique up to an orthogonal transformation. Specifically, for any orthogonal matrix Q, if we set I.' IQ and F' = Q" F, the transformed matrices L/ and F' also satisfy the model criteria.
where X
C.2.2 Proof of Structure Identifiability
Proof. Suppose latent factors exist and influence the embedding of the observed states h(s), which preserve the causal structure intrinsic to the states s. According to Lemma C. the rank of A B, should be less than equal to min[|CAr
[CB,I)- In the absence of latent factors, according to the CI skeleton, the minimal configuration to t-separate A; from B; is by
(fh(s).),... (). 0) or (0, (h(s..)........ (.(.,,))...,((s..)))
(16)
Consequently, the rank of the covariance matrix is rank(EA,,B,) [(h(s1,;), ((sa, |0] dg. the calculated rank is greater than dg, it implies the presence of latent variables accounting for the unexplained variance since the observed variables alone would not result in such rank deficiency.
In scenarios with latent factors, the maximum rank deficiency observed across covariance submatrices, representing the discrepancy between the expected and actual ranks, establishes a lower bound for the number of latent variables. Considering the minimal t-separation of A; and B; occurs via
(4h(s1.)),... hssa,,) K.,..., km), }.
(0, (h((s..) ... h(sa..)), Ki...., Rm))).
(17)
Then the rank of the covariance matrix rank(EA,,B) = [(h(s,,),..... h(sa,,), KI. Ki...., Kmm) [0] = + d,. By iteratively computing the rank of (ank(2AA,, B,), consistent value corroborates the existence of latent factors influencing all observed states. Furthermore, the count of latent factors can be deduced by rank(EA,
dy.
In conclusion, under the conditions of the theorem, the observed rank deficiency in the covariance matrix of observed variables cannot be explained by the observed variables alone, it implies the existence of latent variables. Furthermore, the number of such latent variables can be inferred from the extent of the rank deficiency.
20

---

C.2.3 Proof of Parameter Identifiability
Proof. For each individual k, consider the proposed model at any time and t + 1:
h($(.e) Σ
Σ
Bisa
Σ Agys + E
1-1
jEP:
JEC,
h(s
h(51++1))
ang aigh(s)fe)
+
Σ
Bega +
Astty + c
jEP:
jEC,
Subtracting these two equations, we obtain:
h(s h(s(tt)
Σ ang(h(5),,)
h(s (,t-))) + Σ
Biylaj, - - + (c (c,,411- -..))
jEP.
jEC,
Define xt,4+1 (s(11+1) h( 3f,t+1 0,,411 d and r/,441 ch,41 - E.e. Substituting these, the model transforms to:
Σ JEP.
auyat a
+
Σ Bupli + nhet1-
JEC.
In that case. the identifiability of a and B can directly derived by Lemma We further assume that the m latent factors follow the Normal distribution. Drawing on methodologies used in factor analysis C.3
then À is orthogonal-wise identifiable.
C.3 Examples For Theorem
this section we present four illustrative examples to describe cases where identifiability is achieved. For the sake of simplicity, we define X as X h(s) and omit the terms and from our illustration for simplicity, under the assumption that they are random and independent variables.
Kit
X1
[Ix,, X,)), B Xu. rank(Einn)
Figure 6: Single latent variable and one- dimension state.
Figure Multiple latent variables and one-dimension state.
Xxu
X
X
Figure 8: Single latent variable and multi multi-dimension states.
Figure 9: Multiple latent variables and -ulti--iimension states.
Figure 10: Examples that illustrate different identifiable cases.
Example 1 In Example 1, as shown in Figure o there is only one latent factor and one- dimensional states. The bottleneck set is (X3, k), and the pairs of minimal rank sets are (A, B) ((1,.X, Xs) (X,.Xa, Xs)). According to the causal graph, the minimal way to t-separate A
from B either ((N, Xa), 0) or (0, {n. Xa)). Consequently, the rank the covariance matrix rank(EA.B) = [K, Xz) - (0] 2. According to Theorem 2, the fact that rank = 2 > 1 indicates the presence of the latent factor. Consequently, the number of latent variables can be
deduced rank(EA.) = = =221=1.
21

---

Example 2
As shown in Figure 7 there are m latent factors and one one-dimensional states. The bottleneck sets are ((Xm+2, Ki...... Kmn) ................................ (XT m- KI,.... Ki,...,, Rm)). Suppose T 2m + 4. then the pairs of minimal rank sets are (A,, B1) = ((X), (X....., 22) (Xm+2;- m........ X2m+3)) and
(A2, B2) = ((X2,..... X Xm+3), (Xm+3.....
X X2m+4)). According to the graph, the minimal config- uration to t-separate A₁ from B₁ is either ((K1:-.. Km: X 2 ()) or (10, (KI..... Km: X x } with rank(2"A,,B,
= KKI,,... Kim: Xm+2) + 10 m + 1. The minimal configuration to t-
separate A2 from B2 is either ([K),..., Rm: X Xm+3) 0) (0, (KI,..... Kmm Xm+3 X }), resulting in rank(EA,, A2 = [KI,..., Km,, Xm+aHl + [0] = m + 1. According to Theorem 2, the fact that rank(_A, B,, ) rank(EA,,,, quently, the number of latent variables can be deduced as m = ran,(_A,,B.)) - = m +
+ 1 > 1 indicates the presence of the latent factor. Conse- m.
Example 3 As shown in Figure g there is one latent factor and two-dimensional states. The bottleneck sets are ((X13, X23.1), (X1s, X24, k)), and one possible pairs of minimal rank sets are (A), ((X12, X13, X14, X24). (X14, X24,
((X11, X12, X13, X23), (X13, X2s, X24, X2s)) and (A2. B2) 124,X,, X25, 15,xX, X26)). According to the causal graph, the minimal configu- ration t-separate A from B, is either (1K, Xia, X23), 0) or (0, {re, X13, X23)). Consequently, the rank of the covariance matrix is rank(2 (EA,,B,) B,) = HR. X1a, X23) + 00 =3 3. Similarly, the minimal configuration
t-separate A2 from B2 is either ((K, X14, x24) 0) or (0, {K, X14, (24
resulting in rank (Ag, Bg) = Kr, X11, X24H + [0] = 3. According to Theorem 2, the fact that rank(Y A, B,) rank(2 AA B, = 3 >> 2 indicates the presence of the latent factor. Consequently,
the number of latent variables can be deduced as m = rank(2A,, B - 2 = 3 3-28 2 =
Example As shown in Figure there are two latent factors and two- two-diment dimensional states. The bottleneck sets are ((X11, X21. k1, K2), (X15. X25, K1, 12)), and one possible pairs of minimal rank sets are (A1. B2) ((X11, X12. X13, X11, X21). (X11, X21, X25, X26. X27)) and (A2,B2)
((122 X13, X14, X15, X25), (X15, X25, X25, X27, X2s)). According to the graph, the minimal con- figuration to t-separate A₁ from B, either (fx1, R2, X14, X24). 0) or (0, (N1, K2, X14, X24)). Consequently, the rank of the covariance matrix is rank(EA,
= [(11,, K2, X14, X24)1 + [0] 4. Similarly, the minimal configuration to I-separate A₂ from B2 is either ((K), K2, X15: X25) J 0) or (0, {K1, , X15, X25 }) with rank B2) = KKI, K2, X15, X25 + |0] = 4. According to Theo- rem 2, the fact that rank(EA,,BB B,) = rank(CA,, = 4 >> 2 indicates the presence of the latent factor. Consequently, the number of latent variables can be deduced as rank(EA,,B) 2 4 - 2
C.4 Related Definitions of Theorem 4.3
C.4.1 Covariance Matrix of Random Vector
In this discussion, we introduce the concept of the covariance matrix within the framework of latent variable models. By examining the properties of the covariance matrix, such as its rank, we are able to identify signs of latent variables-rank deficiencies, which serve as a measure of the cardinality of the minimal set of latent variables required to explain the observed dependencies. Such rank deficiencies indicate the presence of latent variables that extend beyond the observable scope.
Consider a directed acyelic graph (DAG), denoted as G, whose vertices V(g) form the set [m] := (1, .,, m.) Each node i in G is associated with random variable X; and an independent error term 6 ~ (0, σx) with di >> O. The DAG structure imposes a recursive relationship among the variables, where the value of X, can be expressed as a linear combination of the variables X; of its parent vertices pa(j), alongside the error term E, and regression coefficients X that correspond to the edges i + j in G:
X, =
Σ XiyXi +ej. jEpa(j)
(18)
where pa()) denotes the set of parent nodes of vertex j. where a parent node is one that has an edge leading to in G. From this recursive sequence of regressions, one can solve for the covariance matrix Σ of the jointly normal random vector X, which is defined as follows.
Definition C.3 (Covariance Matrix of Random Vector 688
The covariance matrix of the random
vector is given by the matrix factorizaionn
A=A-T4N 4A
(19)
22

---

where matrix & is defined as diagonal matrix with the variances of the error terms as its diagonal elements: diag(0) ....... 6m) The matrix M is an m m upper triangular matrix where Mij >ij if + j is an edge in G, and Maj 0 otherwise. Thus the matrix A is defined as A=l M, where I the m x m identity matrix.
Specifically, given two subsets A, [m], CA.B (aab))a is defined as the submatrix of covariance with row index set A and column index set B.
C.4.2 Trek and Trek Separation
The concepts of Trek and Trek Separation precede crucial need to address the presence of latent variables and intricate dependency structures that are not directly observable. The Trek represents a particular path that interconnects variables within graph, even if they not directly linked, while Trek Separation delineates a criterion to ascertain whether two sets of variables are independent, conditional on a set of other variables. Below, give the formation definitions of these two concepts.
Definition C.4 (Trek (68). A trek in G from to is ordered pair of directed paths (Pi, P2) where Pi has sink i, P2 has sink j. and both P1 and P2 have the same source k. The common source is called the top of the trek, denoted top (1, P2). Note that one or both of Pi and P2 may consist of a single vertex, that is, a path with no edges. A trek (P1, P2) is simple if the only common vertex among P1 and P2 is the common source top(P1, P2). We let (i. and 5) denote the sets of all treks and all simple treks from to J. respectively.
Lel A, B, CA and CB be four subsets of V (g) which need not be disjoint. We say that the pair (CA, CB) trek separates (or 1-separates) A from B ifr every trek (P1, from a vertex A to a vertex B, either P1 contains vertex C or P2 contains vertex in CB.
Definition C.5 (Trek Separation 688
C.4.3 Conditional Independence Skeleton
The Conditional Independence (CI) skeleton in graphical models refers to a structure that represents the conditional independence among observed variables. The CI skeleton can be used to infer the existence of latent variables. If the observed data suggests dependencies not represented in the CI skeleton, it may indicate hidden factors at play. The formal definition is given follows.
Definition C.6 (Conditional Independence Skeleton [10). A CI skeleton of X is an undirected graph where the edge between X, and X2 exists if and only if there does not exist a set of observed variables C such that X1, X2 & C and X1 1L C2
D Background
Reinforcement Learning In RL, an agent learns to make decisions by interacting with the envi- romment. The agent receives rewards for taking actions in the environment and uses this feedback to learn optimal behavior. It is often modeled as Markov Decision Process (MDP) represented by tuple (S,A, P. 7) where S denotes finite set of states representing different situations an agent might encounter, A a finite set of actions representing different decisions an agent can make, state transition function defining the probability of transitioning a new state s' given current state and action a, denoted as P(s'|ss a). It a reward function assigning a scalar value to each state-action pair (s, a), representing the immediate reward received after performing action in state y [0, 1] is the discount factor, representing the agent's 's consideration for future rewards. The agent's 's goal is learn an optimal policy π², which defines the optimal set of actions in different states to maximize the expected cumulative discounted reward over the long run. Developing this optimal policy involves estimating value functions such as the action-value function, defined as Q"((s, a) Ex Ei= 7 R. = 8, Ag = a). which represents the expected reward of taking action a in state following policy и. The pursuit of optimal policy n= involves maximizing the value functions over all possible state-action pairs: πT" = arg max, QT (s,
Variational Autoencoder Variational Autoencoders (VABs) 42 are a class of generative models in deep learning, adept at unsupervised learning of complex data distributions. Rooted in the framework of Bayesian inference, VAEs are designed to approximate probability density functions of input data. The architecture of VAE consists of two primary components: an encoder (s (z) and decoder
23

---

(=(z))). The encoder maps input data to latent space, represented by probability distribution, typically Gaussian, with parameters j and signifying the mean and standard deviation, respectively. The decoder reconstructs the input data from sampled latent representation z.
The distinct feature of VAEs lies in their probabilistic approach. The encoder outputs parameters of latent distribution, from which a sample is drawn:
(0(2)²) (2(2;
(20)
The decoder then attempts to reconstruct the input from this latent sample. VAEs optimize the Evidence Lower Bound (ELBO) objective, which balances two aspects: the reconstruction quality and the regularization of the latent space. The traditional ELBO is given by:
ELBO (xl() [or]=() ([=((()\[)]lp(())
(21)
Here, the first term measures the reconstruction quality, while the second term, the Kullback-Leibler (KL) divergence, imposes a regularization by encouraging the latent distribution (a(2[x) ) to be close to prior p(z). typically assumed to be a standard normal distribution ((0, I). VAEs, through this optimization, are capable of generating new data points that are similar to the input data, making
them highly valuable in applications like image generation, denoising, and anomaly detection within the domain of unsupervised learning.
E Detailed Related Work
Applications Machine learning has been increasingly lever- aged to create highly individualized solutions in a variety of domains. In health and wellness, it supports tailored interventions to increase physical activity [86 (55), promote weight loss [66 IS). and improve adherence to diabetes management (86). For the elderly, personalized algorithms facili- tate technology adaptation and specialized care for specific conditions [30]. In the financial sector, machine learning improves the optimization of technical indicators and stock market predictions, making them more individualized [54). In education, Information and Communication Technology (ICT) uses machine learning to provide personalized education systems such adaptive e-learning systems [14] and individualized tutorial planning (8). The transportation sector benefits from indi- vidualized car-following control strategies designed for specific driver behaviors (67]. Furthermore, multimedia platforms, such as YouTube and TikTok, are enhancing the user experience by using reinforcement learning recommend video content tailored to individual preferences (5) 31]. These examples underscore the breadth of individualized machine-learnin applications, highlighting their transformative impact across various industries.
Individualized
Reinforcement Learning for Latent State- Transition Processes RL has witnessed significant advancements in recent years, particularly through the integration of latent variable models to capture the underlying dynamics of environments. A primary focus in this domain is learning
low-dimensional, latent Markovian representations from observed data 552 46
40
23
79
92 47
60 [7] 18 (8)) Common strategies for state representation learning include reconstructing the observations, learning forward models, learning inverse models. Additionally, prior knowledge, such as temporal continuity [80), is often used to constrain the state space. Numerous studies have proposed methods to estimate the underlying state-transition processes from high-ddmensiinal input
sequences [79 1 23 25 91 17 39 26). Leveraging these learned world models, agents can perform modell-asee RL or planning. Such methods typically encode structural constraints, ensuring the sufficiency and minimality of the estimated state representations from both generative and selection processes. Recently, there has been growing interest in estimating state- transition processes in the Some studies [88 63 address similar settings involving individual- -eiiidu factors. However, to the best of our knowledge, we have yet to identify systemic approach that provides clear identifiability guarantees for the state-transition processes in the presence of individual-specifie latent factors.
presence of latent confounders [58 53
75
76
3
63
Comparisons with Related Works Existing methods for modeling latent factors in Markov Deci- sion Processes (MDPs) often fail to provide identifiability guarantees, particularly for time-invariant latent confounders. For instance, Contextual MDPs (7 consider contextual variables that influence transition probabilities and rewards, but these variables are assumed to be partially observable, and their identifiability is not guaranteed. Factored Non-stationary MDPs [3 achieve identifiability
24

---

under specific structural assumptions but do not account for confounders that remain fixed over time.
Other frameworks, such as Block MDPs 89 and POMDPs [2], and Latent MDPs [48] consider latent states or spaces but lack comprehensive modeling of how these latent factors influence state transitions and fail to provide theoretical guarantees for identifiability. Additionally, some works that do provide theoretical guarantees [36, [3], primarily focus on time-varying latent variables, leveraging advances in nonlinear independent component analysis (ICA) to establish identifiability. While these approaches are effective in their specific settings, the identifiability of time-invariant latent confounders remains underexplored, despite its practical importance in applications such as personalized decision decision-making and policy adaptation.
In contrast these works, our method introduces a novel framework that explicitly models latent group factors influencing each state in the transition process. We provide theoretical guarantees of identifiability in both finite and infinite latent factor settings. Specifically, our approach ensures group-wise identifiability even under nonparametric transition processes and extends identifiability to infinite latent factors under appropriate assumptions. This establishes novel theoretical insights for learning state-transition processes with latent factors.
F
Experiment Details
F1 Evaluation Metries
Pearson Correlation Coefficient Pearson Correlation Coefficient (PCC) 8 is a statistical measure that quantifies the degree of linear relationship between two variables. It provides value between -I and I, where I implies a perfect positive linear relationship. -1 implies a perfect negative linear relationship, and 0 implies no linear relationship between the variables. The equation for calculating the Pearson Correlation Coefficient r between two variables X and Y is as follows:
n2E my) - ( ev)
(22)
Σ x) " ΣE
where n is the number of paired samples, Σ xy is the sum of the product of paired scores, Σ x and Σ y are the sums of the scores and y scores respectively, Σ x² and Σ y² are the sums of the squared x scores and y scores respectively.
Canonical Correlation Analysis Canonical Correlation Analysis (CCA) 32 is designed to identify bases for two sets of variables in order to maximize the mutual correlations between the projections onto these bases. Let X and Y be the two sets of observed variables. This algorithm starts by centering the columns of X and Y so that they have zero mean. Then the covariance matrices Cxx = X Cyy Y, and Cxy X Y are calculated. After that, the canonical correlations obtained by solving the following generalized eigenvalue problem: square roots of the eigenvalues À indicate the canonical correlations between the linear combinations of X and Y. The corresponding eigenvectors v and u = CXY are the canonical weights used to construct the canonical variables. Finally, the canonical variables of X and Y are U X and V = Yu, respectively, representing the linear combinations of the original variables that are maximally correlated. The correlation of the primary pair of canonical variables is the highest, followed by the secondary pair, and so on. When employing CCA as an evaluation metric, a higher canonical correlation indicates stronger and more relevant relationship between the recovered latent variable and the ground truth latent variable.
Cxy Cyxu = Av. The
To extend the capability of CCA for analyzing nonlinear relationships, Kerne Canonical Correlation Analysis (KCCA) 81 is employed in the experiment which uses kernel functions to map the original variables into higher-dimensional feature space. In work, KCCA is used an evaluation metric to validate that the recovered latent variable is meaningfully related to the ground truth latent variable, thus proving the relevance of the estimated representations. This allows for the capture of more complex, nonlinear correlations between the variables, thus potentially increasing the robustness and relevance of the relationships discovered in scenarios where linear methods fall short.
F.2 Dataset Descriptions
Synthetic Data Generation Processes In this paper, we created three synthetic datasets: Case 1 corresponds to finite latent factor that satisfies our assumptions, and Case 2 and Case allow
25

---

for multiple finite and infinite latent variables. The dimensions of states and actions are set to and 2. respectively. The actions taken are generated randomly, following a uniform distribution Uniform(O, 1). The noise term follows a mean-7ero Gaussian distribution. The mixing function corresponds to the post-nonlinear model [90], where fi represents the nonlinear effect, and f2 denotes the invertible post-nonlinear distortion on 8t, embodied by randomly initialized three-layer MLP with Tanh activation function. The data generation process follows:
1.- k,) E:))
(23)
PersuasionForGood The PersuasionForGood dataset reveals the mechanics of persuasion in the context of charitable giving. It contains 1017 dialogues from 1285 participants in which one participant, called the persuader (ER), tries to convince the other participant, called the persuadee (EE), to donate to charity. An example dialog is shown in Figure 11 All participants underwent personality assessments, which included detailed participant-level information such as demographics. Big Five personality traits, moral foundations, and so on, allowing for a multifaceted analysis of persuasion strategies and allowing us to use the labeled 32-dimensional personalities of each persuader as the ground-truth latent factor in our experiments.
We use this dataset evaluate the performance of our estimation framework. We compare the estimated factors to the documented personality traits of persuaders. By examining the interactions between participants with different backgrounds and personalities, we aim to identify underlying patterns that could create effective persuasive agents. Specifically, we use BERT embeddings to generate a (68-dimensional feature vector for each dialog utterance. This process starts with tokenization, segmenting words into smaller units. BERT then processes these tokens to produce contextual embeddings.
Speakker
Ulterracee
Extrovert
Ayrorable
Consiennton
Neuretic
Open
ER
Hello, How you?
4,4
4
EC
I'm good, how are you doing?
5
4.2
3.6
4,8
Very well. I'm fust up organizing info for my charity. Are you involved with charitie?
4.4
EE
Yee! I work with childrm who have terminal illsosses. What charity are involved in?
4.2
3.6
4.8
ER
That's greec! I help with Save The Children.
3.6
4.4
4.4
4
EB
Amazing! Working with kids the best. What do you do for Save the Children?
3.6
4.8
help mise donations volunteer time.
That's inportant. How you donations?
BR
By directly asking for aid. Do you currenlyy donate to your charity?
3.6
4.4
4.4
EE
Yes do, but I'ms happy to donate to yours ss well!
4.2
3.6
4.8
Wonderfill! Would you be will to donate $1 11.00 of your takk money to help Save the Children?
an intemational non-govemmental organization that promotes childran's rights, provides relief and helps support chikkren developing countries.
BR
3.6
4.4
4.4
Yes, I would be happy
3.6
4.8
Would be
No, I can do
3.6
4.8
ER
Thank you Can we make $1.50? These chikdre really need the ussistance.
3.6
4.4
4.4
EE
1150 sounds good then.
3.6
4,8
Why not $1.75 then? :)
can $2.00! Happy help.
4.8
ER
Thank you so much! Do you have more questions for me?
3.6
4.4
4
ER
Nope. Thank you!
3.6
4.8
Figure 11: A sample persuasive dialog between persuader (ER) and persuadee (EE) from the PersuasionForGoood corpus, along with the Big Five personality test scores, including Openness, Conscientiousnesss Extroversion, Agreeableness, and Neuroticism.
Pendulum The pendulum environment, provided by OpenAl Gym, classic control task used for the evaluation RL models. This environment presents continuous control task where the agent must learn control frictionless pendulum with the goal of swinging it to the highest point and keeping it in the inverted position. The pendulum starts at random position, and the goal is to bring a standstill at the inverted position with the least amount of effort. The system characterized by continuous action space, representing the torque applied to the pendulum's 's fulcrum. For a pendulum of length and mass m, subject to gravity and a control input u, the equations of motion can be described by the following second-order nonlinear ordinary differential equations 0 w, à f sin(O) + miz where 0 is the angle of the pendulum from the vertical upright position, and
26

---

w is the angular velocity of the pendulum. The state of the pendulum at any time can be represented as s, = [cos f,, sin 0,, wi), action represents the torque applied to the free end of the pendulum in the range (4 E [-2, 2]. and the reward function is defined as: r; = - - + 0. * wf + 0.001 * of).
The goal of RI algorithms is to determine an optimal control policy xr that minimizes the effort to swing and balance the pendulum upright, typically by minimizing cost function defined over states and actions. Each episode provides a continuous stream of observations, actions, and rewards, allowing the development and evaluation of algorithms capable of learning effective control policies in continuous action spaces. In academic studies, the Pendulum environment serves as a benchmark to investigate the effectiveness of RL algorithms in handling continuous control tasks.
HeartPole HeartPole provides straightforward scenario for assessing healthcare treatment, high- lighting the complex interplay between productivity, health, and decision-making strategies. It simulates a professional's quest for increased productivity and examines the long-term health impacts of short-term choices, such as insufficient sleep, and intake of coffee and alcohol. The states include alertness, hypertension, intoxication, time since last sleep, total elapsed time, and total work done.
A productivity function and a heart attack risk function are defined over these variables, rewarding incremental productivity while imposing a significant penalty for heart attacks. Every thirty minutes, the agent evaluates the current state and chooses from a of actions: work, drink coffee (which increases alertness and hypertension), drink alcohol (which decreases alertness while increasing hypertension and intoxication), or sleep (time-consuming but essential reduce hypertension and intoxication to maintain alertness).
Half Cheetah Half Cheetah is an integral part of the Mujoco physics engine, designed to simulate the agility and mechanies of cheetah through a 2D robotic model. This model consists of 9 body parts, including a torso, two front and two back thighs, shins and feet, connected by 8 joints to provide fluid motion reminiscent of a cheetah's 's natural gait. The primary goal for this robot is to achieve maximum forward speed while maintaining stability, mirroring the efficiency and speed of its biological counterpart.
The observational data in this environment includes both the position and velocity of each segment of the half cheetah, methodically ordered with all position data provided before velocity information. This systematic ordering allows for a detailed understanding of the dynamics of the robot at any given time. Actions within this simulation are defined by the torque applied to the joints, which directly affects its acceleration and motion patterns. The reward function for Half Cheetah is designed to encourage rapid forward motion and operational efficiency and consists of two main components: forward motion reward proportional to the increase in the robot's ' horizontal displacement over time and control cost penalty for unnecessary control effort and external force application. This reward structure is carefully designed to encourage the optimization of forward motion, with an emphasis on reducing control effort and mitigating forces that may interfere with the robot's streamlined motion.
F.3 Additional Experiment Results
Ablation Study: Variability in number of samples Here, we vary the number of samples to further verify this effectiveness. The data generation process is the same as Case 1, except that we change the number of samples to {100, 150, 200, 300, 500, 800, 1000). The comparison results shown in Figure 14 iidicate that our method can achieve consistently good recovery performance under different numbers of individuals. This further confirms that the identifiability of our framework is guaranteed by the mathematical relationship between the trajectory length and the number of groups, which is constrained by the sample sufficiency assumption under the conditions given in Theorem 4.1] Moreover, Figures [4(b)and [1[[)]show the successful recovery of the latent group factor, validated by high-frequency similarity and a remarkable PCC value, confirming the ability of our method to skillfully recover latent variables in practical pendulum tasks.
Ablation Study: Variability in initial states We conduct additional experiments to verify how the variability in initial states across individuals affects performance. The initial state distributions are defined with two types: normal and uniform. For the normal distribution, the means are set to [0, 1, 1] and the standard deviations are set to [1, 1], respectively. For the uniform distribution, the range for each dimension is defined with lower bounds [0, -1, 1] and upper bounds [1, 1, 1.5]. The experiment
27

---

results in Figure Tigure[]]show show that although the initial states have high variability, the estimated values of the latent factors corresponding the 200 individuals are ultimately highly classified and can be divided into 4 groups.
Consideration of transformer as encoder Our framework flexible enough to integrate various encoders and decoders, depending on the application tasks. To demonstrate this flexibility, we incorporated Transformers into our framework and conducted a comparative analysis against the existing models. The result, shown in [))c)) indicates that while both frameworks achieve identifiability, the Transformer-baseee encoder demonstrates faster convergence compared to our previous approach.
Ablation Study:
Added Experiment: Inventory Inventory management [69] is an important real-world problem that aims to keep inventories of goods at optimal levels to minimize inventory costs while maximizing revenue from demand fulfillment. We tested the performance of our algorithm on the inventory with state dimensions of 50, 100, and 200 and added additional baselines (8) Meta gradient RL, (9) Multitask RL. (10) Policy distillation, and (11) Non- Non-policy adaptation to verify the model. The experimental results in Figure 12 show that our framework outperforms other algorithms in terms of initial reward and final reward.
Added Experiment: AhnChemo AhnChemoEnv [59] is designed to simulate cancer treatment through chemotherapy, allowing realistic modeling of tumor growth and response to treatment. We create different groups with PK/PD variation. The experimental results in Figure 13 show that our framework outperforms other algorithms in terms of initial and final reward. Our method achieves the highest initial and final rewards compared to the baselines. Specifically, it shows a significant jump-start compared to non-policy adaptation, validating the effectiveness of our adaptation approach.
The meta-gradient method optimizes the hyperparameters of the learning algorithm by calculating the gradient of the learning process, allowing rapid adaptation to new tasks as they change. However, due to the continuous adjustment of learning strategies during training, it converges more slowly and the adaptation effect is less significant compared our algorithm. Multitask RL improves learning efficiency by sharing model strategies across different tasks. This requires first training policies on multiple tasks, which can be con-cconu (and even risky) during exploration. Moreover, identifying which new task corresponds to a previously trained task can be challenging. Our algorithm addresses this by estimating directly without requiring prior knowledge. Policy distillation transfers the knowledge of already trained teacher models to a student model, allowing the student to perform well across multiple tasks. However, this approach highly relies on the performance of the teacher models; insufficiently trained teacher models can negatively impact the final performance. Our algorithm does not depend on the source policy performance; subsequent policy optimization is based on the new environment, leading to better final performance.
F4 Training Details
The estimation framework is trained using AdamW optimizer for maximum of 200 epochs and early stops if the validation ELBO loss does not decrease for ten epochs. A learning rate of 0.001 and mini-batch size of 32 are used, We used three random seeds in each experiment and reported the mean performance with standard deviation averaged across random seeds. We used a machine with the following CPU specifications: 1 11th Gen Intel(R) Core(TM) i7-118005 1800H @ 2.30GHz with 16 logical processors. The machine has one GeForce RTX 3080 GPU with 32GB GPU memory.
G Impact Statement
Our work has significant impact on ethics, society, and future applications. We emphasize the importance of individualized policies in systems and advocate for a deeper understanding and respect for individual differences. Tailoring interventions to different individuals has the potential to improve user experience and outcomes in healthcare, education, and other areas. This approach avoids a one-size-fits-all policies. Our method can greatly improve individualized services, transforming the delivery of educational content, the management of healthcare, and the recommendation of products. This makes these services more effective and aligned with individual needs. However, the implementation of this method requires careful consideration of privacy and data security, as
28

---

11363
31303
2.303
Reward
Rewardd
25:00
Reward
Non-Polcy Adapation
Non-Pelicy Adapation
Non-Policy Adapation Distllationn
Policy Distilation
Po Distilation
Mulh
Mithtask
Moto Grodiant
Hota Grodknn
Mota Gradient
2009
2:8
2008
Episode
Episode
Episode
(a) Reward curve with d, = 50. (b) Reward curve with d, 100. (c) Reward curve with d, 200.
Figure 12: Results Inventory. We evaluated the performance of different methods under different state dimensions. Our algorithm scales well to high-dimensional cases and outperforms other baselines in terms of initial reward and final reward.
59.88
Initial Reward Reward
Meta Gradient
77 48.48
Multitask RL
Reward
Policy Distillation
6.93
68.03
Method
Adapation
Non-Policy Adaptation 12.7
52.1
Policy Diciilation
Meta Gradient
Proposed Method
33.17
904
Wue
Episode
Reward
(a) Accumulative reward curves.
(h) Initial and final reward under diffcrent henchmarks.
Figure 13: Results in AhnChemoEny. We evaluated the performance of our method against several baselines, including (1) meta gradient RI., (2) multitask RI,, (3) policy distillation, and (4) non-policy adaptation. Our method outperforms these benchmarks and achieves superior performance in terms of initial reward and final reward.
personalized systems require the collection and analysis of personal data. Maintaining user trust, preventing misuse, and ensuring ethical use of such data are of utmost importance.
H Estimation Framework Details
The proposed framework is customized based on the requirements of the identifiability theorems given in Section 4 We would like emphasize that our proposed framework differs from the traditional VAE and model-based RL in three main aspects: (1) Our framework uses a quantization layer to discretize the continuous latent representations. This mapping continuous latent repre- sentations to an embedding dictionary is well suited to the group determinacy requirement. (2) Our decoder reconstructs individualized state transition processes to simulate the data generation process, incorporating additional conditions as well as the estimated latent factor. (3) We further extract latent factors for each individual as additional information to facilitate individual policy learning. The detailed implementations of each component are summarized below.
Encoder For any individual m, the Conv 1D layer transforms input sequence s," using learned kernel filters. These filters slide over the sequence produce feature map, denoting the response of the filter at each position. Mathematically, the transformation by single filter in the Conv ID layer time is described as o, = a (W' s + b), where o, the feature map, W the kernel to be learned during training, * the convolution operation, s the input sub-sequence from time to t + H. where H is the size of the kernel. the activation function, and is the bias term to be learned during training. The layer may contain multiple such filters, each learning different features
29

---

Tos Idtert
Fdlee andd Mind
120
POC-0.96
PCC Value
Frequeency
2arrano Latter
R53 30)
15.0 17.5 Velue
20.
25
Thiiiin Epoch
15
150
35
200
Trual Latent
(a)
(b)
(c)
Figure 14: (a) PCC trajectory comparisons under di different numbers of individuals. (b-c) Successful recovery of the latent group factor in the Pendulum.
State Dimersioo
Pearson Correlation Confficient
State Dimeraaio
State Dimersion
Original Framework
Transformer Encoder
Epoch
(a) Initial states with high vari- (b) Estimated latent factor. ability.
(c) PCC trajectories comparison.
Figure 15: (a-b) Evaluation on variability in initial states. The estimated values are highly clustered into four classes. (c) Incorporating Transformers. The transformer encoder achieves faster convergence compared to the original framework.
of the input sequence. The resulting feature maps serve transformed representation 2m. which embeds the information about the latent group factor K.
As for the LSTM, let the hidden states and cell states of the LSTM time t denote as h4 and ct respectively. The final hidden state LSTM hy, after the sequential processing of the entire trajectory, serves the representative 2m that embeds the information about the latent group factor
Quantization Layer Let the output the encoder be a continuous latent representation denoted as 2m E IR, and define an embedding dictionary E consisting of G vectors, where each vector represents unique discrete category:
(e1, .,, EG), where e R. The quantized vector Rm is obtained by mapping 2m the nearest dictionary vector. The mapping can be expressed mathematically as Rm arg mine, E || e 22 Subsequently, the quantized output is the vector from the dictionary that is closest to the encoder output. Thus, the continuous representation is effectively mapped to discrete Rm by finding the nearest neighbor in the dictionary, aligning the representation learning with the discrete nature of the latent variable.
Decoder Suppose st-, and a!" as the true previous state and action, respectively. Let &, be the approximated latent group factor for the m-th individual. The inputs to the conditional decoder are combination the aforementioned variables: Input, (s (5--1, Rm)) The output of the decoder is the reconstructed next state, št, which is function of the decoder input: $, = De(Input,). The true subsequent state, which is defined as CReamn PRecon (s." ,, a!" R.m). The objective in this process is to optimize the decoder parameters to maximize the reconstruction likelihood max Crseon so that the reconstructed state is as close possible to the true next state St.
reconstruction likelihood measures how closely the reconstructed state matches
30

---

I
Algorithm
The pseudocode for the proposed algorithm is presented in Algorithm ! and Algorithm 2
Algorithm 1 Algorithm of Individualized Policy.
1: Input: {fEny) JM m= individualized environments; Encoder: encoder; Quantization: embed-
ding dictionary, Decoder: decoder; и: policy network
2: Output: {Rg 1iglg-1 estimated latent factor; (m 1m= optimized individualized policy
1M
3:
4: ## Main loop
5: Main(J Encoder, Quantization, Decoder, π)
6: Encoder, Quantization, Decoder, π ~ N(O, 1) # Randomly initialize the network
7: H +- {Tm) 1M # Collect individual trajectories by interaction with 3
8: for each individual m do
Capture the high-level representcatinss 10: Rm Quantization( (zm) # Vector quantization
9: 2m Eucoder(s"..))
for each state s in the trajectory do
12: 8,th Decoder(s"". a," Rm)) ) # Reconstruct the next state
end for 14: end for return
{ = PolieyTearning(H, )=1)))) # Optimize the individualized policies
16:
EncoderFunction(ss? 18: if encoder is Conv ID then
19: for each in $0.7 do
20: 0, +- Conv ID (m (
21: end for
22: else if encoder is LSTM then
23: Initialize hg" c
24: for each in shh do
25: h +- LSTM(M" :))
26: end for
27: end if
28: return 2m < Final output of Conv ID final hidden state of LSTM
29:
30: QuantizationFuncttoni...
31: Initialize E = {ei, Ce..,,......ee ), dmin 0o
for each e; in do
33: if 1m e:|12< dmin then
34: Update dmin and Km 4- 6
35: end if
end for
37: return &.
38:
39: DecoderFunctionss"" a )m))
40: Reconstruct state based on condition S. + Decoder(s"" a/= kmm)
41: return Reconstructed state $?"
42:
43: PelicyLearninguunction
44: for each individual m do
45:
Update policy input (s:; 8²) n1 (s Rmm; QM) Update training objective:
46:
47:
I [(0m) E [> 7'Q (Se, p (s Rymi 0H) 09)]
48:
Optimize p: for individual vn
49: end for
50: return Optimized individual policy 14
31

---

Algorithm 2 Training Process with Augmented ELBO Objective.
1: Initialize parameters of the Encoder and Decoder
2: Initialize weights a and
3: repeat
4: for each individual m do
5:
Compute encoded representation: Zm < Encoder(sh::
6:
Estimate -ndivicual-specific factor: Rm + Quantization (zm)
7:
Compute reconstructed state: st" -eecoder(s)- a &m
8:
Calculate Lecon E. || $/ 1|
9:
Calculate Lquant 2: [m. em.ill LCommit E; [| sg(em, em 111
10:
Compute extended ELBO objective: LELBO = LRroon + CeL + / B.oomit
11:
Update parameters optimize LELBO
12:
end for
until convergence
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We list the contributions and scope in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
See Section 4 and Appendix B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details in both the main content and appendix, with code link.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
The access to the data referred and code is linked.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ceiiionnoilccy ic for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are provided in the figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
35

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htppprs////uripss httpr//sscccpuullecccccccccceeea
Answer: [Yes]
Justification: Checked.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
36

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not related to work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All are referred.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
37

---

Answer:
[NA]
Justification:
Not related to our work.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not related to work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not related to work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "4hIxAFyS8n", "reviewer": "Reviewer_cEn6", "summary": "The paper titled \"Identifying Latent State-Transition Processes for Individualized Reinforcement Learning\" addresses the challenge of optimizing individualized reinforcement learning (RL) policies by focusing on latent individual-specific factors that influence state transitions. This is particularly significant in domains like healthcare and education, where individual differences can causally affect outcomes.\n\nThe primary contribution of the paper is the establishment of the identifiability of latent factors that drive individualized state-transition processes. The authors propose a novel framework called Individualized Markov Decision Processes (iMDPs) that incorporates these latent factors into the RL framework. The key advancements include:\n\n1. The paper provides theoretical guarantees for the identifiability of latent factors under both finite and infinite conditions, making it possible to distinguish different underlying components in state transitions.\n\n2. The authors develop a generative-based method to effectively estimate these latent individual-specific factors from observed state-action trajectories. This method employs a variational autoencoder with a vector quantization layer to discretize the latent space and ensure accurate factor estimation.\n\n3. The proposed method is empirically validated across various datasets, demonstrating its effectiveness in identifying latent state-transition processes and improving the learning of individualized policies. \n\n4. The paper outlines a two-stage approach for policy learning, where the estimated latent factors are used to tailor RL policies for individuals, thereby enhancing the policy’s adaptability to different environments and individuals.", "questions": "How effective is the policy adaptation process for new individuals compared to existing methods? Can the authors provide quantitative benchmarks or metrics to evaluate this process?\n\nHow does the variability in initial states across individuals impact the performance and identifiability of the latent factors? Are there specific strategies to handle high variability in initial states?", "limitations": "Privacy Concerns: The individualized nature of the proposed method implies that it relies on detailed personal data. This raises potential privacy concerns, especially in sensitive domains like healthcare and education. The authors should discuss how they plan to handle privacy issues, possibly suggesting the use of privacy-preserving techniques such as differential privacy.\n\nRobustness and Reliability: Another potential impact is related to the robustness and reliability of the individualized policies. In critical applications, such as healthcare, incorrect policy recommendations could have serious consequences. The authors should address its limitations from this angle.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of the Individualized Markov Decision Processes (iMDPs) framework is a novel approach to integrating latent individual-specific factors into reinforcement learning (RL). This approach goes beyond existing models by considering the fixed influence of these latent factors on state transitions, addressing a significant gap in the current literature.\n   - The paper establishes theoretical guarantees for the identifiability of these latent factors under both finite and infinite conditions. \n   - The use of variational autoencoders (VAEs) with vector quantization to discretize the latent space and estimate the latent factors is not surprisingly a straightforward application of these techniques in the context of RL.\n   - The experimental validation is thorough, with the proposed method tested on multiple datasets, including synthetic data and real-world scenarios like the Persuasion For Good corpus. The experiments are well-designed to demonstrate the effectiveness of the method in different settings, showcasing its robustness and versatility.", "weaknesses": "**Assumptions on Latent Factors**\nThe paper assumes that the latent individual-specific factors are time-invariant, which might not hold in all real-world scenarios. For instance, in healthcare, a patient’s condition can change over time, affecting the state-transition processes. Addressing this limitation could involve extending the framework to account for time-varying latent factors or providing a discussion on how to handle such scenarios.\n\n**Limited Real-World Validation**\nWhile the empirical validation includes synthetic datasets and a real-world dataset (Persuasion For Good corpus), the application domains are somewhat limited. The healthcare and education examples are discussed theoretically but not empirically validated with real-world data. To strengthen the paper, the authors could include experiments using real-world healthcare or education datasets to demonstrate the practical utility and robustness of their method in these critical applications. A possible healthcare platform is [1], where there are 4 environments with customizable PK/PD variation setups.\n\n**Policy Adaptation for New Individuals**\nThe approach for policy adaptation for new individuals involves initializing the policy based on the estimated group factor and fine-tuning it with new interactions. However, the paper lacks detailed evaluation metrics or benchmarks to assess the effectiveness and efficiency of this adaptation process. Including more quantitative analysis and comparisons with existing methods for policy, adaptation would enhance the understanding of the benefits and limitations of the proposed approach.\n\n### Actionable Suggestions\n1. add a discussion of time-varying latent factors in your limitation section\n2. Incorporate real-world experiments in healthcare or education to validate the practical utility of the method.\n3. Provide detailed evaluation metrics and benchmarks for the policy adaptation process.\n\n\nReferences\n[1] Luo, Zhiyao, et al. \"DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime.\" arXiv preprint arXiv:2405.18610 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Discussion of time-varying latent factors:** Thank you for pointing out this interesting scenario. It is indeed possible for the latent individual-specific factor to be time-variant in the considered problem. We believe our framework can be extended to handle time-varying cases, although establishing the theoretical identifiability is highly non-trivial. For instance, if we directly allow these latent factors to be time-varying, it seems hopeless to recover the latent variables, since each individual may have a specific latent factor. Therefore, further constraints would be needed. For instance, the theoretical identifiability might benefit from the constraint that the total number of possible values of the latent factors over time and across different individuals is finite. We hope this important extension can be achieved by researchers in the field by leveraging our framework and further considering the idea in [1].\n\n[1] Hu, Yingyao, and Matthew Shum. \"Nonparametric identification of dynamic models with unobserved state variables.\" Journal of Econometrics 171.1 (2012): 32-44.\n\n**Q2: Additional real-world experiments in healthcare:** Thank you very much for your suggestion and for providing the healthcare platform. In the updated manuscript, we validated our algorithm in the AhnChemoEnv environment in DTRGym [2] and created different groups with PK/PD variation. The experimental results (see Figure 1 in REBUTTAL.pdf) show that our framework outperforms other algorithms in terms of initial & final reward. Detailed analyses are provided in Q3.\n\n[2] Luo, Zhiyao, et al. \"DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime.\" arXiv preprint arXiv:2405.18610 (2024).\n\n**Q3: Evaluation and discussion on policy adaptation process:** We appreciate your suggestion. In the updated manuscript, we have added more baselines and further compared our algorithm against (1) meta gradient RL, (2) multitask RL, (3) policy distillation, and (4) non-policy adaptation. Additionally, we have included two evaluation metrics: (1) initial reward, which measures the initial performance benefited from policy adaptation, and (2) final reward, which measures the performance after the full training process. Please refer to REBUTTAL.pdf for the detailed results.\n\nOur method achieves the highest initial and final rewards compared to the baselines. Specifically, it shows a significant jump-start compared to non-policy adaptation, validating the effectiveness of our adaptation approach. The meta-gradient method optimizes the hyperparameters of the learning algorithm by calculating the gradient of the learning process, allowing rapid adaptation to new tasks as they change. However, due to the continuous adjustment of learning strategies during training, it converges more slowly and the adaptation effect is less significant compared to our algorithm.\n\nMultitask RL improves learning efficiency by sharing model strategies across different tasks. This requires first training policies on multiple tasks, which can be time-consuming (and even risky) during exploration. Moreover, identifying which new task corresponds to a previously trained task can be challenging. Our algorithm addresses this by estimating directly using $\\kappa$ without requiring prior knowledge.\n\nPolicy distillation transfers the knowledge of already trained teacher models to a student model, allowing the student to perform well across multiple tasks. However, this approach highly relies on the performance of the teacher models; insufficiently trained teacher models can negatively impact the final performance. Our algorithm does not depend on the source policy performance; subsequent policy optimization is based on the new environment, leading to better final performance.\n\n**Q5: Discussion on variability in initial states:** This is a great question! We can still establish the identifiability of the latent variable for each trajectory provided by each user. However, since each trajectory has a finite (usually pretty small) length, the estimated values might be noisy. The second step is to merge latent variables that should have the same value. A straightforward method is as follows: we first estimate the value of the latent variable for each individual and then see how they can be clustered together. We check whether they should be classified into the same or different groups by conducting clustering. \n\nA more principled way to address this issue has yet to be developed, and we have conducted additional experiments to verify the feasibility and report the results in Figure (a-b) in REBUTTAL.pdf. The initial state distributions are defined with two types: normal and uniform. For the normal distribution, the means are set to [0, 1, 1] and the standard deviations are set to [1, 2, 1], respectively. For the uniform distribution, the range for each dimension is defined with lower bounds [0, -1, 1] and upper bounds [1, 1, 1.5]. As you can see from the experiment, although the initial state has a high variability, finally the estimated values of the latent factors corresponding to the 200 individuals are highly classified and can be divided into 4 groups. \n\n**Q6: Impact on privacy issues \\& robustness and reliability:** In the updated manuscript, we have included a separate limitation section to discuss how we address these limitations. \n\n- Potential privacy risks: We could use de-identifying techniques and remove direct identifiers such as names and zip codes, apply masking techniques such as data perturbation, and use pseudonymization to replace private identifiers with artificial ones. Incorporating differential privacy techniques is also interesting and we will leave it as future work.\n\n- Robustness and reliability: We could provide an assumption checklist to help users determine whether our work is applicable to their specific scenarios, thereby avoiding misuse and improving reliability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HUGLtKVPhr", "reviewer": "Reviewer_ZPqL", "summary": "The authors of this paper establish the identifiability of latent state-transition processes in reinforcement learning (RL) and propose a practical method for learning these processes from observed state-action trajectories. The focus is on personalized reinforcement learning (RL), where different individuals may exhibit different state-transition processes influenced by latent individual-specific factors. The study introduces the concept of a personalized Markov decision process (iMDP) and provides theoretical guarantees for learning state-transition processes with latent factors. Experiments on multiple datasets demonstrate the effectiveness of the method in inferring these factors and learning personalized strategies.", "questions": "1. **Consideration of Transformer as Encoder**: In Section 5.1, \"temporal dependency from the sequential observations,\" the use of Transformers as the primary encoder was not considered. Supplementing with relevant experimental comparisons would provide clearer insights for readers.", "limitations": "The paper explicitly discusses its limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Innovation and Importance**: The paper addresses a significant challenge in RL—identifying latent state-transition processes influenced by individual-specific factors. This is crucial for optimizing personalized strategies in fields like healthcare and education.\n2. **Theoretical Contribution**: The authors provide theoretical guarantees for the identifiability of latent factors, which is a significant contribution to the field of personalized RL.\n3. **Experimental Validation**: The method has been experimentally validated on multiple datasets, showcasing its effectiveness in practical applications.\n4. **Modular Framework**: The proposed iMDP framework is modular, allowing for independent study and improvement of each component, which is beneficial for further research and practical applications.", "weaknesses": "1. **Broader Experimental Scope**: Although the experiments cover multiple datasets, including more diverse environments to test the generality of the findings would be valuable, such as testing in more complex high-dimensional RL environments.\n2. **Comparison with Other Methods**: A more comprehensive comparison with other state-of-the-art methods in meta-RL and multi-task reinforcement learning could strengthen the validation of the proposed method.\n3. **Discussion in Appendix**: Adding a discussion in the appendix about the differences and connections between iMDP and meta-RL, context-based RL, and multi-task reinforcement learning would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Broader experimental scope:** Thank you very much for your suggestion. In the updated manuscript, we validated our algorithm in the AhnChemoEnv in DTRGym [1] and inventory management tasks [2]. AhnChemoEnv is designed to simulate cancer treatment through chemotherapy, allowing realistic modeling of tumor growth and response to treatment. Inventory management is an important real-world problem that aims to keep inventories of goods at optimal levels to minimize inventory costs while maximizing revenue from demand fulfillment. We tested the performance of our algorithm on the inventory with state dimensions of 50, 100, and 200. The experimental results (see Figure 1 and Figure 3 in REBUTTAL.pdf) show that our framework outperforms other algorithms in terms of initial reward and final reward. Detailed analyses are provided in Q2.\n\n[1] Luo, Zhiyao, et al. \"DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime.\" arXiv preprint arXiv:2405.18610 (2024).\n\n[2] Sun, Yuewen, et al. \"ACAMDA: Improving Data Efficiency in Reinforcement Learning Through Guided Counterfactual Data Augmentation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.\n\n**Q2: Comparison with other methods:** We appreciate your suggestion. In the updated manuscript, we have added more benchmarks and further compared our algorithm against (1) meta gradient RL, (2) multitask RL, (3) policy distillation, and (4) non-policy adaptation. Additionally, we have included two evaluation metrics: (1) initial reward, which measures the initial performance benefited from policy adaptation, and (2) final reward, which measures the performance after the full training process. Please refer to REBUTTAL.pdf for the detailed results.\n\nThe comparisons suggest that our method achieves the highest initial and final rewards compared to the benchmarks. Specifically, it shows a significant jump-start compared to non-policy adaptation, validating the effectiveness of our adaptation approach. The meta-gradient method optimizes the hyperparameters of the learning algorithm by calculating the gradient of the learning process, allowing rapid adaptation to new tasks as they change. However, due to the continuous adjustment of learning strategies during training, it converges more slowly and the adaptation effect is less significant compared to our algorithm.\n\nMultitask RL improves learning efficiency by sharing model strategies across different tasks. This requires first training policies on multiple tasks, which can be time-consuming (and even risky) during exploration. Moreover, identifying which new task corresponds to a previously trained task can be challenging. Our algorithm addresses this by estimating directly using $\\kappa$ without requiring prior knowledge.\n\nPolicy distillation transfers the knowledge of already trained teacher models to a student model, allowing the student to perform well across multiple tasks. However, this approach highly relies on the performance of the teacher models; insufficiently trained teacher models can negatively impact the final performance. Our algorithm does not depend on the source policy performance; subsequent policy optimization is based on the new environment, leading to better final performance.\n\n**Q3: Discussion in Appendix:**  In the updated manuscript, we have added a section discussing the differences and connections between iMDP and meta-RL, context-based RL, and multi-task reinforcement learning, summarized below.\n\n- **Meta-learning** trains a learning model on a variety of tasks so that it can efficiently apply what it has learned to new tasks. Unlike our method, it does not assume a time-invariant latent factor, has no guarantee of identifiability, and does not provide a clear clue of adaptation. iMDP captures how an individual's belonging to a certain group affects their interactions within an environment, allowing for individualized policy adaptation. Moreover, iMDP provides a guarantee of identifiability and develops a corresponding estimation framework that potentially offers better interpretability.\n\n- **Contextual MDPs** consider the general contextual influence on transition probabilities and rewards. However, the context variables are assumed to be partially observable and do not guarantee the identifiability of the context variables. In our case, when the latent factors are finite, our method guarantees group-wise identifiability even when the transition processes are nonparametric. In the cases of infinite latent factors, identification could be achieved under proper assumptions. \n\n- **Multi-task RL** involves learning policies for a variety of tasks simultaneously. The goal of the agent is to perform well on all these tasks, which may have similar or different objectives. It often involves sharing information between tasks to improve learning efficiency and policy performance. Instead of focusing on policy optimization for all tasks, iMDP identifies latent individual-specific factors that implicitly influence the decision-making process. These factors indicate the unique properties of each individual, providing explanatory clues for policy adaptation.\n\n**Q4: Consideration of transformer as encoder:** We appreciate your insightful suggestion. In the revision, we have incorporated transformers into our framework to perform a comparative analysis against the models currently in use, and the results are reported in Figure (c) in REBUTTAL.pdf. Although both frameworks can achieve identifiability, the Transformer encoder can achieve faster convergence compared with our previous framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UQstuereX0", "reviewer": "Reviewer_t6CB", "summary": "The authors consider a problem where the underlying dynamical process is an MDP but with a time-invariant latent factor. The authors provide examples of such problems in the real-world. To solve these problems, the authors propose a new mathematical framework called Individualized Markov Decision Processes (iMDPs). Furthermore, the authors provide theoretical guarantees and an algorithm for learning these latent factors from observational data. The authors then demonstrate the performance of this approach on multiple datasets.", "questions": "1. How does this compare with meta learning or transfer learning or a POMDP where each individual is a \"slightly different\" environment? While the authors state that these models are different, in terms of learning and practical considerations, how do these frameworks differ?\n2. Does identifiability of these latent factors get influenced by the policy being followed to generate the data?\n3. Why is the individual-specific latent factor $\\kappa$ not included in the reward function?\n4. What does the term $\\mathbb{P}(s, a | u)$ imply in equation (1)? Since $u$ is not observed by the decision maker, how can its action $a$ be conditioned on $u$?\n5. $d_s$ and $d_a$ are not defined in line 110. Can the authors add these definitions?\n6. Is this learning online or offline? From Line 246 it appears that the authors only consider the offline case. However, their control part requires an online setup as mentioned in Line 335. Specifically is the learning of $\\kappa$ and optimal policy simultaneous?\n7. If an online setup was available, how would the authors propose to use exploration to better identify $\\kappa$? Will this exploration to learn $\\kappa$ be harmful/costly?\n8. Can the authors define the term \"post-nonlinear temporal\" model in Line 162?\n9. Can the authors define the covariance matrices $\\Sigma_{\\mathbf{A_i}, \\mathbf{B_i}}$ in Line 173?\n10. Is $n$ in Theorem 4.3 same as $d_s$ in Line 110?\n11. Is the iMDP assumed to be an infinite horizon setup?\n12. Why are the quantization loss and commitment loss defined after equation (4) the same? Also in Line 9 in Algorithm 2.\n13. Is the reconstruction loss differentiable? Can the authors explain as it involves a quantization step as well? Or does this term only compute the gradient with respect to parameters of the decoder?\n14. Is the policy adaptation step specified equivalent to augmenting the input given to the policy function or Q function with $\\kappa$?\n15. In the caption for Figure 3, it would be helpful if the authors could define the shaded regions around the curves.\n16. While considering RL baselines, why haven't algorithms for POMDPs that take the entire history into account been considered?\n17. Can the authors explain the motivation behind the ablation study starting in Line 317?\n18. What do the authors mean by the limitation about \"instantaneous causal influences\" in Line 389?\n19. Can the authors give more details in the proof of Theorem 4.2? (Line 678 onwards)\n20. In Line 678, is $n$ the number of state components?\n21. I could not follow the proof for Theorem 4.3. There are several concepts introduced in the appendix and often the proof refers to a concept explained later. It would be helpful if the authors could provide some motivation/intuition for these various constructions/concepts.\n22. In line 912, arg max is a set operator and hence $=$ should be replaced with $\\in$.\n23. In Line 7 in Algorithm 1, what is the policy used to collect the trajectory samples?\n24. Can the authors give the specific values for the dimensions and other architecture details used in their numerical examples?", "limitations": "The authors discuss the limitations of their work in the Conclusions section and in the Impact Statement section (Appendix H).", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I think the problem considered is a very relevant and useful problem. The authors propose a novel approach to solve this problem and present theoretical guarantees and analyses along with some empirical experiments to validate their approach. The paper is, for the most part, well written and clear.", "weaknesses": "The authors could have considered simultaneous estimation and control, which is often needed in real-world problems. Furthermore, in choosing baselines for comparisons, the authors have not considered RL methods that take into account trajectory history.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1:** Thanks for the helpful questions. Here is a brief summary of the differences. We also empirically compared our method with meta-learning and transfer-learning techniques and reported results in REBUTTAL.pdf.\n\n- **Meta-learning** trains the model on a variety of tasks so that it can efficiently apply what it has learned to new tasks. Unlike our method, it does not assume a time-invariant latent factor, has no guarantee of identifiability, and does not provide a clear clue of adaptation.\n\n- **Transfer learning** focuses on leveraging knowledge from one domain to improve performance in a new domain while facing challenges such as negative transfer. Our method can identify the latent individual-specific factors and realize individualized decision-making with interpretability. \n\n- **POMDPs** focus on dealing with incomplete information and making decisions under uncertainty with time-varying hidden states. Our work emphasizes individualized decision-making considering latent factors that influence state transitions with fully observable states.\n\n**A2:** In the current theoretical treatment, yes, the identifiability is influenced by the policy. According to our proof technique, this assumption is needed. Thank you for pointing it out and we have made this condition explicit in the updated paper. At the same time, it might be possible to avoid this assumption although the proof seems to be nontrivial.\n\n**A3:** In our setting, we assume that different groups have no different reward weightings. Thus $\\kappa$ only influences the states and transition dynamics.\n\n**A4:** We believe this might be a misunderstanding. $\\mathbb{P}(s,a|u)$ represents the joint distribution of $(s,a)$ for each individual. Here, $u$ denotes different individuals which is explicitly observed. This notation is used for mathematical convenience.\n\n**A5, A10, A20:** Both $n$ in Theorem 4.3 and $d_s$ in line 110 represent the state dimension, and $d_a$ denotes the action dimension. We didn’t find $n$ in line 678–perhaps you referred to another line number? If you mean the $n$ in lines 166, 728, or 739, it also represents the state dimension.\n\n**A6, A7:** The estimation network is pre-trained offline. When a new individual arrives, we estimate $\\kappa$ and adapt the policy simultaneously, and the policy is adapted through new interactions. Such exploration is necessary to better identify $\\kappa$ and discover the optimal policy in RL. This has been provided explicitly in the updated paper.\n\n**A8, A9:** The definition of the post-nonlinear temporal model and covariance matrices are provided in Appendix C.1 and C.4.1, respectively. In light of your question, we have also moved them to the main paper in the updated paper.\n\n**A11:** Yes, we assume an infinite horizon setup in iMDP.\n\n**A12, A13:** This might be a misunderstanding. The quantization loss and the commitment loss are different and serve unique purposes. The key is in the positioning of the stop gradient (sg[·]) [1], showing how these losses influence the encoder and the codebook differently. The stop gradient allows the model to handle the non-differentiable quantization step, ensuring that gradients are computed for the encoder and the rest of the model, making the reconstruction loss differentiable.\n\n[1] Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" Advances in neural information processing systems 30 (2017).\n\n**A14:** Yes, we identify and incorporate $\\kappa$ into policy function to learn individualized policy.\n\n**A15:** The shaded regions represent the standard deviation. We have added an explanation in the revision.\n\n**A16:** Our problem and POMDP address different settings. We assume fully observable states influenced by latent individual-specific factors, rather than partially observable states. Under the similar setting, we compared our method with meta-learning and transfer learning techniques. \n\n**A17:** The ablation study analyzes the contributions of different components in the framework. The sequential encoder aligns our framework with the theorem requirements, and the conditional decoder meets our data generation process. The noise estimator is used to increase robustness.\n\n**A18:** Our current framework only considers transitions from time $t$ to $t+1$. However, in some cases, there may be causal influences that happen instantaneously within the same time step, which is beyond the scope of our current discussion.\n\n**A19:** Individuals sharing the same latent factor can be grouped due to similar state transitions. For a specific state-action pair $(s^*, a^*)$, the next state $s'$ should be consistent across individuals in the same group. We define $t^j$ as the time of the $j$-th occurrence of $(s^*, a^*)$ and collect states $X^j$ = {$s_{t^j+1} \\mid t^j = 1, 2, \\dots$}. By comparing $X^j$ across different individuals, we group those with similar transition behaviors and achieve identifiability by Lemma B.1.\n\n**A21:** We appreciate your feedback–indeed, this part is dense and involves many concepts and derivations. The proof for Theorem 4.3 has two parts.\n\n- Structure Identifiability: Lemma C.1 shows that the rank of the sub-covariance matrix of the observed variables equals the number of minimal choke sets, which represent the minimal sets that separate the two observed variable sets forming the sub-covariance matrix. The difference between the calculated rank and the expected choke factors indicates the number of latent factors.\n\n- Parameter Identifiability: We estimate $\\alpha$ and $\\beta$, which describe the influence of states and actions, using a regression model. The latent coefficient $\\lambda$ is identified using orthogonal techniques in factor analysis.\n\n**A22:** We have revised the notation accordingly.\n\n**A23:** The trajectory samples are collected using a random policy.\n\n**A24:** The summary of architecture details is provided in REBUTTAL.pdf and more details have been added in the updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h0oOkQIZt1", "reviewer": "Reviewer_pu6U", "summary": "The paper addresses the challenge of identifying individualized state-transition processes in reinforcement learning (RL) when individual-specific factors are latent. The authors propose a novel framework, the Individualized Markov Decision Processes (iMDPs), to incorporate these latent factors into the state-transition processes. They provide theoretical guarantees for the identifiability of these latent factors and present a practical method for learning them from observed state-action trajectories. Experiments on various datasets demonstrate the effectiveness of their method in identifying latent state-transition processes and optimizing individualized RL policies.", "questions": "1.How does the proposed framework scale with larger datasets and more complex environments? Are there any computational limitations or bottlenecks that need to be addressed?\n2. Can the framework handle real-time adaptation in dynamic environments where latent factors might change over time? If so, how would this be implemented?", "limitations": "1. The framework assumes that the latent individual-specific factors are time-invariant. In real-world scenarios, individual-specific factors can evolve over time, which this model does not account for.\n2. The framework might face challenges in scaling to environments with very high-dimensional state and action spaces, which are common in some real-world applications.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  Theoretical guarantees for identifiability under various conditions. \n2.  Effective practical method for learning individualized RL policies from observed data.\n3. Comprehensive experiments on synthetic and real-world datasets demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The method might be challenging to generalize to all types of RL problems, especially those with instantaneous causal influences within states.\n2. Some sections could be simplified for broader accessibility, and future work should address the limitations mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The method might be challenging to generalize to all types of RL problems, especially those with instantaneous causal influences within states.**\n\n**A1:** Thank you for pointing out this scenario. It is indeed possible that there are instantaneous causal influences within states in the considered problem. However, even with these influences, our framework can still identify the latent individual-specific factors, and the theoretical results will not be affected. In the instantaneous case, since the states are observed, we can treat the states at each time step as a whole and input them into the current estimation framework. Thus the instantaneous causal influences within states will not affect our current estimation framework and theorem results.\n\n**Q2: Some sections could be simplified for broader accessibility, and future work should address the limitations mentioned.**\n\n**A2:** Thank you for your suggestion. In the updated manuscript, we have reorganized the sections to improve accessibility and clarity. Additionally, we have added a separate Limitations section to discuss how we address limitations, including instantaneous causal influences. \n\nFuture work could extend our current framework from a causal perspective. For instance, we could develop modules that explicitly account for causal dependencies within states, using techniques such as causal graphical models and advanced inference methods that handle instantaneous causal relationships [1]. Such extensions could be directly integrated into our current framework, and we have been working on this problem.\n\n[1] Li, Zijian, et al. \"On the Identification of Temporally Causal Representation with Instantaneous Dependence.\" arXiv preprint arXiv:2405.15325 (2024).\n\n**Q3: How does the proposed framework scale with larger datasets and more complex environments? Are there any computational limitations or bottlenecks that need to be addressed? The framework might face challenges in scaling to environments with very high-dimensional state and action spaces, which are common in some real-world applications.**\n\n**A3:** In the updated manuscript, we validated our algorithm in the AhnChemoEnv in DTRGym [2] and inventory management tasks [3]. AhnChemoEnv is designed to simulate cancer treatment through chemotherapy, allowing realistic modeling of tumor growth and response to treatment. Inventory management is an important real-world problem that aims to keep inventories of goods at optimal levels to minimize inventory costs while maximizing revenue from demand fulfillment. We test the performance of our algorithm on the inventory with state dimensions of 50, 100, and 200. The experimental results (see Figure 1 and Figure 3 in REBUTTAL.pdf) show that our framework outperforms other algorithms in terms of initial reward and final reward.\n\nSpecifically, our method shows a significant jump-start compared to non-policy adaptation, validating the effectiveness of our approach. Unlike the meta-gradient method, which requires continuous adjustment of learning strategies during training and thus converges more slowly with less significant adaptation effects, our algorithm is more efficient. Additionally, training across multiple tasks in multitask RL can be time-consuming, and identifying which new task corresponds to a previously trained task can be challenging. Our algorithm addresses this by directly estimating $\\kappa$ without requiring prior knowledge. Furthermore, policy distillation heavily depends on the performance of teacher models; insufficiently trained teacher models can negatively impact the final performance. Our algorithm, however, does not rely on source policy performance. Instead, it optimizes the policy based on the new environment, leading to better final performance.\n\nOur findings suggest that while the algorithm scales reasonably well, certain computational limitations, such as increased processing time and memory consumption, need to be addressed. In high-dimensional scenarios, the use of hardware accelerators such as GPUs, as well as fine-tuning the model, is essential to maintain optimal performance.\n\n[2] Luo, Zhiyao, et al. \"DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime.\" arXiv preprint arXiv:2405.18610 (2024).\n\n[3] Sun, Yuewen, et al. \"ACAMDA: Improving Data Efficiency in Reinforcement Learning Through Guided Counterfactual Data Augmentation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.\n\n**Q4: Can the framework handle real-time adaptation in dynamic environments where latent factors might change over time? If so, how would this be implemented? The framework assumes that the latent individual-specific factors are time-invariant. In real-world scenarios, individual-specific factors can evolve over time, which this model does not account for.**\n\n**A4:** Thank you for pointing out this interesting scenario. It is indeed possible for the latent individual-specific factor to be time-variant in the considered problem. We believe our framework can be extended to handle time-varying cases, although establishing the theoretical identifiability is highly non-trivial. For instance, if we directly allow these latent factors to be time-varying, it seems hopeless to recover the latent variables, since each individual may have a specific latent factor. Therefore, further constraints would be needed. For instance, the theoretical identifiability might benefit from the constraint that the total number of possible values of the latent factors over time and across different individuals is finite. We hope this important extension can be achieved by researchers in the field by leveraging our framework and further considering the idea in [4].\n\n[4] Hu, Yingyao, and Matthew Shum. \"Nonparametric identification of dynamic models with unobserved state variables.\" Journal of Econometrics 171.1 (2012): 32-44.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time dedicated to reviewing the paper and valuable comments. We have revised the manuscript accordingly as described below. Concerns about the experiments are addressed collectively in the REBUTTAL.pdf. Your further feedback, if any, would be appreciated.

Meta-review: This paper demonstrates the identifiability of latent factors and introduces a practical method for learning these processes from observed state-action trajectories. Experiments across a few [somehow similar] domains confirm that the method can effectively identify latent state-transition processes and help developing individualized RL policies. While in general the paper's presentation is good, it heavily relies on theoretical results which are provided in the appendix. As such supplementary material is not reviewed here, I would strongly recommend that the authors double-check all the results, proofs, and examples in the appendix before camera-ready version.
Decision: Accept (poster)