Idiographic Personality Gaussian Process for Psychological Assessment

Authors: yehu  chen
muchen  xi
jacob  montgomery
joshua  jackson
roman  garnett
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 18:42:41

Abstract: We develop a novel measurement framework based on Gaussian process coregionalization model to address a long-lasting debate in psychometrics: whether psychological features like personality share a common structure across the population or vary uniquely for individuals. We propose idiographic personality Gaussian process (IPGP), an intermediate model that accommodates both shared trait structure across individuals and "idiographic" deviations. IPGP leverages the Gaussian process coregionalization model to conceptualize responses of grouped survey batteries but adjusted to non-Gaussian ordinal data, and exploits stochastic variational inference for latent factor estimation. Using both synthetic data and a novel survey, we show that IPGP improves both prediction of actual responses and estimation of intrapersonal response patterns compared to existing benchmarks. In the survey study, IPGP also identifies unique clusters of personality taxonomies, displaying great potential in advancing individualized approaches to psychological diagnosis.

Main paper: 
 introduction: 
 Building models for the evaluation of latent traits from observed responses is crucial to understand long-term behaviors through repeated quantitative assessments. These are used, for example, to study emotional stability after medical treatment or the development of academic ability during secondary education [1] [2] [3] . However, existing frameworks face several interrelated limitations. First, there are strong reasons to believe that standard taxonomies may over-generalize, failing to distinguish between related psychological phenomena that often differ in etiology, symptoms, and biological processes between individuals [e.g., 1, 4] , and this may lead to inaccuracy when making predictions [5] . A related issue is that measurement models are rarely individualized, instead assuming that (1) the correlation between latent traits of interest and survey responses are invariant across individuals and (2) the relationship between the latent traits themselves are the same for everyone. Lastly, current models are almost always developed for cross-sectional data that are collected only once from each respondent, which overlooks any potential dynamics of psychological processes.To address these limitations, previous research has adopted three different approaches, each inadequate in its own way. First, recent work has proposed an idiographic approach that builds a completely distinct taxonomy for each individual [6] [7] [8] . However, complete personalization can sacrifice generalizability and interpretability for clinicians, as any possible population commonality is ignored. A second line of research focuses on building dynamic psychometric models of time-series data via some variant of item response theory [9, 10, 3] , longitudinal structural equation modeling [11] [12] [13] , vectorized autoregression [14, 15] and/or Gaussian process (GP) latent trajectories [16] [17] [18] . However, all these models adopt the nomothetic approach, assuming that the responses of all individuals share an identical latent structure. Finally, there is a smaller body of work that adopts intermediate approaches to create individualization while maintaining group commonality [e.g., 19 ]. However, prior research models quantitative responses directly, ignoring the latent structures that are often the actual focus of domain researchers.In this work, we propose an idiographic personality Gaussian process (IPGP) framework for assessing dynamic psychological taxonomies from time-series survey data. This framework combines nomothetic and idiographic approaches by employing a common structure to explain typical circumstances, while allowing individual structures to accommodate deviations into distinct forms. We leverage the Gaussian process coregionalization model based on multi-task kernels to conceptualize responses of grouped survey batteries, adjusted to non-Gaussian ordinal data, and utilize IPGP for hypothesis testing of domain theories. Methodologically, our approach combines Gaussian process latent variable models (GPLVM) [20] , Gaussian process dynamic systems (GPDM) [17, 18] and GP ordinal regression for Likert-type survey data [21, 22] . Despite involving latent variables and GPs, IPGP differs from GPLVM in two ways: (1) it optimizes the factor loading matrix while marginalizing the latent variables, and (2) it accommodates categorical data using a non-Gaussian ordered probit likelihood. Computationally, our framework exploits stochastic variational inference for latent factor estimation, contrasting with other GP measurement models relying on Gibbs sampling that may not scale efficiently to intensive longitudinal setups [18, 23] .To our knowledge, our work presents the first multi-task Gaussian process latent variable model for dynamic idiographic assessment. While multi-task GPs have found recent applications in areas such as causal inference [24] [25] [26] , environmental science [27] [28] [29] , and biomedical research [30] , their potential remains largely unexplored in psychology. Current psychometric models typically focus on cross-sectional settings without dynamics [6, 31, 8] or single task settings that ignore inter-battery correlations [32, 33] . We conducted an extensive simulation study comparing IPGP against benchmark methods and analyzed an existing cross-sectional personality dataset. Our results demonstrate that IPGP simultaneously enhances the estimation of idiographic taxonomies and improves the prediction of responses. Additionally, we collected a novel IRB-approved longitudinal dataset. When applied to this data, IPGP not only shows superior performance in response prediction, but also suggests unique personality taxonomies. These findings highlight IPGP's significant potential for advancing individualized approaches to psychological diagnosis and treatment. 
background: 
 We start by laying out the ordinal factor model for building standard taxonomy from survey data [34, 35] . We then briefly discuss several existing idiographic longitudinal models in psychological assessment and review the Gaussian process model.Ordinal factor analysis. Consider the scenario where some set of units, i ∈ {1, . . . , N }, repeatedly answer the same set of j ∈ {i, . . . , J} survey items over t ∈ {1, . . . , T } periods with ordinal observations y ijt ∈ {1, . . . , C} up to C levels. For example, the responses could be Likert-typed, ranging from "strongly disagree" to "strongly agree". The latent factor model posits that the jth underlying latent variable f (i) j (t) for unit i at time t can be recovered as w T j x i (t), where x i (t) ∈ R D are unit-level latent factors and w j ∈ R D are factor loadings. The f (i) j (t)s are then mapped to ordinal responses via an ordered probit model:p y ijt = c | f (i) j (t) = f = Φ(b c -f ) -Φ(b c-1 -f ) with threshold parameters b 0 < • • • < b C .Usually b 0 and b C are fixed to -∞ and +∞ so that the resulting categorical probability vector sums to 1, while b 1 , . . . , b C-1 are allowed to move freely. Stacking x i (t)s, w j s and y ijt 's into matrices x, w and tensor y, the joint likelihood can be written as L(y | x, w) = i j t p y ijt | x i (t), w j . Identification is guaranteed with standard orthogonality and normalization constraints [36] . This factor model is also known as an item response model [37, 38] , and can be estimated via maximum likelihood, weighted least squares, or an EM algorithm [39] [40] [41] .Idiographic longitudinal assessment. In psychological assessment, the idiographic approach emphasizes intrapersonal variation by requiring distinct loadings, while the nomothetic approach identifies general interpersonal variation assuming shared factor loadings [42] . In terms of data collection, the idiographic approach usually surveys each individual multiple times (N = 1 and large T ) for learning personalized taxonomy rather than many individuals at a single shot (large N and T = 1). To extract individualized dynamics from time-series data, recent psychometric models have utilized longitudinal structural equations by explicitly specifying any intrapersonal and temporal dynamics. However, these typically require strong model-based assumptions from domain-theory about this dynamic process, and may be sensitive to model misspecification [11, 13] . Meanwhile, hierarchical vector autoregression models may automatically learn individual trajectories over time, but are designed to model observed responses directly rather than latent traits of interest to domain scholars [14, 15] .J D W pop K pop ⇒ J D * Z (i) K ind ⇒ Factor Loadings Cov Matrix K (i) time ∼ RBF T Factors x i (t) D ⇑ T y (i) J ⊗ ⊕ ⇓ ⇓ T f (i) J Ordinal ModelGaussian process. A Gaussian process (GP) can be used to define a distribution over f such that the values of f at arbitrary finite subset of X have a joint multivariate Gaussian [43] . A GP(µ, K) is specified with a mean function µ : X → R and a positive-definite kernel function K : X × X → R; evaluating these functions pointwise determines the mean and covariance of these finite-dimensional distributions. The most common kernel is the squared exponential (RBF) kernel K(x 1 , x 2 ) = exp (-1 2 x 1 T Px 2 ) with precision matrix P = diag(1/ℓ 2 1 , . . . , 1/ℓ 2 d ) and d = card(X ). The posterior of a GP can be derived analytically for a Gaussian likelihood but must be approximated in modeling latent variables with categorical indicators. We discuss the variational approximation method we use for inference in Sec. (3).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work is discussed in Sec (6) .Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 We propose an idiographic personality Gaussian process (IPGP) framework for assessing individualized dynamic psychological taxonomies from time-series survey data. Instead of joint estimation of latent factors and their loadings that cannot guarantee rotational and scaling invariance, we marginalize out the latent variables and focus on learning taxonomies of loadings. The overall architecture of IPGP is illustrated in Figure (1) , where the ordinal input observations across the indicators are modeled as ordinal transformations of latent dynamic GP with individualized RBF kernels and loading matrices.Typically in psychological assessment, survey questions are meticulously grouped such that each group gauges a particular latent trait (e.g. dimension of personality). Hence, we conceptualize the assessment of psychological traits as a multi-task learning problem, where each question represents a distinct task but can be correlated with other tasks. A multi-task GP is an extension of the single-task GP but for vector-valued functions [31] . To motivate the multi-task framework, first consider the two-task scenario with two T × 1 vector f 1 (i) and f 2 (i) denoting the latent temporal processes of unit i for question j = 1, 2. To fix the scale of latent factors, a time-level Gaussian process prior is placed on x i (t) ∼ GP(0, K (i) time ). By exploiting affine property of Gaussians, the induced joint distribution of vectorized [f 1 (i) , f 2 (i) ] T can be written as:EQUATIONwhose covariance of shape 2T × 2T contains four block matricesK (i)time scaled by different w T j w j ′ (j, j ′ ∈ {1, 2}). Specifically, w T 1 w 2 controls the inter-task covariance between these two tasks and w T j w j s (j ∈ {1, 2}) control their intra-task variance. This multi-task structure is also known as the linear model of coregionalization (LMC) [44] , which models output functions as linear combinations of several independent latent processes. In our case, each dimension in x i (t) represents one latent process, which jointly defines the functions as f j (i) = w T j x i (t). To extend this, letf (i) = [f 1 (i) , . . . , f J (i) ]T represents the flattened JT × 1 vector consisting of all J tasks. We write f (i) in a formal multi-task GP notation using Kronecker product ⊗:EQUATIONwhereK (i)task denotes the unit-specific task covariance matrix, consisting of the self inner products of D × J shared interpersonal loading W pop and D * × J unit-specific low-rank Z (i) (D * < D) for intrapersonal deviations that serves to be the additional idiographic component, independent of W pop .In our experiments, we found degraded performance for D * = 2 but extra computational costs so we focused on D * = 1. The Kronecker product ⊗ then multiplies each entry in the J × J task covariance with K (i) time , and returns the stacked JT × JT covariance for f (i) . Through the use of time kernelK (i)time , properties of the latent trait trends such as periodicity or autocorrelation could be incorporated. Here we use the common RBF kernel K (i) time (t, t ′ ) = exp -1 2 (tt ′ ) 2 /ℓ 2 i to account for dynamic changes in the latent attributes, whose bandwidth is determined by the unit-specific length scale ℓ i , but any other kernel can substitute RBF as practitioners see fit. Finally, the latent variables f (i) s are further projected to response space by the ordered probit model.Due to the non-Gaussian ordinal likelihood, we adopt the variational inference technique (VI) with inducing points introduced in [33] . Dropping superscript for demonstration, VI utilizes a variational distribution q(u) = N (µ u , Σ u ) on inducing variables u to approximate p(f | y) using the conditional p(f | u). Hence, the conditional log-likelihood log p(y | u) can be lower bounded by the expected log-likelihood w.r.t. p(f | u), after exploiting the non-negativity of Kullback-Leibler (KL) divergence between p(f | u) and p(f | y):EQUATIONFurthermore, a lower bound on model evidence (ELBO) can be obtained by combining Eq. ( 4) and an inequality derived by another KL divergence KL[q(u) ∥ p(u | y)] ≥ 0 (see Appendix B for details):EQUATIONwhere the KL divergence KL[q(u) ∥ p(u)] between the variational q(u) and prior p(u) can be computed in closed form as both distributions are Gaussians. The expectation of log-likelihood log p(y | f ) under the marginal distribution q(f ) = p(f | u)q(u)du is intractable but can be numerically approximated using Gauss-Hermite quadrature method. The variational parameters µ u and Σ u , individualized loadings w i and diag(v) as well as likelihood parameters {b c }s are then optimized to maximize this lower bound. Finally, the predictive likelihood of new p(y * ) = p(y * | f * )p(f * | u)q * (u)du is obtained by marginalizing out the optimized q * (u). Throughout our experiments, we adopt stochastic inference for computational scalability. Our IPGP framework also naturally facilitates downstream tasks such as domain theory testing between models with and without shared or idiographic components. We adopt posterior odds ratio test, using posterior p(M i | y) = p(y|Mi)p(Mi) i p(y|Mi)p(Mi) over a pool of models {M i } conditioning on observations y with prior weights p(M i ), as the hypothesis test to determine whether the latent structures for each individual are indeed distinct or are simply explainable by interpersonal commonality. Specifically, we refer the multi-task model in Eq. ( 3) as the idiographic model, and compare it with an nomothetic model without unit-specific components: K pop task = W pop W T pop . Note that compared to this baseline nomothetic model, our proposed idiographic model in Eq. ( 3) introduces additional unit-level JN loading parameters that enlarge the hyperparameter optimization space. Hence, we propose to first learn the interpersonal loading matrix W pop using the standard cross-sectional data from a nomothetic model that focuses on learning of population taxonomy, and then use the estimated W pop as informative prior in the full model. We will show empirically in Sec. (4) that with this stronger prior IPGP achieves a more precise estimation of individual taxonomies.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed the societal impacts of our work in Sec (6) . Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 We now evaluate IPGP in learning idiographic latent taxonomies and predicting actual responses against baseline methods from both psychometrics and Gaussian process literature in three experiments: a simulation study, a re-analysis of a large cross-sectional dataset, and a pilot study of repeated measures of the Big Five [45] personality traits.Setup. Our simulation considers longitudinal data of N = 10 units over T = 30 periods. We assume that the latent traits of each unit i have dimension D = 5, and each latent vector is generated independently from a GP x(d) i (t) ∼ GP(0, K (i)time ) with a unit-specific length scale uniformly randomly picked from ℓ (i) time ∈ [10, 20, 30] . We split J = 20 items into D subsets of size J/D = 4, such that each subset dominates one dimensional in the latent traits. Specifically, we set high value of 3 in the population factor loading matrix W pop for entries corresponding to the kth subset for dimension k, and low values drawn from Unif[-1, 1] otherwise. We also set each unit-specific loading w i from Unif[-1, 1]. To introduce sparsity and reverse coding, we randomly set half of the loadings to zero and invert the signs of the remaining half. Finally, we generate the y ijt s according to the ordered probit model with C = 5 levels, and apply 80%/20% splitting for training and testing. 2 (LOWER) 0.897 ± 0.004 -0.313 ± 0.010 0.884 ± 0.005 -0.334 ± 0.011 0.397 ± 0.007 5 (TRUE) 0.957 ± 0.002 -0.159 ± 0.005 0.942 ± 0.002 -0.184 ± 0.006 0.128 ± 0.006 8 (HIGHER) 0.957 ± 0.002 -0.161 ± 0.004 0.945 ± 0.002 -0.183 ± 0.005 0.124 ± 0.006 Metrics and baselines. We consider two sets of metrics for evaluation: (1) the in-sample and out-of-sample predictive accuracy (ACC) and log-likelihood (LL) of the actual responses, (2) the correlation matrix distance (CMD) between the estimated factor loading matrix and the true ones, which is defined for two covariance matrices [46] where f is the Frobenius norm. Note that CMD becomes zero if R 1 , R 2 are equal up to a scaling factor, and one if they are orthogonal after flattening. We compare IPGP to (1) various latent variable models for ordinal responses, including the graded response model (GRM) [37] , the generalized partial credit model (GPCM) [47] and the sequential response model (SRM) [48] , (2) Gaussian process dynamic model (GPDM) [17, 18] where the continuous predictions are rounded to the nearest ordinal level, (3) latent structural model (LSM) [13, 49] with trait-dependent latent variables and (4) time-varying vector autoregression (TVAR) with regularized kernel smoothing [15] . We also compare IPGP with several ablated models: (1) IPGP-NOM without the idiographic kernel, (2) IPGP-IND without the population kernel, (3) IPGP-LOW with lower-rank factors of 2 than actual rank of 5 in the synthetic setup and (4) IPGP-NP where the population kernel is learned from scratch rather than fixed to the informative prior. Note that W pop in the full IPGP model is fixed as learned from IPGP-NOM.R 1 , R 2 as d(R 1 , R 2 ) = 1 -tr(R1R2) ∥R1∥ f ∥R2∥ fResults. We use 100 inducing points and the ADAM optimizer with learning rate 0.05 to optimize ELBO for 10 epoches with batch size of 256. We repeat our simulation with 25 different random seeds using 300 cores on Intel Xeon 2680 CPUs. Table 1 shows the comparison of the average predictive accuracy, log-likelihood, and correlation matrix distance between IPGP and baselines and ablated models. Our IPGP model (indicated in bold) significantly outperforms all ablated models and baseline methods in estimated correlation matrix, predictive accuracy, and log-likelihood of both training and testing sets in paired-t tests. We found that IPGP is able to predict the correct labels due to its idiographic components and proper likelihood, since IPGP-NOM and IPGP-GL are two of the worst ablations for all prediction metrics. In addition, IPGP-IND and IPGP-NP have the worst correlation matrix estimation, implying that a well-informed population kernel is crucial in recovering the underlying factor structures.To assess IPGP's robustness to rank misspecification, we conducted additional exploratory factor analysis using our simulated data. We tested model performance with ranks of 2, 5, and 8, where 5 represents the true rank of the data. As shown in Table 2 , both the true-rank (5) and higher-rank (8) models significantly outperform the low-rank (2) model. However, increasing the rank beyond the true rank of 5 yields no additional performance benefits: the high-rank model is not significantly better than the true-rank model in a paired t-test. These results demonstrate two key points: first, low-rank approximations inherently lack the capacity to fully capture the underlying structure, and second, increasing the rank beyond the true rank provides no additional benefit. This underscores the importance of careful exploratory analysis in practical applications to determine the appropriate rank.We next validate the popular Big Five personality theory using standard cross-sectional data via a factor analysis, where a range of factors are tested and then compared according to model evidence. This serves to show that the model works appropriately to detect known latent traits even in nondynamic settings, and to validate the informative prior for the W pop matrix in our next experiment. We utilize an existing dataset called life outcomes of personality replication (LOOPR) [50] , which is collected from 5,347 unique participants on the Big Five Inventory [51] consisting of 60 questions. Our validation considers a range of latent trait dimension counts from D = 1, . . . , 5. For each dimension count, we first apply principal component analysis (PCA) directly on the correlation matrix Best model for each D is indicated in bold and the best model across different Ds is further indicated in italic. of the cross-sectional observations to learn a vanilla population factor loading matrix. We then initialize W pop in our model with this vanilla loading matrix, and optimize the loading matrix jointly with the variational parameters. Note that T = 1 in LOOPR, so we drop the idiographic components.ACC ↑ LL / N ↑ MODEL D = 1 D = 2 D = 3 D = 4 D = 5 D = 1 D = 2 D = 3 D = 4 D =Validation of Big Five. We further evaluated IPGP's performance through exploratory analysis, testing model ranks from 1 to 10. Results shown in Table 4 provide strong support for a rank-5 model, which achieves both higher model evidence and lower BIC, strengthening the evidence for the Big Five theory. The BIC follows a V-shaped pattern, decreasing as the rank approaches 5 and increasing thereafter, indicating that rank-5 represents an optimal balance point: ranks below 5 provide insufficient model capacity, while higher ranks lead to overfitting. These findings demonstrate that when analyzing psychological measurements from standard cross-sectional data, IPGP successfully identifies the correct underlying factor structure, making it valuable for downstream applications. Estimated interpersonal variation. Figure 2 compares the raw correlation matrix with our estimated Big Five correlation matrix. Both matrices exhibit a distinctive block pattern, characterized by strong correlations between questions within each Big Five factor and weak correlations across different factors. We also observe that questions related to negative emotionality demonstrate slight negative correlations with questions measuring extraversion and conscientiousness, suggesting the presence of meaningful trait-by-trait interaction effects.To further demonstrate IPGP in longitudinal setting for learning idiographic psychological taxonomies, we collected intensive longitudinal data using experience sampling measures (ESM). We highlight the predictive ability of IPGP through a prediction and a leave-one-trait-out cross-validation task. We also illustrate how IPGP identifies unique personality taxonomies that might advance individualized approaches to psychological diagnosis and inspire new theory.Data collection. We employed an experience sampling method (ESM) design where participants completed personality assessments six times daily over a three-week period, allowing for a maximum of 126 assessments per person. Our study included 93 valid student participants, yielding 8,770 total assessments with an average of 94 assessments per participant. We based our assessment on the BFI-2 [52] , which provides comprehensive coverage of the trait space and ensures proper identification of latent factors. While the original BFI-2 contains 60 items (four items for each of the three sub-factors within each Big-Five domain), we modified it for our ESM design by removing items unsuitable for contextualized assessment. To reduce participant fatigue and minimize learning effects from repeated measures, we implemented a planned missing design: participants randomly responded to two out of three items for each sub-factor, resulting in a streamlined 30-item assessment. Note that our data is collected from a student sub-population as non-representative samples, and future studies may explore the model's applicability across diverse populations.Comparison between nomothetic and idiographic models. We run the full IPGP model with idiographic component and unit-specific time kernel on the collected longitudinal data. Again we set the ranks of the population and individual loading matrices to 5 and 1 respectively, and incorporate prior knowledge of the cross-sectional data by fixing the population loadings as the Big Five loadings estimated in Sec. (4.2) and optimizing the individual loadings. We contrast our proposed idiographic model (IPGP) and baselines in Table 5 , which shows the in-sample prediction, averaged log-likelihood and posterior odds ratio. We found that IPGP outperforms IPGP-NOM with higher predictive accuracy and log-likelihood, and is decisively favored by a posterior odds ratio of exp(1.06 × 10 4 ). Predictive performance of IPGP. We also evaluate the out-of-sample performance of the idiographic and nomothetic models using two prediction tasks: forecasting future responses and leave-one-trait-out cross validation. For the forecasting task, we train both models with data from the first 40 days and predict future responses for the last 5 days. For the cross-validation task, we predict responses of each trait by training on data belonging to the other four traits, where 20% of responses for one trait was held out (randomly choosing the trait and items to remove). Figure (4) shows the predictive accuracy and log-likelihood of IPGP and IPGP-NOM for the forecasting task over varying horizons and for the leave-one-trait-out cross-validation task. IPGP has consistently better performance than IPGP-NOM in both tasks, except for being slightly less accurate in predicting extraversion. Overall, IPGP is favored than IPGP-NOM by posterior odds ratios of exp( 43) and exp(716) in these two tasks.Discovery of taxonomies. Despite our small cohort size (93 respondents), we also explore how the profiles of personality that substantially differ from the interpersonal commonality cluster into informative groups. Specifically, we first perform a k-mean clustering using all 93 estimated individual correlation matrix with CMD as the distance metric, and then compute the residual correlation between each estimated clustering centroid and the population correlation. Figure (3) illustrates four residual correlations identified by our k-mean clustering. Each heatmap displays the residual correlation at the trait level averaged between the corresponding batteries for one cluster, with darker red and blue indicating larger positive and negative deviations. For instance, agreeableness (A) is more correlated to extraversion (E) than population profile in the first profile, but less correlated to openness (O) in the second profile. Moreover, these two directions of deviations are even exacerbated in the third and fourth profiles.The discovery of unique personality taxonomies suggests a potential resolution to the longstanding idiographic versus nomothetic debate in personality and psychometric sciences. Our findings indicate that the optimal approach lies between these two extremes, rather than fully embracing either perspective. The four distinct profiles that we identified, while derived from the Big Five framework, demonstrate how individuals can meaningfully deviate from a common taxonomy. These deviations may provide valuable insights into individuals' motivations, behavioral patterns, and self-concepts. For example, individuals matching Profile 4 show a strong correlation between Extraversion and Agreeableness, possibly reflecting a tendency toward warm and socially engaging behavior (such as someone who naturally connects with and shows kindness to everyone at social gatherings). Furthermore, these distinct profiles enhance the predictive power of individual-level (N = 1) models by allowing them to learn from people with similar characteristic patterns. 
related work: 
 Idiographic assessment captures critical individual characteristics that are often lost in simplified taxonomies of psychological behaviors [53] . Evidence from multiple psychometric fields demonstrates that nomothetic models, which focus solely on interpersonal variation, often lack generalizability [1] . Researchers have proposed various solutions to address this limitation. Song and Ferrer [54] enhanced dynamic factor models with random effects to analyze psychological processes. Jongerling et al. [55] developed a multilevel first-order autoregressive model incorporating random intercepts to track daily positive effects across weeks. Beltz et al. [19] bridged nomothetic and idiographic approaches by introducing individual components to the group iterative multiple model (GIMME) for clinical data analysis. However, these methods share a common limitation: they model in response space rather than latent space when handling ordinal survey data.Gaussian process latent variable model (GPLVM) is a dimensional reduction method for Gaussian data, where the latent variables are optimized after integrating out the function mappings [20, 56] . Our proposed framework differs from GPLVM as we optimize the factor loading matrix while marginalizing the latent variables. In addition, our model contrasts GPLVM and (variational) Gaussian process dynamical model (GPDM) [16, 17] in our non-Gaussian ordered logistic observation model and the use of multi-task kernel for latent structure. Finally, our longitudinal framework with stochastic variational inference learning differs from the static GP item response theory (GPIRT) [23] with more computationally demanding Gibbs sampling.Longitudinal measurement models have gained prominence as researchers increasingly incorporate temporal dynamics into psychological theories through longitudinal survey designs [57, 58] . This development has spawned various methodological approaches. One family of methods includes longitudinal structural equation models (SEM), such as multiple-group longitudinal SEM and longitudinal growth curve models, designed for repeated measurement studies [11] . The Mplus software later extended these capabilities by implementing dynamic SEM with Bayesian Gibbs sampling [13, 49] .Another stream of research produced dynamic item response models [9, 10, 3] and time-varying vector autoregressive models [14, 15] to track latent trait trajectories. While multi-task Gaussian process time series have been successfully applied to Gaussian observations in behavioral research [18] and found applications in causal inference [24, 26] , environmental science [28] , and biomedical research [30] , they remain unexplored for survey batteries with non-Gaussian likelihood where exact inference is not possible. 
conclusion: 
 We introduce the idiographic personality Gaussian process (IPGP) model, a novel approach for personalized psychological assessment that learns intrapersonal taxonomies from longitudinal ordinal survey data, a configuration that remains underexplored in Gaussian process dynamic systems literature. Our model leverages Gaussian process coregionalization to capture the between-battery structure and employs stochastic variational inference to ensure scalable computation. Looking ahead, we envision extending IPGP to other psychological domains, such as emotion research, and enhancing it by incorporating contextual information about behaviors and activities.Our proposed IPGP framework also provides insight for domain theory testing, contributing to the substantive debate in psychometrics surrounding the shared versus unique structures of psychological features. Our experimental results show that IPGP is decisively favored over the nomothetic baseline, and substantive deviations from the common trend persist in for many individuals. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All assets used in this paper are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Throughout our notation, superscript (i) indicates unit and underscript j indicates task.• y (i) = [y 1 (i) , . . . , y J (i) ] T represents the flattened JT × 1 response vector consisting of all J tasks and T periods for unit i.• f (i) = [f 1 (i) , . . . , f J (i)] T represents the flattened JT × 1 latent vector consisting of all J tasks and T periods for unit i, mapped to response space via ordered-probit likelihood.• W pop is the D × J (D latent dimensions, J tasks) shared interpersonal loading matrix.• Z (i) is the D * by J unit-specific low-rank loading matrix that serves to be the additional idiographic component, independent of W pop .•K (i)task is the unit-specific task covariance matrix with shared component W T pop W pop and a low-rank approximationZ (i) T Z (i) of D * < D for unit-specific deviations. • K (i)time is the time covariance for dynamic changes in the latent attributes. We used an RBF kernelK (i) time (t, t ′ ) = exp -(t -t ′ ) 2 /ℓ 2 i with unit-specific bandwidth ℓ i s.W pop is a global parameter estimated for the entire population while Z (i) is a unit-level parameter. These are combined in Eq.( 3) to create a unique kernel for each unit that combines both components.We provide the full mathematical details of the evidence lower bound defined in Eq. ( 6). As KL divergence is always non-negative, we first consider the KL divergence between p(f | u) and p(f | y):EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONMoving E p(f |u) log p(y | f ) to the R.H.S of the above inequality will lead to Eq. ( 4). We then exploit the inequality given by KL[q(u) ∥ p(u | y)] ≥ 0:EQUATIONEQUATIONEQUATIONRearranging the above inequality, applying Eq. ( 4) and exploiting notation q(f ) = p(f | u)q(u)du leads to the ELBO:EQUATIONEQUATIONThe LOOPR dataset from Soto [50] in our first case study used Qualtrics and quota sampling to ensure representative samples of the U.S. population, and hence was very diverse:• age: 11% ages 18-24, 18% ages 25-34, 17% ages 35-44, 19% ages 45-54, 17% ages 55-64, 18% ages 65 and older • sex: 52% female, 48% male • race/ethnicity: 74% non-Hispanic white/Caucasian, 11% black/African American, 10% Hispanic/Latino, 3% Asian/Asian American, 2% American Indian/Native American • educational attainment: 10% did not complete high school, 33% high school graduate, 28% some college, 19% college graduate, 10% graduate or professional degree • annual household income ($): 14% <20,000, 12% 20,000-29,999, 11% 30,000-39,999, 15% 40,000-49,999, 26% 50,000-79,999, 22% 80,000+As social and personality psychology often faces challenges with model building due to reliance on non-representative samples, past researches rely heavily on student populations. We follow such practice but aim to broaden our sample within budget and available diversity. Participants in our longitudinal study (the third experiment) are mostly college students with an average age of 20.23 (SD = 1.94); 70% are female, 26% are male and 4% are self-identified as other; ethnicities self-reported as 42% Caucasian, 39% Asian, 12% African American, and 7% Other. We used the BFI-2 [52] , a widely used personality measure across cultures and ages. We acknowledge the typical biases of convenient sampling in higher education, including socio-economic and ethnic diversity limitations.Table 6 shows the average runtime of IPGP and its competitors in the simulation study. IPGP does require more time to train due to the enlarged model space. Justification: The main claims accurately reflect this paper's contribution and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: NOTE: One reviewer provided vague and highly formulaic criticism that they were unable or unwilling to defend during the discussion period; I am ignoring their review in making my recommendation.

This is a very interesting applied paper, that connects recent methodology on multi-output Gaussian processes to a long-standing debate in psychometrics on the whether "features like personality share a common structure across the population, vary uniquely for individuals, or some combination"---i.e., the "nomothetic" versus "idiographic" approach.

The paper (and authors) should be credited for introducing a novel longitudinal data set composed of survey responses from N=93 subjects who were asked to fill out a personality assessment six times per day for 3 weeks. This data seems very labor-intensive to collect, and pairs well with the proposed model and theoretical question.

There was back-and-forth between the authors and reviewers about the proposed model, whether it was novel outside its applied domain, and whether it can be said to contribute broadly to the area of GPs and GP latent variable models. The paper hedges in several places about the novelty of the model, generally saying that it is novel for this (fairly narrow) applied domain, and the authors did not clarify this in their responses. The authors did however agree in the the discussion period to walk back some statements in the paper about whether the proposed model advances general methodology in GPLVMs. All-in-all, the discussion period clarified that the technical novelty of the model itself should not be overstated, but that the model nevertheless represented a non-trivial and correct adaptation of current methodology to an interesting applied problem, and was a contribution to the literature in the sense of providing a new illustration and use-case of multi-output GPs.

The authors cleared up several other problems raised by the reviewers during the discussion period. The reviewers detected some sloppiness in the equations; however, the discussion revealed that these were all minor typos which were easy to fix, and did not impede understanding of the technical content. One reviewer pointed out a major flaw in the main applied experiment which purported to validate the "Big Five" personality trait by running the model with K=1...5, and showing that K=5 performed best; the flaw was that the paper did not report values of K beyond 5. However, in their response, the authors provided supplemental experimental evidence that showed that the model's performance did in fact decline for K > 5.

I am in favor of acceptance. From the strict perspective of advancing general-purpose GP methodology, this paper is not very strong. However, the paper stands as a good example of translational applied work. It encodes an ongoing theoretical debate in psychology into a non-trivial multi-output GP model and applies the model to a novel data set that was difficult to obtain and bears directly on the applied question. I think NeurIPS should welcome work like this.

In the event of acceptance, the authors should make the following revisions to their camera-ready version:
- Any statements about the connection to GPLVMs should be made more precise and edited in accordance to the discussion with reviewers.
- Any novelty statements about the proposed model should be made more precise and/or omitted. If the model is not novel generally, but novel to this particular application, then this should be stated clearly, and citations to papers that employ the same (or very similar) models to other domains should be cited in the related work section. 
- The typos in equations that reviewers pointed out should be fixed.
- The supplementary experiments reported during the discussion phase should be incorporated into the paper.

Reviews: review: paper_summary: paper_summary: UPDATE: I am updating my scores in light of the excellent authors' response.

This paper considers psychometric data composed of ordinal responses $y_{ijt}$, each of which represent how unit _i_ answered survey item _j_ during time period _t_. The key characteristic of this item-response data is that the same $N$ units are longitudinally surveyed on the same $J$ items over $T$ periods. 

The paper takes an ordered logit factor modeling approach to analyzing such data, wherein $f_j^{(i)}(t) = \mathbf{w}_j^\top \mathbf{x}_i(t)$ is the latent ideal point of unit _i_ on on item _i_ at time _t_, and $\mathbf{w}_j$ and $ \mathbf{x}_i(t)$ are the $K$-dimensional loadings and factor vectors, respectively.

The paper moreover advocates an "idiographic approach [which] emphasizes _intrapersonal_ variation by requiring distinct loadings $\mathbf{w}_j^{(i)}$" which are different for each unit $i$. It is able to effectively achieve this by exploiting the repeated measurements of each unit-item $(i,j)$ pair over different time periods $t$.

The proposed model is an instance of a multi-task Gaussian process (MTGP). Intrapersonal variation is modeled using a unit-specific kernel $K_{time}^{i}$. The full covariance is then $JT \times JT$, resulting from a Kronecker product of $\mathbf{K}_{\textrm{time}}^{(i)}$ (applied to input $T$ periods) with a low-rank matrix $J \times J$ matrix that represents covariance between survey items (or "tasks"). This all extends the previously-presented linear model of coregionalization (LMC), which was originally developed for the simpler case where observations of $(i,j)$ pairs are not repeated over time.

The paper derives a variational inference inference algorithm for the model, which follows closely from previous work.

The paper then reports three sets of experiments: 1) a synthetic study that reports excellent results on parameter recovery (when the true parameters are known), 2) a re-analysis study that purports to show that a non-dynamic (i.e., non-idiographic) version of proposed model is able to identify the correct factor structure (K=5) from survey data designed to assess the "Big Five" psychometric traits, and 3) an illustrative case study involving a novel longitudinal data set.
strength_weakness: strength_weakness: The paper is vague about its technical contributions. It makes the following hedged novelty statement: "the first multi-task GP latent variables model _for dynamic idiographic assessment_". I took this to mean that the proposed approach is not very new technically, but it has never yet been applied to dynamic idiographic assessments. However, the paper also makes the general claim that it "advances the literatures on Gaussian process latent variable models...". I am of the opinion that tailoring existing modeling frameworks to new applications does provide a technical contribution, as it contributes a new view and set of interpretations/metaphors that can help to better understand the abstract model. I think the paper does contribute in this way. But I am unsure if the paper further contributes more substantially to the area of Gaussian process latent variable models, as the paper is vague about that.

There is sloppiness in some of the main equations. For instance, equation 2 confuses a distribution with a random variable stating $p(\mathbf{f}^{(i}) \sim \cdots$, and I believe equation 3 confuses an inner with an outer product (shouldn't it be $w_i w_i^\top$?). There's also some confusing overloading of symbols between the background and the model sections. The background sets up the idea that an ideographic approach has distinct loadings $w_j^{(i)}$ different across $i$. But the proposed model itself does not seem to sport this as each $w_j$ is global for each task. I believe effectively the model does achieve an idiographic interpretation through the unit-specific kernel, but the connection between the background and model section is not clear.

I think the second experiment has some major flaws. The paper claims this experiment validates the "Big Five" because model performance peaks at $K=5$. However, the model is only fit for $K=1...5$. We can clearly see from the log likelihood numbers that model performance for IPGP is monotonically increasing in $K$ for the values considered, suggesting that it will likely be even better at larger values of $K$; if that were true, it would not validate the "Big Five" theory. As this is an applied paper, I would expect a much higher level of rigor on one of the two applied case studies.
questions: questions: As this is an emergency review, after the reviewer discussion period, I am not sure if the authors will be able to reply. But if they are able, I would just ask that they respond to the various points made in previous sub-sections.
limitations: limitations: This paper did involve a fairly intense human subjects experiment for data collection, but the paper reports it was IRB-approved. Psychometrics of this kind are fairly common; I do not think the paper introduces any new potential negative societal impacts.
paper_summary: paper_summary: 1. This paper introduces an innovative measurement framework utilizing the Gaussian process coregionalization model to resolve the question of whether psychological attributes such as personality exhibit a universal structure among the populace or are uniquely individualized. 
2. An Idiographic Personality Gaussian Process (IPGP), a hybrid model that accounts for both the commonality of traits across people and individual-specific "idiographic" variations. 
3. Gaussian process coregionalization model to interpret the responses from grouped survey batteries, adapted for non-Gaussian ordinal data, and employs stochastic variational inference for estimating latent factors. 
4. The application of IPGP on both synthetic data and an original survey demonstrates its performance.
strength_weakness: strength_weakness: 1. there are many MTGPs, the author ignored comparison with them.
2. incorrect description, such as: "multi-task structure is also known as the linear model of coregionalization (LMC)".
3. This represents a particular implementation of MTGP, but it doesn't introduce any novel methodological advancements.
4. The paper presents a valuable contribution, but its innovative aspects are limited, as it largely builds upon existing theories without introducing significant new insights.
5. The literature review appears somewhat limited. Expanding it to include more recent or relevant MTGPs could provide a more comprehensive context for the research.
6. The statistical analysis used in the study seems inadequate given the complexity of the data.
7. The research seems to be more of an incremental advancement rather than a novel contribution, which may limit its impact on the field.
8. The experimental evidence provided in the paper is not as comprehensive as it should be to support the claims made, suggesting a need for more extensive testing.
9. Only one experiment is related to psychological assessment.
10. No idiographic personality is discovered and presented in the paper.
questions: questions: Please see the weakness above
limitations: limitations: Please see the weakness above
paper_summary: paper_summary: Gives a multi-task/output GP formulation for multiple time-series (or intrinsic co-regionalisation model) for pyschological assessments. Design of the factor loadings informs the task-correlations and reflects the knowledge of individual's correlation between his responses and inter-person correlations. More of an application paper rather than a paper advancing machine learning; but a good application nonetheless.
strength_weakness: strength_weakness: 1. The paper is weak from the perspective of advancing state-of-the-art machine learning algorithms.
2. The paper is totally unrelated to GPLVM and tasks that GPLVM are designed for. Mentions of GPLVM only confuses the reader.
3. Equation 3 needs fixing.
4. In section 3.2, the method is variational inference, and not stochastic variational inference.
5. Description of the setup in section 4.1 needs to be clearer to explain also in terms of $K_{task}$.
6. Line 192 mentions "informative prior". What is this "informative prior"?

*Minor*
7. Line 319: "addressing" is too strong. Suggest to change to "contributing".
8. Line 321: "than" -> "over"
questions: questions: I do not have any critical questions.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents a multi-output Gaussian process model for classification in the context of psychological studies. It's based on a linear model of co-regionalisation leveraging unit-level latent factors and RBF kernels to jointly model population and individual personality traits to tackle an old debate in psychometrics regarding their original nature. The authors leverage a variational formulation to derive a lower bound for the model evidence and optimise the individual-specific, population, and GP-related parameters. An extensive simulation and real-data study is proposed, presenting longitudinal and multi-subject survey analyses, personality correlations and predictions.
strength_weakness: strength_weakness: One could argue that the methodological novelty is limited as the presented multi-output GP model is fairly well-known in the GP community. However, I think its similarities and differences with GPLVM and GPDM are well presented, and the overall application is far from trivial regarding the nature of data (categorical outputs, latent variables, missing data, ...). This relative weakness is more than compensated by the strength of results and the potential such a methodology brings in a field like psychology, where the signal-on-noise ratio is generally low, and the nature of measurements is often challenging.
questions: questions: I only have a few questions regarding computation times, scaling and robustness of the inter-output covariance matrices, which are well-known limitations of co-regionalisation GP models in practice. 

- In this sense, could you provide numerical evidence and comparison regarding training and prediction times for IPGP and competitors?  
- Even though the number of tasks/topics remains probably limited in psychological applications, could you discuss a bit more, in Section 4.1 **Result**, the decrease in performance between full IPGP and the low-rank version. What could we expect for higher dimensions? Would the low-rank approximation remain robust for a more significant gap with the actual rank? 
- In my experience, several multi-output GP models (or maybe their implementations) are somewhat unstable when it comes to estimating $\textbf{K}_{task}$, and pathological cases can often arise. Have you experienced such problems? And if so, how did you manage to tackle this issue? 

**Typos:**

Page 5: The indices of the CMD definition have a slight error. All terms are written as $R_1$, and the norm is 'f' instead of '$l_2$'  
Page 6: The title of 4.2 "Cross-secitonal" ==> Cross-sectional  
Page 7: "Both correlation matrices **displace** a block pattern" ==> display?  
Page 7: "questions corresponding negative emotionality" ==> corresponding to
limitations: limitations: In my opinion, the limitations are adequately discussed. The application spectrum and position in the related literature are well documented.

Comments: comment: Dear Reviewer 5m3j,

Thanks for your valuable feedback. Please see our response below.

> If I am reading the paper correctly, each subject was asked to complete a survey six times per day.

Yes, that's correct. Collecting this data was a significant undertaking, and to our knowledge, we are among the very few groups that have ever collected data of such scope.

> The paper is vague about its technical contributions. I am unsure if the paper further contributes more substantially to the area of Gaussian process latent variable models. 

We describe our model as a "latent variable GP model" since it belongs to a broader category of models involving latent variables and GPs (including but not exclusive to the famous GPLVM from Lawrence). Technically, our model also differs with GPLVM in that: (1) optimizes the factor loading matrix while marginalizing the latent variables, (2) accommodates categorical data through a non-Gaussian ordered logistic likelihood. Please see our other rebuttals for more clarification on our contributions.

> Equation 2 confuses a distribution with a random variable stating $𝑝(𝑓^{(𝑖)})$, and I believe equation 3 confuses an inner with an outer product (shouldn’t it be $𝑤_𝑖𝑤_𝑖^⊤$?).

Thank you for pointing out the typo in eq 2. We will fix it in revision. Regarding eq 3, please see our discussion of updated notations below.

> The background sets up the idea that an ideographic approach has distinct loadings $𝑤_j^{(𝑖)}$ different across 𝑖. But the proposed model itself does not seem to sport this as each $𝑤_𝑗$ is global for each task. 

Thank you for bringing this up. We agree the notation could be improved -- the confusion here might come from overloading the use of variable w depending on its indexing. We will rename $w_i$ in Figure 1 and eq 3 to $Z^{(i)}$. The full updated notation is listed below: 

- Throughout our notation, superscript $(i)$ indicates unit and underscript $j$ indicates task.
- $W_{\text{pop}}$ represents the K x J (K latent dimensions, J tasks) shared interpersonal loading matrix.
- $Z^{(i)}$ represents the $K^*$ by J unit-specific low-rank loading matrix that serves to be the additional idiographic component, independent of $W_{\text{pop}}$. The rank $K^*$ of $Z^{(i)}$ need not be the same as the rank K of $W_{\text{pop}}$, and in fact we used $K^*=1$ in our experiments (see below).
- $K_{\text{task}}^{(i)}$ is the unit-specific task covariance matrix with shared component $W_{\text{pop}}'$ $W_{\text{pop}}$ and unit-specific deviations ${Z^{(i)}}' Z^{(i)}$ of $K^*<K$ (with this revised notation the transpose in equation (3) is correct as written). 

Now $W_{\text{pop}}$ is a global parameter estimated for the entire population while $Z^{(i)}$s are unit-level parameters.  These are combined in eq (3) to create a unique kernel for each unit reflecting both components. To aid identification and performance, we focused on $K^*=1$ in the experiments; we also tried $K^*=2$ only to find degraded performance but extra computational costs.  Note that there is much less data to estimate unit-level deviations than there is to estimate population-level structure, so in general we might expect to take $K^* < K$ when modeling. However, the reviewer is correct that even in this rank-1 case, our setup still induces a unique $K_{\text{task}}^{(i)}$ as specified in eq 3. We will clarify the discussion here and include a list of notations in the appendix.

> The paper claims this experiment validates the “Big Five” because model performance peaks at 𝐾=5. However, the model is only fit for 𝐾=1...5. 

Thank you for this valuable comment; we ran two additional factor analysis experiments accordingly. We first examine the performance of IPGP with higher model ranks $K \ge 5$ in the second case study (LOOPR); the results are shown in the table below.

The results consistently support the model with rank precisely 5, which has both higher model evidence and lower BIC, providing much more convincing evidence for the Big Five theory than originally presented. In particular, the BIC has a decreasing trajectory from lower rank to 5 and an increasing one from 5 to higher rank, suggesting that increasing the rank to 5 is necessary for the model to capture the ideal structure of the data, but that exceeding this rank only overfits.

**Performance of IPGP with ranks from 1 to 10 for LOOPR.**
| RANK | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| - | - | - | - | - | - | - | - | - | - | - |
| **LL/N** ↑ | -1.478 | -1.477 | -1.477 | -1.477 | **-1.476** | -1.477 | -1.477 | -1.477 | -1.478 | -1.477 |
| **BIC** (×10^11) ↓ | 1.2736 | 1.2726 | 1.2728 | 1.2726 | **1.2722** | 1.2725 | 1.2726 | 1.2732 | 1.2732 | 1.2734 |

Please also see the related additional study described in our rebuttal to reviewer A4W7, where we varied the model rank among {2, 5, 8} in our simulation study (the true rank is 5). We saw similar results there, where the true rank consistently yielded the best model fit.
**Weaknesses**

The paper is vague about its technical contributions. It makes the following hedged novelty statement: "the first multi-task GP latent variables model _for dynamic idiographic assessment_". I took this to mean that the proposed approach is not very new technically, but it has never yet been applied to dynamic idiographic assessments. However, the paper also makes the general claim that it "advances the literatures on Gaussian process latent variable models...". I am of the opinion that tailoring existing modeling frameworks to new applications does provide a technical contribution, as it contributes a new view and set of interpretations/metaphors that can help to better understand the abstract model. I think the paper does contribute in this way. But I am unsure if the paper further contributes more substantially to the area of Gaussian process latent variable models, as the paper is vague about that.

There is sloppiness in some of the main equations. For instance, equation 2 confuses a distribution with a random variable stating $p(\mathbf{f}^{(i}) \sim \cdots$, and I believe equation 3 confuses an inner with an outer product (shouldn't it be $w_i w_i^\top$?). There's also some confusing overloading of symbols between the background and the model sections. The background sets up the idea that an ideographic approach has distinct loadings $w_j^{(i)}$ different across $i$. But the proposed model itself does not seem to sport this as each $w_j$ is global for each task. I believe effectively the model does achieve an idiographic interpretation through the unit-specific kernel, but the connection between the background and model section is not clear.

I think the second experiment has some major flaws. The paper claims this experiment validates the "Big Five" because model performance peaks at $K=5$. However, the model is only fit for $K=1...5$. We can clearly see from the log likelihood numbers that model performance for IPGP is monotonically increasing in $K$ for the values considered, suggesting that it will likely be even better at larger values of $K$; if that were true, it would not validate the "Big Five" theory. As this is an applied paper, I would expect a much higher level of rigor on one of the two applied case studies.
Hi all,

We re-opened the discussion for this paper to Aug 17 11:59pm ET for authors to reply to the emergency review. The review is pasted below.

======

**Summary**

This paper considers psychometric data composed of ordinal responses $y_{ijt}$, each of which represent how unit _i_ answered survey item _j_ during time period _t_. The key characteristic of this item-response data is that the same $N$ units are longitudinally surveyed on the same $J$ items over $T$ periods. 

The paper takes an ordered logit factor modeling approach to analyzing such data, wherein $f_j^{(i)}(t) = \mathbf{w}_j^\top \mathbf{x}_i(t)$ is the latent ideal point of unit _i_ on on item _i_ at time _t_, and $\mathbf{w}_j$ and $ \mathbf{x}_i(t)$ are the $K$-dimensional loadings and factor vectors, respectively.

The paper moreover advocates an "idiographic approach [which] emphasizes _intrapersonal_ variation by requiring distinct loadings $\mathbf{w}_j^{(i)}$" which are different for each unit $i$. It is able to effectively achieve this by exploiting the repeated measurements of each unit-item $(i,j)$ pair over different time periods $t$.

The proposed model is an instance of a multi-task Gaussian process (MTGP). Intrapersonal variation is modeled using a unit-specific kernel $K_{time}^{i}$. The full covariance is then $JT \times JT$, resulting from a Kronecker product of $\mathbf{K}_{\textrm{time}}^{(i)}$ (applied to input $T$ periods) with a low-rank matrix $J \times J$ matrix that represents covariance between survey items (or "tasks"). This all extends the previously-presented linear model of coregionalization (LMC), which was originally developed for the simpler case where observations of $(i,j)$ pairs are not repeated over time.

The paper derives a variational inference inference algorithm for the model, which follows closely from previous work.

The paper then reports three sets of experiments: 1) a synthetic study that reports excellent results on parameter recovery (when the true parameters are known), 2) a re-analysis study that purports to show that a non-dynamic (i.e., non-idiographic) version of proposed model is able to identify the correct factor structure (K=5) from survey data designed to assess the "Big Five" psychometric traits, and 3) an illustrative case study involving a novel longitudinal data set.

**Soundness**: 3: good
**Presentation**: 3: good
**Contribution**: 2: fair

**Strengths**

The paper is very well-written. It brings an interesting application to life and convinces the reader that the proposed modeling approach is well-tailored to the problem at hand. The paper gives a clear review of the prerequisite concepts and presents its modeling approach clearly. 

The paper covers related work in psychometrics well is convincing that the modeling approach is novel within that applied community. Experiments bear out this claim, as the proposed model performs much better than models which are currently used in psychometrics.

The paper presents a novel longitudinal survey dataset composed of $N=93$ subjects who were given personality assessment surveys over the course of three weeks. If I am reading the paper correctly, each subject was asked to complete a survey _six times per day_. This data sounds highly non-trivial to collect, and should be considered a main contribution in itself.

**Questions**

As this is an emergency review, after the reviewer discussion period, I am not sure if the authors will be able to reply. But if they are able, I would just ask that they respond to the various points made in previous sub-sections.

**Limitations**

This paper did involve a fairly intense human subjects experiment for data collection, but the paper reports it was IRB-approved. Psychometrics of this kind are fairly common; I do not think the paper introduces any new potential negative societal impacts.

**Flag For Ethics Review**: No ethics review needed.

**Rating**: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

**Confidence**: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Dear Reviewer EAmK,

Thank you for your effort in reading our paper. We are glad that you found no ethical issues.

We will regardless include discussion on selection bias and data demographics as suggested by Reviewer f9td.
Dear Reviewer f9td,

Thank you for your feedback. We understand your concerns about selection bias and its impact on our results, though we believe the model still holds value. Here is our response:

Social and personality psychology often faces challenges with model building due to reliance on non-representative samples, typically from student populations. We follow current standards by aiming to broaden our sample within budget and available diversity. We will clearly state these limitations in the manuscript and suggest how future studies can test the model's applicability across different populations and cultures.

We will also include a more direct discussion on the diversity of our sample. For the first case study, the dataset from Soto (2019) was relatively diverse, using Qualtrics and quota sampling to ensure representative samples of the U.S. population:
* age: 11% ages 18-24, 18% ages 25-34, 17% ages 35-44, 19% ages 45-54, 17% ages 55-64, 18% ages 65 and older
* sex: 52% female, 48% male
* race/ethnicity: 74% non-Hispanic white/Caucasian, 11% black/African American, 10% Hispanic/Latino, 3% Asian/Asian American, 2% American Indian/Native American
* educational attainment: 10% did not complete high school, 33% high school graduate, 28% some college, 19% college graduate, 10% graduate or professional degree
* annual household income (\$): 14\% <20,000, 12% 20,000-29,999, 11\% 30,000-39,999, 15\% 40,000-49,999, 26\% 50,000-79,999, 22\% 80,000+

In the second study, participants in our longitudinal study (the third experiment) are mostly college students with an average age of 20.23 (SD = 1.94); 70% are female, 26% are male and 4% are self-identified as other; ethnicities self-reported as 42% Caucasian, 39% Asian, 12% African American, and 7% Other. We used the BFI-2 [Soto & John, 2007], a widely used personality measure across cultures and ages [Yoshino et al., 2022; Beatrice et al., 2024] (see supplementary materials). We acknowledge the typical biases of convenient sampling in higher education, including socio-economic and ethnic diversity limitations, which we will explain more clearly in the manuscript.
> I always prefer to see beforehand 'how much exactly' it will cost me when I intend to test a method on my data, and I suspect that future readers would, too. This is especially true for methods like MTGP, which I know are coming at a not negligible cost.

This view resonates with us. We're more than happy to add this discussion and these results regarding running time to the paper to help complete the story. We appreciate your original comment and the invitation to run these additional experiments.

> That being said, I commend the authors' efforts to provide additional evidence and thus improve the confidence one can grant this paper. More generally, I support the authors' view about 'building bridges' between ML methodologies and their applications. Although I consider myself more of a methodological researcher, I notice that we too often focus on 'novelty' by principle while neglecting meticulous and well-conducted applications. Such studies are crucial for advancing more experimental disciplines like psychology with all the rigour it deserves from ML researchers. We all agree that this paper aims not to advance ML vastly, but I appreciate that it uses ML to advance.

We truly appreciate your support!
Thank you for your thorough answers. I understand the technical constraints coming from such costly and time-consuming studies involving human subjects. While I acknowledge that this discussion about running time might be superfluous for your application, I always prefer to see beforehand 'how much exactly' it will cost me when I intend to test a method on my data, and I suspect that future readers would, too. This is especially true for methods like MTGP, which I know are coming at a not negligible cost. 

That being said, I commend the authors' efforts to provide additional evidence and thus improve the confidence one can grant this paper. More generally, I support the authors' view about 'building bridges' between ML methodologies and their applications. Although I consider myself more of a methodological researcher, I notice that we too often focus on 'novelty' by principle while neglecting meticulous and well-conducted applications. Such studies are crucial for advancing more experimental disciplines like psychology with all the rigour it deserves from ML researchers. We all agree that this paper aims not to *advance ML* vastly, but I appreciate that it *uses ML to advance*.
Thank you for your response, although we respectfully disagree.

We strongly believe work such as ours has a place at this conference.

> As this is an applied work, I suggest the authors submit their paper to a journal focused on bioinformatics.

The call for papers explicitly encourages submissions of applications and highlights the participation of diverse communities beyond core ML: 

> [NeurIPS] brings together researchers in machine learning, neuroscience, statistics, optimization, computer vision, natural language processing, life sciences, natural sciences, social sciences, and other adjacent fields.

> We invite submissions presenting new and original research on topics including but not limited to the following:

> - Applications 
> [...]
> - Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences) [...]

We do not believe it is in the spirit of the call for papers to dismiss applied work out of hand. 

> The innovation of the method proposed in this paper is quite limited.
> This article makes a very small contribution to the field of Gaussian processes

The reviewer guidelines explicitly encourage diversity in contributions beyond purely methodological improvements to (MT)GPs:

> There are many examples of contributions that warrant publication at NeurIPS.  These contributions may be theoretical, methodological, algorithmic, empirical, connecting ideas in disparate fields (“bridge papers”), or providing a critical analysis.

Our contributions here required both considerable expertise in GP modeling (including the expertise required to build several necessary innovations highlighted by reviewer A4W7) and considerable and sustained engagement with another domain to ensure success. Our strong results reflect a sophisticated model shedding light on an important psychological question.

We do not believe it is in the spirit of these guidelines to dismiss our non-methodological (empirical, bridging communities mentioned in the CFP, etc.) contributions out of hand -- especially as they are supported by methodological contributions as well!
Thank you for your response!

> Claiming in the introduction that you "advances on the ... GPLVM" is simply too much for me to take.

We are happy to rephrase this passage, in particular to avoid the acronym GPLVM entirely. We intended to indicate that our model was among a family of spiritually related models incorporating latent variables and Gaussian processes in their construction (it is "a" latent variable GP model), not that we were advancing "the" famous GPLVM model from Lawrence.
I think it is a matter of interpreting "bridge". It is clearly a "bridge" in terms of bringing new applications in. It is not a "bridge" in terms of bringing new ideas to advance ML --- an example of which is Random Matrix Theory.

For GPLVM, I am agreeable to it being mentioned in related work. Claiming in the introduction that you "advances on the  ... GPLVM" is simply too much for me to take.

I may reconsider the score during the reviewer discussion stage.
Employing other advanced MTGPs for this psychological task is straightforward. However, the author intentionally abandoned more comparisons with other MTGPs. 

The innovation of the method proposed in this paper is quite limited. 

This article makes a very small contribution to the field of Gaussian processes

Compared to state-of-the-art methods, its advancements and advantages are minimal. 

As this is an applied work, I suggest the authors submit their paper to a journal focused on bioinformatics.