Multiple output samples for each input in a single-output Gaussian process
Authors: Jeremy Wong
Huayun Zhang
Nancy F. Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Gaussian process
multiple outputs
subjective
uncertainty
spoken language assessment

Abstract: The standard Gaussian Process (GP) is formulated to only consider a single output sample for each input in the training set. Datasets for subjective tasks, such as spoken language assessment, may be annotated with output labels from multiple human raters for each input. This paper proposes to generalise the GP to allow for multiple output samples per input in the training set. This differs from a multi-output GP, because all output samples are from the same task here. The output density function is formulated to be the joint likelihood of observing all output samples. Through this, the hyper-parameters are optimised using a criterion that is similar to minimising a Kullback-Leibler divergence. The test set predictions are inferred fairly similarly to a standard GP, with a key difference being in the optimised hyper-parameters. This approach is evaluated on spoken language assessment tasks, using the public speechocean762 dataset and an internal Tamil language dataset. The results show that by using the proposed method, the GP is able to compute a test set output distribution that is more similar to the collection of reference outputs annotated by multiple human raters.

Under review as a conference paper at ICLR 2023
MULTIPLE OUTPUT SAMPLES FOR EACH INPUT IN A SINGLE-OUTPUR GAUSSIAN PROCESS
Anonymous authors Paper under double-blind review
ABSTRACT
The standard Gaussian Process (GP) is formulated to only consider single out- put sample for each input the training set. Datasets for subjective tasks, such as spoken language assessment, may be annotated with output labels from multiple human raters for each input. This paper proposes to generalise the GP to allow for multiple output samples per input in the training set. This differs from multi- output GP, because all output samples are from the same task here. The output density function is formulated to be the joint likelihood of observing all output samples. Through this, the typer-parameten are optimised using criterion that is similar minimising a Kullback-Leiberr divergence. This is computationally cheaper than repeating the input for each output sample. The test set predictions are inferred fairly similarly to a standard GP, with key difference being in the optimised syper-parameters. This approach is evaluated on spoken language as- sessment tasks, using the public specchocean762 dataset and an internal Tamil language dataset. The results show that by using the proposed method, the GP is able compute a test set output distribution that is more similar to the collection of reference outputs annotated by multiple human raters.
INTRODUCTION
The Gaussian Process (GP) (Rasmussen & Williams 2006) expresses prediction uncertainty that naturally increases for inputs further away from the training data. As opposed to this, Neural Net- works (NN) have been observed yield overly confident predictions, even when the input is from a mismatched domain (Guo et al. (2017). This behaviour of GP may allow better explainability of the model's predictions. Having explainable predictions of uncertainty may be especially desirable for tasks that are subjective in nature. In such subjective tasks, multiple human annotators may pro- vide differing output labels for the same input. collection of human annotations for the same input may therefore be interpreted as reference of uncertainty that an automatic model should also aim to compute. In such settings, the uncertainties expressed by the model and the human annotators can be explicitly compared.
However, the standard GP formulation assumes that each input in the training set paired with only single output, which is treated as the ground truth. This paper proposes to extend the GP formu- lation, to accommodate for situations where multiple samples of output labels for the same task are provided for each input. The dyper-parameter can be optimised and the test set predictions can be inferred, with the consideration of having multiple training set output samples, in a computationally cheaper manner than simply repeating the inputs for each output sample.
RELATED WORK
The multi-output GP is formulated in a multi-task framework (Yu et al., 20055 Bonilla et al. (007). This treats the multiple outputs for each input separate tasks. On the other hand, this paper considers a single-output GP with a single output task, where multiple output samples for each input present for that task.
This paper considers optimising the GP hyper- parameters using a criterion that similar to min- imising distance to reference output density function. When training NN, the reference output

---

Under review as a conference paper at ICLR 2023
can be in the form of distribution, opposed scalar single class. Full-sum training in speech recogmition Yan et al. 1997) and handwriting recognition (Senior & Robinson 1995) trains NN toward the distributional reference formed the soft forced alignment. In BLIND NNs for Spoken Language Assessment (SLA) are trained toward the distribution represented by the scores from multiple human raters. The distributional output from one NN can also be used as reference to train another NN toward (Li et al. 2014] Hinton et al. 2014).
GAUSSIAN PROCESS REGRESSION
When given a collection of N input feature vectors of dimension D, X E RRDD GP places jointly Gaussian prior over latent variables, f E RR,, as
- (f
(1)
Here, X(f is abbreviation of = fI with the interpretation of being the like-
lihood of continuous random variables ff taking the values f. A multivariate Gaussian density function with mean ii and covariance V is represented as (f: μ., V). In a GP, the covariance of the latent variable defined as a pair-wise distance between the inputs, with the notion of distance being defined by the kernel, K. In this paper, the squared exponential kernel is used, with kernel matrix elements defined
(x: x.,) (æ; xzj)
kis (X =² exp
exex
212
where: and j are the matrix indexes, / is length hyper-parameter, and s is a scale hyper-pane parer-erretter.. The GP makes the assumption that the outputs, y NR.. are conditionally independent of the inputs when given the latent variables. The outputs are Gaussian distributed about mean of the latent variable, with noise hyper-parameter, a, to accommodate for observational noise,
(yll) = Ny
(3)
where is the identity matrix.
3.1 TRAINING
Training GP involves estimating the hyper-parameters of the kernel, and and of the observation noise a. One approach is to find the hyper-parameters that maximise the marginal log-likelihood of the training data,
F=
log fy X)
(4)
where the training data comprises pairs of observed features and reference outputs, " The marginal likelihood can be computed
p x p(ylf)
(5)
N (x.
(6)
Optimising the hyper-parameters using gradient-based methods requires the inversion of (XX,X)+o' o2I (Rasmussen & Williams, 2006). which entails a number of computational opera- tions that scale as O(N"), when using Gaussian elimination or when computing the singular value decomposition in pseudo-inverse implementation. Algorithms, such as Petković Stanimirović 2209), are able to reduce the polynomial power, but still require more than O(N*) operations.
3.2 INFERENCE
When performing evaluation, a test set of input feature vectors, X, is given, and the task is to infer the predicted outputs, ÿ. Inference through GP can be performed by first computing the density
2

---

Under review as a conference paper at ICLR 2023
function over the predicted latent variables,
p f X
p (f.
X)
K (Xxx))+o') 1 K x, X
N
;0,
K xx,X)
K X
(8)
Wv; +
= -f(f.a.D)
(9)
where
ñ K
X X (X, + o'11-'y y
(10)
V K
x x - K [ (x [K (XXXX)++00 K X
(11)
which, similarly to training, also requires the computation of |K(X, X) + o² 0211
The predictive density function for the output can then be computed as
X. = -N N
(12)
Finally, the predicted scalar output for each test data point, i, can be inferred from equation using a decision rule. One possible decision rule is to choose the mean of equation 12 i7 = 144- As result of the symmetry and uni-modality of a Gaussian density function, the decision rules of choosing either the mean, mode, or median of equation 12 are equivalent.
12
4 MULTIPLE OUTPUT SAMPLES
In the standard GP formulation, each input data point is paired with a single output sample. As is described in section situations may arise where modelling multiple output samples for each input data point may be beneficial. This paper considers such scenario, whereby all output samples for the same input data point are drawn from the same output density p(y(f) and kernel K. is thus assumed that these multiple output samples belong to the same task. This differs from the multi-output GP formulation, which assumes that each separate output represents a different task.
Let i be the training set data point index. Let each input training data point, æ;, be associated with R output reference samples, yıd [yait ... yitl where yfr the rth reference output sample for the ith training data point. It is assumed here for simplicity that the number of output samples is the same for all inputs, but an analogous formulation can be derived to allow for varying numbers of output samples. These outputs are stacked together across all input data points to yield output matrix rf E N
4.1 REPETITION OF INPUTS
A naive approach to accommodate multiple output samples may be to simply flatten Y into a vector and repeat the inputs, x,, for each output sample. This will yield N kernel of K= K(X. X) 1 ( where (1)) a Rxx matrix of is and @ represents a Kronecker (tensor) product. Although it may be theoretically questionable as to what it may mean to have such a non- full-rank kernel to represent the Gaussian covariance in the GP, there is no need to explicitly compute its inverse. Instead, only the inverse of K + + needs to be computed during training and infer- ence, which is full-rank. Computing this matrix inverse requires a number of operations that scales as O(N3 (2)), as opposed to ((N3) when using only a single output sample. Computational con- straints may then impose the need for sparse approximation methods, such Quinonero-Candela & Rasmussen (2005), even for modest training set sizes.
above matrix vector used here to emphasise that comprises repeated elements.
3

---

Under review as a conference paper at ICLR 2023
4.2 OMISSION OF REDUNDANCY IN THE PRIOR
When using this repeated kernel, the latent variables are distributed as p(fiX) = N(fo K). The repeated structure of K causes all latent variables that are associated with the same training data point to be perfectly correlated. Therefore. it seems computationally redundant to separately model the repeated latent variables. Instead, this paper proposes that only a single instance of each latent variable per training data point be modelled using equation [ and the joint density of the multiple output samples be conditioned on these non-repeated latent variables, p (Y))). The diagonal covariance in the output density function of standard GP in equation [] implies that the outputs from different training data points are conditionally independent of each other when given the latent variables. This conditional independence, applied to the case of repeated inputs in the naive approach, yields in this proposed approach a joint output density that can be factorised as
p (Y
II N (yri f.rr)
(13)
=N (T(Y): 0² o (1)x0 N (x)); ( o² 1 (2TT²) (2-2) R-N,
(14)
where
1 R R
1 (Y)
yr and T/ (y:) =
(Uhr phi (y:))²
(15)
R
and yr- E R vector of the rth output sample for all inputs. The factorisation and diagonal covariance in equation 13 assume that the multiple output samples for the same input and across different inputs are independent of each other, when given the latent variable. As a result, the output density does not depend on the order of the multiple output samples for each input. This can be explicitly seen in the re-parameterisation of the multiple outputs as the empirical mean je and empirical biased standard deviation 7). both of which are independent of the ordering of the output samples. Also, equation [4 may seem unnormalised over z and 17. However, it should be noted that ffY is expected to sum to one when integrated over Y. and not over çE and 77.
4.2.1 TRAINING
The marginal likelihood can be computed from the non-redundant prior of equation I and the output density of equation 14 as
x( = ( (X115)
(16)
= =N -, + 21 g()),
(17)
where
9 ( ( -
7 (Y): 0. 0² 1
(2-10) R-N
(18)
Analogously to the standard GP, and equivalently to the naive approach, the hyper-parameter of the proposed approach can be optimised by maximising the marginal log-likelihood of output samples,
F= log (y x)
(19)
Unlike a naive implementation, gradient-based optimisation of the proposed method has a number of operations that scales O(N)), similarly to standard GP with single output sample.
In appendix A maximising equation 19 is shown to be related to minimising a Kullback-Leinn (KL) divergence between the marginal likelihood and the reference density function over output samples, under several assumptions. This suggests that maximising equation 19 encourage the GP's output density to be more similar to the reference density.
4

---

Under review as a conference paper at ICLR 2023
4.2.2 INFERENCE
When performing evaluation, the predicted output for each input test data point can be inferred from the predictive density function. To compute the predictive density function, the joint density between the training set outputs and the test latent variables first needs be computed. When the training set comprises multiple output samples per input, this paper again proposes that there is no need to separately model redundant latent variables, by expressing this joint density as
- ff. X, X PY
(Y, f x x
(20)
NN
K K(X,X) X)
K x x
(MOM)
( of
:0,
N
K x
K X x
(21)
=N
T (Y) {{
00
K K X)
K X
(00)
(22)
f
K X
x
K
x(X,8)
Analogously to equation] 7 the predictive density over the test latent variables is
x p x
p xfF}x,,,x)
(23)
=N (f: v)
(24)
where
0²
i = K (x,
K (X,X) +
(Y)
(25)
R
+ 0² 2
V K x x
K
X xx
K (X,
K (x,x))
(26)
R
The predictive density over the test outputs is then
p x X x) = -N (V..,.++''')
(27)
Finally, the output can be inferred for each test input data point by choosing the mean of this predic- tive density, similarly to how a score is chosen from equation 12
In this proposed approach, the matrix inverted in equations 25 and 26 is of size NNN as opposed to N in the naive repetition approach. The number of operations thus scales as O(N3), similarly to a standard GP with a single output sample.
The covariance of the predictive densities in both equations 12 and 27 are independent of the train- ing set outputs. These covariances depend only on the training set inputs and the As such, information about the proximity between the multiple output samples for the same input in the training set does not influence the resulting predictive covariance. In fact, the GP formulation such that whether the training set output values are near or far from each other, for two inputs that are near as measured by the kernel, does not influence the resulting predictive covariance for test inputs near to these training inputs. What does affect the predictive covariance is the distance between the test and training set inputs. This implies that GP may be more suited for computing distributional uncertainty, and less so for data uncertainty (Xu et al.] (002)). Perhaps, a future re- search direction may be to allow the GP to take training set output information into account for the predictive covariance.
5 SPOKEN LANGUAGE ASSESSMENT
Section 4 proposes a modification of a single-output GP, to allow the training set to comprise multi- ple output samples for each input, where it is assumed that all output samples for the same input
5

---

Under review as a conference paper at ICLR 2023
related to the same task. This formulation may be applicable for tasks that not strictly determin- istic. For example, noise in the environment may induce a variance in the observed output, a task may have multiple valid output values that correspond to the same input value, or a task may be subjective such that different human experts may annotate the same input with different output values. In such scenarios, including multiple output samples per input data point in the training set may be more informative than using only a single output sample for each input.
5.1 TASK
Examples of subjective tasks are language assessment, emotion recognition (Busso et al. 2008 and medical diagnostics. The experiments in this paper are applied to SLA. In SLA, the imput an audio recording of speech from a student. The task is to predict a score for that input that is similar to a score that an expert human rater would have given. This score is meant to relate to aspects of the oral proficiency of the student. Examples of potential assessed aspects include the pronunciation accuracy, intonation, fluency, and prosody, In practice, examination committees often try to standardise the assessment criteria through a rubric. However, it may be difficult to concisely document rubric that takes all possible circumstances into account. As such, much may still be left to the human rater's s subjective opinion when assigning a score.
To account for this subjectivity, training datasets for SLA may include the reference scores from multiple human raters for each input. It is standard practice to compute combined reference score, by taking either the majority vote (Lin & Wang 2021). median (Zhang et al.) 202Tb), or mean (Zhang et al., 2b2 of these multiple scores. An automatic SLA model may then be trained toward and evaluated against this combined reference. Examples of such approaches are Metallinou & Cheng (2014); Duan & Chen (2220); Lin & Wang (2021) using NNs, and van Dalen al. (2015) using GPs. However, collapsing the scores from multiple human raters into a single score may result in loss of information about the output uncertainty that should be associated with each input. The proposed framework allows a GP to take into account multiple output samples for each input in the training set.
As can be seen from equation 25 the predictive mean under the proposed formulation is dependent on the mean of the multiple oulput samples, jı, and independent of 1j. This is similar to a standard GP formulation, when using a mean combination of the multiple rater scores to get the reference. As mentioned previously, the predictive covariance is independent of the training set outputs. Thus, during inference, the remaining difference between the standard GP and the proposed modification in the hyper-parameters. The proposed approach optimises the hyper-parameters by maximising the marginal log- ligeliiel of all training set outputs, which may be related to minimising the KI. divergence between this marginal likelihood and the reference density function. As opposed to this, the standard approach maximises the marginal log- likeliiol of the combined output reference.
5.2 EVALUATION MEASURES
Previous works on SLA often evaluate a model's performance by comparing the predicted output score from the model against the combined reference score. Two evaluation measures that are often used are Pearson's 's Correlation Coefficient (PCC) and Mean Squared Error (MSE) (Zhang et al.) 2021b). These are written out in appendix B Both measure how well the predicted score matches with the combined reference score. This assumes that the combined reference score is correct.
However, comparing only against the combined reference score omits information about the agree- ment between the multiple human raters. This agreement may be representative of a reference of the uncertainty that should be associated with each input. It may be useful to assess how well the predictive density agrees with this reference uncertainty. This is measured using the KL divergence between the predictive and reference densities, through both the unnormalised continuous and the discrete forms described in appendix

---

Under review as a conference paper at ICLR 2023
6 EXPERIMENTS
6.1 DATASETS
Experiments were performed on the speechocean762 dataset (Zhang et al. 2021b) and an internal Tamil language dataset. These are regression tasks with multiple output reference samples provided. The public speechocean762 dataset allows for experimental repeatability, while the internal Tamil dataset is used to verify that the observed trends are generalisable. Speechocean762 comprises a training and test set, each with 2500 sentences and 125 disjoint speakers. The native Mandarin speakers read the sentences in English. Each sentence is annotated with a variety of score types at different linguistic levels, but only the sentence-level pronunciation accuracy was used in this paper. Each sentence is annotated by 5 human raters, each assigning an integer score between 0 and 10.
The Tamil language dataset (Zhang et al., 2021a) comprises a total of 4211 sentences, read by 100 speakers, aged between 9 to 16 years, from Singapore. This was split into training and test sets with 2118 and 2093 sentences respectively, with speaker disjointment. Stratified sampling was used to split the data, to ensure similar demographics between the different educational levels and assessment scores across the two sets. Each sentence was annotated with pronunciation accuracies by 4 human raters, with integer scores between 1 and 5.
For both datasets, combined reference scores were computed as a mean of the multiple rater scores. This differs from Zhang et al. (2021b), which instcad computes the median score. Using the mean may yield a baseline that is a more appropriate match for the formulation in equation [4 Following the standard in ZZang et al. (2021b), the predicted and combined reference scores were first rounded to the closest integers, before computing the PCC and MSE, for both datasets.
These datasets are chosen to assess the proposed approach for three reasons. First, the training set sizes allow for computational feasibility with a GP, without having to resort to sparse approximations or to use only a subset of the training data. The use of sparse approximations may potentially occlude experimental trends. Second, both datasets are annotated with multiple output reference samples for each training data point, thereby allowing the specific experiments in this paper be run. Third, these are regression tasks, thereby avoiding any complications that may arise from using a GP for classification. Datasets for emotion recognition and stutter detection are also often annotated with multiple output reference samples. Potential emotion recognition datasets that can be considered for future exploration include IEMOCAP (Busso et al, 2008) with regression task annotated with two output samples and may thus be too few to observe interpretable trends, and MSP- & Busso 2019) with a regression task annotated with five output samples but has a much larger training set. For stutter detection, SEP-28k (Lea et al.. 2021) and KSoF (Bayerl et al.) 2022) classification tasks annotated with three output samples.
(Lotfinn
6.2 MODELS
The feature extraction for NN SLA model was similar to Zhang et al. 2202 First, speech recognition models described in appendix D were trained. These were then used to force align the audio with the transcriptions. From the forced alignment, Goodness Of Pronunciation (GOP) (Witt & Young. 2000), Log Phone Posterior (LPP) (Hu et al.) (2013), Log Posterior Ratio (LPR) (Hu al. [2015), and tempo (Zhang et al. (0221a) features were extracted for each phone. Pitch features GGhahremani et al. 2014) were extracted from the audio, and pooled across all frames within each phone. A continuous skip-gram model (Mikolov et al. 2013), with one recurrent NN (Elman 1990)
hidden layer having 32 nodes, was trained on the training set non non-silence phone sequences, and used to extract phone embeddings (Zhang et al., 2021a). The GOP, LPP. LPR, tempo, pitch, and phone embedding features were concatenated to form one feature vector per phone.
In baseline NN SLA models, the sequence of feature vectors, with length equal to the number of phones in the sentence, were fed into a Bidirectional Long Short-Term Memory (BLSTM) layer (Graves & Schmidhuber] 2005), with 32 nodes per direction. The output was then pooled across all phones in the sequence with equal weights, and fed through a linear layer. Two forms of NN output layers were used. In the first variation, referred to as NN the linear layer had a single dimensional output, which then passed through a sigmoid (Zhang et al. 2021a). This sigmoid output was then scaled to the bounds of the output score range, and the computed output was treated as the predicted score. Such a regression model was trained by minimising the MSE toward the combined
7

---

Under review as a conference paper at ICLR 2023
reference score. In the second variation, referred as NN eaaegoricall the linear layer output dimension was equal to the number of integer scores, and the output was fed through a softmax (Duan & Chen 2020). This classification model was trained using the cross- cross-entropy eriterion toward the combined reference score. Both models used dropout (Srivastava et al. 2014) with an omission probability of 60%, before the BLSTM and linear layers.
may not be straight forward to allow GP to utilise sequential inputs. Work in Lodhi et al. (2000) investigates designing kernels to operate on sequences, while van Dalen et al.) (2015) extracts hand- crafted sentence-level features for SLA. In this paper, the activations after the NN pooling layer were treated as bottleneck features (Konig et al. 1998), and used as sentence-level inputs to the GP. A principle component analysis whitening transformation, estimated on the training set, was applied to these features to better abide by the diagonal covariance assumption of the sealar length hyper- hyper-parameter parametaaeee in the kernel in equation
6.3 STATISTICAL SIGNIFICANCE
The statistical significance, P. of the difference between either the MSE or KL divergence evaluation measures of two models was computed using two-tailed paired t-test. As seen equation [3] the PCC is not easily expressed as a sum over data points, making it difficult to apply the central limit theorem. The significance was instead computed using the Z approach (Steiger 1980). Here, an approximately normally distributed transformation (Dunn & Clark 1969) was first computed from the two PCCs being compared. Then the significance was computed as the two-tailed cumulative density of this transformed quantity.
6.4 BOTTLENECK FEATURES AS INPUTS TO A GAUSSIAN PROCESS
The first experiment investigates the soundness of using bottleneck features inputs to GP, and the impact that the training approach used for the NN bottleneck feature extractor can have. The NN.attrr and NN catteooriaal models were trained as described in section 6.2 Sentence-level bottleneck features from after the pooling layer were computed, and used as inputs to the GP. The standard GP was used, with only single reference output sample per training data point, referred as GP hase- The GP
were optimised separately for each of the feature types, by using gradient descent maximise the marginal ligelikelihood of the training set, in equation The mean combined score between the multiple raters was used as the training set reference output.
Table 1: Impact of the bottleneck feature extractor Gaussian process
Bottleneck
Dataset
SLA model
extractor
PCC
MSE
NN
0.711
1.232
NNcategorical
0.701
1.208
speechocean762
GPhase
NNscalar
0.710
1.149
GP base
NN Neettooricaal
0.694
1.209
NN
0.638
0.716
Nattogoriaal
0.599
0.621
Tamil
GP base
NN
0.565
0.641
GP
NN cattegorical
0.555
0.648
The results in table Lable[show show that both the NN and NN eaaateguical models yield comparable PCC and MSE performances, with no clear victor over both measures. A GP that uses the NN bottle- neck features may yield better PCC and MSE performances than when using NNcategorical features. The statistical significance between GPs that use either type of features are PPCC = 0.033 and PMSE = 0.078 for s6eechocean762, and PPcC 0.334 and PMSE 0.617 for Tamil. Thus the performance differences may not be significant. Although not significant, the improvements from using bottleneck features from ANsallr consistent across both datasets. Therefore, all subsequent experiments use bottleneck features from aaaaaa as inputs to the GP. The GP and NN SLA models exhibit comparable performances.
8

---

Under review as a conference paper at ICLR 2023
6.5 MULTIPLE OUTPUT SAMPLES PER INPUT DATA POINT
The next experiment assesses the proposed extension, of allowing the GP to use the separate out- put scores from multiple human raters in the training set, referred to as GP This is compared against base. and also against a naive multiple output reference samples extension of repeating the training set input features, referred to as GP repeat-- The GP juim hyper-parameter were optimised by maximising equation 19 and the predicted scores were inferred using equation 27
Table 2: Using training set scores from multiple raters in a Gaussian process
KL divergence
Inference
Dataset
Model
PCC
MSE
continuous
discrete
time (s)
GP base
0.710
1.149
5.04
3.10
136 =3
speechocean762
GP repeat
8476 #510
0.713
1.136
1.88
0.85
GP joint
138 =4
GP base
0.565
0.641
1.52
0.72
106 =3
Tamil
GP repeatt
2777
#350
0.569
0.634
1.19
0.54
GP Gi joint
107 =3
Table compares GPs that use either only the mean combined reference score or the separate scores from the multiple raters in the training set. The PCC and MSE results may suggest that the proposed GP Join approach yields performance gains over GP GP but these may not be significant, with Prec = 0.065 and PMSE 0.036 for speechocean762, and Prcc 0.624 and PMSE = 0.585 for Tamil. The PCC and MSE compare the prediction against the mean combined reference, thereby assuming that this mean combined reference is correct. Furthermore, this omits consideration of how well the output uncertainty expressed by the model matches the uncertainty between the human raters. The KL divergence is used to assess this uncertainty matching. As bascline, the discrete KL divergence of the NNcatogorical model is 1.26 for speechocean762 and 0.70 for Tamil. The results show that using the separate scores from multiple training set raters improves both the continuous and discrete KL divergences of the GP, with significances of Pdiscrete-LL < 0.001 for the discrete KL divergence of both datasets. This suggests that the proposed extension allows the GP predictive density function or distribution to better match that represented by the collection of scores from the multiple raters. Reiterating from section 5. the primary difference between the standard and proposed approaches is in the estimation of the hyper-parametert during training.
Multiple output samples in the training set can also be considered by repeating the inputs in GP repeat as is described in section 4.1 The proposed GP joint approach has an advantage of a reduced number of operations required to perform matrix inversion. This computational saving was assessed by repeating inference 10 times on a 48 core Intel Xeon Platinum 8268 2.90GHz CPU using the Numpy pseudo-inverse implementation, and the mean and standard deviation of the total time to infer the whole test set accumulated over all threads are shown in the right-side column in table 2 Inference using a GP can be run efficiently by pre-computing the matrix inverse before any test data points are seen. therefore seems more interpretable to view the total time across the whole test set, rather than the time per data point or real time factor, assuming that the computation time is dominated by matrix inversion. The results show that GP GP Jumn indeed faster to infer from than GP repeest- GPpoint and GP require comparable durations to infer from. Although not shown here, this computational saving also benefits the matrix inversion computation during training.
7 CONCLUSION
This work has proposed an extension to the GP, to allow for multiple output samples for each input in the training set. The hyper-parameters of such a model can be optimised using a criterion that may be related to KL divergence. Inference is performed in a fairly similar way to standard GP, with the main difference being in the the predictive density to better match the uncertainty expressed by the collection of multiple human annotations, and reduces the computational cost compared against a naive approach of repeating the inputs.
that are optimised differently. This allows

---

Under review as a conference paper at ICLR 2023
REFERENCES
S. P. Bayerl, A. W. von Gudenberg, Hönig, E. Nöth, and K. Riedhammer. KSoF: The Kassel state of fluency dataset a therapy centered dataset of stuttering, LREC, pp. 1780-1787, Marseille, France, Jun 2022.
BLIND. Modelling inter-rater uncertainty in spoken language assessment, IEEE/ACM Transactions on Audio, Speech, and Language Processing (in review).
E. Bonilla, K. M. A. Chai, and C. K. Williams. Multi-task Gaussian process prediction. In NIPS, Vancouver, Canada, Dec 2007.
H. A. Bourlard and N. Morgan. Connectionist speech recognition: a hybrid approach. Kluwer Academic Publishers, 1994.
C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, N. Chang, S. Lee, and S. S. Narayanan. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335-359, Nov 2008.
N. Dehak, J. Kenny, R. Dehak, Dumouchel, and P. Ouellet. Front-end factor analysis for speaker verification. IEEE Transactions Audio, Speech, and Language Processing, 19(4): 788-798, May 2011.
R. Duan and N. F. Chen. Unsupervised feature adaptation using adversarial multi-task training for automatic evaluation of children's speech. In Interspeech, pp. 3037-3041, Shanghai, China, Oct 2020.
O. J. Dunn and V. Clark. Correlation coefficients measured on the same individuals. Journal of the American Statistical Association, 64(325):366377 377, Mar 1969.
Elman. Finding structure time. Cognitive Science, 14(2):179-2) 211, Apr 1990.
Ghahremani, B. BabaAli, Povey, Riedhammer, Trmal, S. Khudanpur. pitch ex- traction algorithm tuned for automatic speech recognition. In ICASSP, pp- 2494-2498, Florence, Italy, May 2014.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. AISTATS, pp. 315- 323, Fort Lauderdale, USA. Apr 2011.
A. Graves and Schmidhuber. Framewise phoneme classification with bidirectional LSTM net- works. In IJCNN, pp. 2047-2052, Montreal, Canada, Jul 2005.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinherger. On calibration of modern neural networks. In ICML, pp. 1321-1330, Sydney, Australia, Aug 2017.
G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. In Deep Learning and Representation Learning Workshop, NIPS, Montréal, Canada, Dec 2014.
W. Hu, Y. Qian, F. K. Soong, and Y. Wang. Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers. Speech Commanication, 67:154-166 Mar 2015.
B.-H. Juang. Maximum--lkellhoon estimation for mixture multivariate stochastic observations of Markov chains. AT&T Technical Journal, 6436):1235-1244) Jul 1985.
Y. Konig, Heck, Weintraub, and K. Sonmez. Nonlinear discriminant feature extraction for robust text-independent speaker recognition. In Workshop Speaker Recognition and its Com- mercial and Forensic Applications, RLA2C, pp. 72-75, Avignon, France, Apr 1998.
C. Lea, V. Mitra, A. Joshi, S. Kajarekar, and P. Bigham. SEP-28K: A dataset for stuttering event detection from podcasts with people who stutter. In ICASSP, pp. 3798-6802, Toronto, Canada, Jun 2021.
Li, R. Zhao, J-T. Huang, and Y. Gong. Learning small-size DNN with output-distribution-bee criteria. In Interspeech, pp. 1910-1914, Singapore, Sep 2014.
10

---

Under review as a conference paper at ICLR 2023
B. Lin and L. Wang. A noise robust method for word-level pronunciation assessment. In Interspeech, pp. 781-785, Brno, Czechia, Aug 2021.
H. Lodhi, J. Shawe-Taylor, N. Cristianini, and C. Watkins, Text classification using string kernels. In NIPS, pp. 563-569, Denver, USA, Nov 2000.
R. Lotfian and C. Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing. 10(4))471-483. Oct 2019.
Mermelstein. Distance measures for speech recognition, psychological and instrumental. In Pattern Recognition and Artificial Intelligence, pp. 374-388. Academic Press, 1976.
A. Metallinou and Cheng. Using deep neural networks improve proficiency assessment for children English language learners. In Interspeech, pp. 1468-1444 1472, Singapore, Sep 2014.
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In ICLR, Scottsdale, USA, May 2013.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In ICASSP, pp. 5206-5210, Brisbane, Australia, Apr 2015.
V. Peddinti, Y. Wang. D. Povey, and S. Khudanpur. Low latency acoustic modeling using temporal convolution and LSTMs. IEEE Signal Processing Letters, 25(3):373-377, Mar 2018.
M. D. Petković and P. S. Stanimirović. Generalized matrix inversion is not harder than matrix multiplication. Journal of Computational and Applied Mathematics, 230(1):270-282, Aug 2009.
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motliček, Y. Qian, P. Schwarz, Silovsky, G. Stemmer, and Vesely. The Kaldi speech recognition toolkit. In ASRU, Hawaii, USA, Dec 2011.
D. Povey, V. Peddinti, D. Galvez, Ghahremani, V. Manohar, Na, Y. Wang, and S. Khudanpur. Purely sequence-traine trained neural networks for ASR based on natice-frree MMI. In Interspeech, pp- 2751-2755, San Francisco, USA, Sep 2016.
J. Quiñonero-Candele and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. JMLR, 6665): Dec 2005.
C. E. Rasmussen and C. K. Williams. Gaussian processes for machine learning. MIT Press, 2006.
A. Senior and T. Robinson. Forward-bacde backward retraining of recurrent neural networks. In NIPS, pp- 743-749, Denver, USA, Nov 1995.
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple
way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929-19588 Jun 2014.
H. Steiger. Tests for comparing elements of correlation matrix. Psychological Bulletin, 87(2): 245-251, 1980.
R. C. van Dalen, M. Knill, and M. J. F. Gales. Automatically grading learners' English using Gaussian process. In SLATE, pp. 7-12, Leipzig, Germany, Sep 2015.
A. Waibel, T. Hanazawa, G. E. Hinton, K. Shikano, and K. J. Lang- Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing. 37 (3):328-339, Mar 1989.
S. M. Witt and S. J. Young. Phone-level pronunciation scoring and assessment for interactive lan- guage learning. Speech Communication, 30(2-3):95-108, Feb 2000.
B. Xu, R. Kuplicki, S. Sen, and M. P. Paulus. The pitfalls using Gaussian process regression for normative modeling. PLoS ONE, 16(9), Sep 2021.

---

Under review as a conference paper at ICLR 2023
Y. Yan, M. Fanty, and R. Cole. Speech recognition using neural networks with forward-backwar probability generated targets. In ICASSP, pp. 3241-3244, Munich, Germany, Apr 1997.
S. J. Young, J. J. Odell, and P. C. Woodland. Tree-based state tying for high accuracy acoustic modelling. In HLT, pp. 307-312, Plainsboro, USA, Mar 1994.
K. Yu, V. Tresp, and A. Schwaighofer. Learning Gaussian processes from multiple tasks. In ICML, pp. 1012-1019, Bonn, Germany, Aug 2005.
H. Zhang, K. Shi, and N. F. Chen. Multilingual speech evaluation: case studies on English, Malay and Tamil. In Interspeech, pp. 4443-4447, Brno, Czechia, Aug 2021a.
J. Zhang, Z. Zhang, Y. Wang, Z. Yan, Q. Song, Y. Huang. K. Li, D. Povey, and Y. Wang, spee- 760cean762: an open-source non-native English speech corpus for pronunciation assessment. In Interspeech, pp. 3710-3714, Brno, Czechia, Aug 2021b.
A APPENDIX: RELATION OF CRITERION TO KULLBACK -LEIBLER DIVERGENCE
Maximising the marginal log-likelihood equation can can related to minimising the Kullback- Leibler (KL) divergence between the marginal likelihood and the reference density function over output samples. To see this relation, consider only a single training data point, i, and ignore the correlations between multiple training data points. The marginal lokelikeli for the multiple output references of this single training data point is
F == log (Vi
rel DER
(28)
If it is assumed that the multiple outputs are independently sampled from p (v:(ar)). then equation 28 can be expressed as
F log (ii (
(29)
X / off (g. vilxx;) log (si: = yi (x,) dy
(30)
where the reference density function is
R
of (1i- = vil2x;) =
Σ
8
(Ui, v:r
(31)
R
The Dirac delta function is defined as
8 (a, b) - 100
-
if
10 otherwise
(32)
The form of equation 30 is of an unnormalised KL divergence, Thus, maximising equation 19 may be related to minimising a KL divergence between p (Y x) and the reference density function, under several assumptions.
B APPENDIX: STANDARD EVALUATION MEASURES
The Pearson's 's correlation coefficient computed as
(tt E (1+}) (7.
E (2)rr))
PCC %
(33)
M Σ
M Σ
of E (
E (2
12

---

Under review as a conference paper at ICLR 2023
where the empirical mean is The mean squared error is computed as
M
I
(v)
1 M
Σ
y/-
(34)
==1
M
MSE
y y
V/
(35)
M
C APPENDIX: EVALUATING UNCERTAINTY PREDICTION
The GP computes a continuous predictive density at its output. This can be compared reference density using an unnormalised continuous KL divergence,
M Σ p (Tila)) (D: Ri, y, X) dge
KI econtnmouu
(36)
M
where M is the test set size and the reference density defined in equation 31 lower value indicates better agreement. However, the measure is not bower-bounded by zero
To allow comparison of this agreement with that of categorical NN model, the GP predictive density can be first discretised by accumulating the likelihoods of scores that would have been rounded to each integer into a discrete posterior,
5+0.
P X) dg.
P
p = T., y. x)
(37)
Σ Je P (7) D y. X) di/
where e are the possible integer score classes, and represents discrete random variable. The discrete KL divergence between this predictive posterior and the reference distribution can then be computed as
1
M
pie (1: c (r,
Kiiiicceee
pref (g: clā,) log
(38)
M
P (ı = き y X)
The discrete reference distribution can be computed from the multiple human rater scores
1
pref (5e c
2 (c. vir)
(39)
R
where the Kronecker delta function is
[1
if a = b 0 otherwise
a (a,
(40)
This assumes that the scores from the multiple raters are provided as integers. The discrete KL. divergence is lower-bounded by zero. An analogous discrete KI. divergence can be computed when comparing the predictive posterior from a categorical NN against the reference distribution. One of the few available public SLA datasets is specchocean762 (Zhang et al. 2021b). In this dataset's publication of Zhang et al. (2021b), the standardised method of evaluating a model is established first rounding both the predicted and reference scores integers, before computing the PCC and MSE. This shifts SLA toward classification-I task, and further from being a fully regression task. The discretisation into the predictive posterior in equation 37 matches well with this practice.
D APPENDIX: SPEECH RECOGNITION MODELS
The speech recognition models used to compute the forced alignment between the audio and the transcripts for both datasets were phonetic hybrid (Bourlard Morgan 1994) models trained using the Kaldi toolkit (Povey al. 201
13

---

Under review as a conference paper at ICLR 2023
The speech recognition model used align speechocean762 was constructed following the steps that are used in Zhang et al. (2021b). This trained model on the Librispeech 960 hours training set (Panayotov et al. (2015), tollowing the standard Kaldi recipe. In this recipe, 5 layer Time Delay Neural Network (TDNN) (Waibel et al., 1989) acoustic model with Rectified Linear Unit (ReL.U) (Glorot et al. 201T) activations, and total left context of 13 frames and right context of 9 frames, was trained toward the cross-entropy criterion. This had about 19.1 million parameters. The cross- entropy targets were obtained from forced alignment using triphone Gaussian mixture model hidden Markov model (Juang 1985). The input features to the TDNN were 40 dimensional Mel frequency cepstral coefficients (Mermelstein) 1976) and 100 dimensional i-vectors (Dehak et al.j 2011 The acoustic model outputs were 5672 tied triphone states Young et al. 1994).
The Tamil language speech recognition model was trained on 220 hours of closed-talking speech from 710 adults. The acoustic model comprised 3 long short-term memory layers and 6 TDNN layers with ReLU activations, interleaved together (Peddinti et al. 2018), with total right context of 20 frames. This used 29 dimensional filterbank features, was trained using the lattice-free imple- mentation of the maximum mutual information criterion (Povey et al.) 2016), and had around 8.5 million parameters. The outputs were 1888 tied left diphone states.
14


Reviews:
[{"review_id": "dMKlEVRtIZ", "reviewer": "Reviewer_yWAJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an extension of GP that is applicable to situations where a single task is assigned several different output labels. The proposed model is evaluated using real-world datasets.", "summary_of_the_review": "This paper challenges an interesting task, but lacks novelty. The organization of the paper should also be re-considered.", "strength_and_weaknesses": "S1. The authors are trying to solve a spoken language assessment task using a Gaussian process. The problem addressed herein is important.\n\nW1. The manuscript is not well organized and the technical contributions are somewhat unclear. \n\nW2. The proposed model seems incremental. The proposed model attempts to represent multiple outputs by independent sampling from a single GP. This does not seem to contain any particular novelty.\n\nW3. I think it would be better to explain the importance of problem settings more carefully, for example, by drawing an easy-to-understand diagram.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Il0q4vhDde", "reviewer": "Reviewer_15fK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a method for considering multiple outputs associated to a single input point in the context of Gaussian process regression. The paper claims that simply incorporating new observations leads to a singular covariance matrix which cannot be inverted, preventing standard Gaussian process inference. The authors proposed a method that is based on using the standard Gaussian process prior and extra likelihood factors, for the multiple observations. The result is a Gaussian posterior and a Gaussian predictive distribution. The proposed approach is evaluated on two problems. It is compared to a baseline that consists in averaging output observations.\n", "summary_of_the_review": " Overall I think that this is paper presents an idea that is way to simple. It also mentions the problem of inverting the covariance matrix in the naive approach in which the prior is extended also for the extra observations. It is well known that this problem can be simply solved by adding some jitter to the diagonal of the covariance matrix. Therefore, the only advantage of the proposed approach is the computational savings from reducing the covariance matrix of the prior. However, this is not discussed by the authors. Also the experimental comparison is too weak since only two datasets are considered and the benefits with respect to averaging observations is marginal.\n", "strength_and_weaknesses": "Strengths:\n\n        - The proposed approach may have the advantage of reducing training cost when compared to considering an extended Gaussian process prior which may lead to a non-invertible covariance matrix. However, this is not evaluated.\n\nWeaknesses:\n\n        - The paper has a sloppy notation in which several terms are introduced without clarification. It not clear what Y_ref or y^º_i are. This makes difficult following the paper.\n\n        - The experimental section is weak. Only two datasets are considered and the baseline the authors compared with is very simple.\n\n        - The results show no particular benefit with respect to the simple baseline the authors compare with.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "IgRkOkTeTDN", "reviewer": "Reviewer_ssqv", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an extension of Gaussian process regression (GPR) for the case that multiple output values can be obtained for one common input. Multiple output values follow the standard GPR noise model, and the authors consider KL based criterion for hyper-parameter optimization.", "summary_of_the_review": "The paper is well-organized, but I currently do not find the fundamental motivation for introducing the proposed method. The standard GPR framework can handle the problem setting that the authors seemingly claim it cannot.", "strength_and_weaknesses": "Strength: \n\nThe paper is well-organized and easy to follow.\n\nWeaknesses:\n\nTo be honest, I do not understand why the proposed approach is required. Although the author mentioned that the stacked kernel approach does not work because eq(13) is not full rank. However, as far as the noise variance is nonzero \\sigma > 0, the predictive distribution of GPR can be calculated even when (13) is not full rank (inverse is required only for (K + sigma^2 I), not for K). Obviously, it is usual that, when \\sigma > 0, multiple different observations can be obtained for the identical x. The standard GPR framework can deal with this situation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2FPiAVvezP", "reviewer": "Reviewer_PFR9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an extension to Gaussian Processes to perform training and inference over multiple outputs for each input. The value of this is demonstrated by showing that the extension is a better predictor of the KL divergence of human raters on a speech intelligibility task.", "summary_of_the_review": "The paper proposes an extension to Gaussian processes to enable training on multiple output samples (while avoiding a naive formulation which is numerically unstable).  While a potentially interesting extension to Gaussian Processes, the paper does not sufficiently demonstrate that this approach is substantially better than other approaches to confidence estimation approaches, or is broadly applicable by evaluating on a single task (speech intelligibility).  ", "strength_and_weaknesses": "Strength\n* Clear mathematical extension of the standard GP formulation to train on multiple outputs for a single input.\n\nWeakness\n* Unclear that this improvement is solving a problem.\nThe speech intelligibility task like many subjective evaluations has substantial uncertainty (noise) in the ratings.   It's unclear what a better measure of uncertainty is important to this task in order to understand the necessity of this improvement\n\n* Limited evaluation.\nThere are a plethora of approaches to measure, predict and account for noise in subjective ratings. In Section 6.5 it is not clear how the KL divergence without multiple input samples is calculated.  (Repeated samples from the vanilla GP?)\n\nHow would this approach compare to a naive approach using a fixed covariance across the full data set? Or an only slightly more complicated approach, predicting the variance for each utterance?\n\nThe proposed approach still has a Gaussian predictive density function.  This limits the distributions that can be represented by this technique.  How would this compare to a variant whose predictive density function were a mixture of Gaussians similar to \"Gaussian Mixture Modeling with Gaussian Process Latent Variable Models\" (https://arxiv.org/abs/1006.3640)?\n\n* Efficiency Discussion\nGaussian Processes are computationally expensive.  Are there other approaches to density estimation that are less expensive that can be compared to the proposed? Or that the use of Gaussian Processes can be shown to be more computationally expressive?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject