International Journal
Transactionsof JapanofSociety
Affectiveof Engineering
Kansei Engineering
J-STAGE Advance Published Date: 2021.02.09
doi: 10.5057/ijae.IJAE-D-20-00013

NOTE

Structural Modeling of Temporal Dominance Responses
Using Covariances of Contemporary Changes in Subjective Qualities
Shogo OKAMOTO
Nagoya University, Chikusa-ku, Nagoya-city, Aichi 464-8603, Japan
Abstract: The establishment of methods for recording and analyzing the dynamic changes in the human affective responses is importantfor understanding and utilizing affective dynamics. This study analyzes the structure of sensory and affective responses recorded throughthe temporal dominance (TD) method, which is an effective dynamic sensory appraisal approach. Temporal evolution is modeled bystate-space local-level equations, and the structure of the responses is estimated based on the covariance matrix of the disturbances of thestate variables. This approach is applied to the TD responses for strawberries recorded in a previous study, and its validity is examined.
Keywords: State-space modeling, Partial correlation coefficient, Affective dynamics

1. INTRODUCTION
The temporal dominance (TD) method [1] has attracted
attention as a sensory appraisal method for measuring
the temporal changes in multiple types of perceptual andaffective qualities using food stimuli. However, few
studies have developed a corresponding mathematicalmodel to understand the dynamic relationships among thequalities. Understanding the dynamics provides productdevelopers with new scientific insights. Previously,
Markov chain models were developed to represent therandom transition of subjectively dominant feelings [2].
Okada et al. developed a method for modeling thecausality among the subjective qualities reported in the TDmethod [3]. Furthermore, Okamoto et al. extracted theprincipal motions in the time series of TD responses todetermine their dimensionality [4]. However, despite theserecent studies, in most studies that used TD methods, theTD responses were analyzed based on their static characteristics. Therefore, general methods for understanding
the dynamics of affective responses are required.
The aim of this study is to establish the relationshipbetween the subjective qualities in the TD responses basedon the covariances among the contemporary changes in thequalities. To the best of the author’s knowledge, thisapproach has yet to be attempted for TD responses. Thisapproach may result in a conclusion similar to that of a study
by the author and his colleagues [3] because the present
study and [3] seek the relationships among the sensory andaffective qualities. Nonetheless, in [3], the time series ofTD responses were modeled by autoregressive methods,
which use time-lagged data. In contrast, the present method
utilizes the contemporary changes in subjective qualities.
Received: 2020.04.30 / Accepted: 2021.01.12

Therefore, it is insightful to compare the results between
the present study and [3] to understand the characteristicsof these methods. To perform this comparison, in this study,
the experimental data obtained in [3] are analyzed.
2. TEMPORAL DOMINANCE METHODThe TD method is a popular approach in the field offood sciences [1]. It allows the collection of the temporalchanges for multiple types of subjective qualities, whereas
previous trace or time-intensity methods were limited toone or two types of qualities.
In this section, the method used in this study is outlined.
The TD method utilizes a graphical user interface, asshown in Fig. 1(a), with several buttons with descriptors.
An assessor presses the start button immediately after
putting a piece of food into his/her mouth. The assessor
then sequentially presses the buttons corresponding to thedescriptors that best suit his/her dominant feelings. This
operation continues until the food is swallowed and thestop button is pressed by the assessor. The same buttons
can be pressed multiple times, and some buttons canremain unselected. Once a button is pressed, the selectionremains in effect until another button is pressed. Oneassessor may test the same food several times.
As a result of the abovementioned task, for each button,a binary (i.e., selected or non-selected) time series isrecorded, as shown in Fig. 1(b). Each time series isnormalized by the period between the start and stop.
For each descriptor, among all the participants and trials,
the binary series are aggregated and the proportion atwhich the corresponding button is selected is computedat each time instance. This value is called the dominance
rate (proportion) and ranges between 0 and 1. The
Copyright © 2021 Japan Society of Kansei Engineering.
All Rights Reserved.

Advance Publication(a) G raphical user interface with randomly arranged buttons for
TD method(b) Single operation of TD task

instantaneous dominance rate of the TD curve fordescriptor i, is modeled to be the sum of its state uit andrandom error εit. The state is expressed as a first-order
autoregressive model. For all p variables yt = (y1t , …, ypt)T,
the observation and state equations are defined as follows:
,(1).
(2)
In the above equations, ε t and η t , which are called disturbances, are random variables subjective to the normal
distributions of zero means and are defined as follows:
,
(3)
,,

(5)
.

(c) Aggregation and smoothing of multiple binary time-series
for each quality(d) TD curves of multiple subjective qualities

(4)
(6)

In (3) – (6), H and Q are the covariance matrices ofthe disturbances for the observed and state variables,respectively. These covariances are unknown parametersthat are estimated using the maximum likelihood
estimation method. Although p types of state variables
seem independent of each other in (2), their disturbances
can be correlated.
The non-diagonal element qij in Q is the covariance
between the disturbances, i.e., the contemporary changes
for the two state variables i and j over the entire periodof the task. If qij is not zero, a relationship between thetwo variables is suggested; hence, the structure among
the state variables can be discussed on the basis of Q.

Figure 1: General process of temporal dominance method(a) Graphical user interface used in a task.
(b) Binary time series of selected buttons in a single task.
(c) Aggregation of binary time series and smoothing of curves
among all assessors.
(d) Temporal dominance curves of multiple subjective qualities.
Adapted, in part, from [4].

dominance rates are transformed into smooth curves
by using a low-pass filter, as shown in Fig. 1(c). Hence,by integrating all the assessors’ responses, one set ofmean TD curves is acquired, as shown in Fig. 1(d).
3. STRUCTURAL MODELING OF TEMPORALDOMINANCE RESPONSES
3.1 State-space local-level models and covariancematrix of state disturbances
The multivariate time series of TD responses aremodeled by state and observation equations. They enable
us to discuss the temporal evolutions of variables whileremoving the effects of random errors in the observed
values. An observed value yit at time t, which is the3.2 Significance of partial correlation coefficientsamong the disturbances of state variables
A covariance matrix can be transformed into a partial
correlation matrix that directly suggests the relationshipbetween variables. In general, in order to judge the significance of partial correlation coefficients, t-tests are used.
However, in the t-tests, the number of samples n is assumed
to be the number of, e.g., participants or objects. For thecase of state-space models, n is determined as the numberof quantization levels of normalized time. Hence, thevalidity of the t-test for state-space modeling is uncertain.
In this study, the significance of partial correlation
coefficients is judged based on the confidence intervals
estimated by bootstrap samples. Bootstrap resampling for
TD responses was introduced in [4], where observations
were randomly sampled with replacement m times. Based onthese m samples, a set of TD curves is computed. A partial
correlation matrix of the disturbances of state variables
is then computed from the set of curves. This processof resampling and computation of matrices is repeated1000 times, and the distribution of partial correlation

Advance Publication
Structural Modeling of Temporal Dominance Responses Using Covariances of Contemporary Changes in Subjective Qualities

coefficients is estimated. The significance of the coefficientsis then judged on the basis of their confidence intervals.
4. EXAMPLE OF STRAWBERRIESAs previously mentioned, in this study, the TD responses to strawberries recorded in [3] are analyzed. In theanalysis, five types of sensory responses (juicy, watery,sweet, sour, and refreshing) and four types of affective
responses (fresh, delicious, flavorsome, and like) are used.
Figure 2 shows the TD curves computed by eightparticipants (m = 8) in [3]. For the computation of thesecurves, a low-pass filter with a cutoff frequency of 1 Hz
was applied, as stated in Section 2.
The mean period between the start and stop buttons wasapproximately 30 s. In terms of the sensory responses inFig. 2(a), in the first half phase, sweet and juicy responses
were salient. In the second half phase, sour became
gradually strong. In terms of the affective and hedonic
responses shown in Fig. 2(b), delicious and flavorsome
responses were salient in the early phase, whereas likeand fresh responses were salient in the last phase.

The continuous TD curves were discretized into 30
temporal points, as in [3]. The discrete TD curves werethen modeled using state-space equations, as describedin Section 3.1. For the computation of these equations,
the KFAS package (ver. 1.3.7, J. Helske) for the R
programming language was used.
Table 1 lists the means and standard deviations of thepartial correlation coefficients among the disturbances forthe states. The significance of each coefficient is judged
on the basis of its confidence interval. Figure 3(a) shows
the suggested link between the sensory and affective
responses obtained from the results reported in Table 1.
The solid and dotted edges indicate the partial correlation
coefficients at the significance levels of 5% and 10%,respectively. Values aside the edges are the mean partial
correlation coefficients listed in Table 1. Figure 3(b) is thecausal model built in [3] using the same TD responses.
(a) Relationship suggested by the partial correlation matrix(a) Sensory responses(b) Causal relationship suggested by Granger causality in [3](b) Affective responses

Figure 3: Suggested structure among sensory and affectiveresponses

Figure 2: Temporal dominance responses to strawberries(a) Sensory responses.
(b) Affective and hedonic responses. Adapted from [3].

(a) Relationships suggested by the partial correlations in thepresent study.
(b) Granger-causality model investigated in [3]. Adapted from [3].
Red and blue arrows indicate positive and negative influences,respectively.

Table 1: Partial correlation coefficients among the disturbances of states corresponding to sensory and affective responses
Juicy
WateryRefreshing
Sour
Sweet
Like
Delicious
FreshJuicy-Watery-.02 ± .21-Refreshing-.38 ± .19*
.01 ± .22-

Sour-.33 ± .25*-.04 ± .19-.28 ± .27-

Sweet-.29 ± .32-.06 ± .23-.17 ± .25-.15 ± .28-

Like.26 ± .23+-.07 ± .25.25 ± .22.24 ± .22.27 ± .24+
-Delicious.37 ± .23*
.09 ± .23.25 ± .23.30 ± .24.52 ± .22*-.23 ± .27-

Fresh.29 ± .28-.07 ± .20.19 ± .25.32 ± .24+
.22 ± .27-.13 ± .26-.30 ± .27+
-Flavorsome.41 ± .20*-.18 ± .23.03 ± .24
.03 ± .21.20 ± .26-.27 ± .21+.03 ± .23-.25 ± .28

Means and standard deviations among 1000 samples. * and + indicate the statistical significance at p < 0.05 and 0.10, respectively.

Advance Publication

5. DISCUSSION
The semantic validity of the structure shown in Fig. 3(a)is discussed and compared with the causal model inFig. 3(b). A strong relationship between sweet and delicious
is seen in both models. Furthermore, with the partial
correlation matrix, the relationships juicy-flavorsome
and juicy-delicious are obtained. The state “juicy” isobtained when there is abundant strawberry juice,and the state “flavorsome” is obtained when there issome type of aroma. These suggested relationships
are semantically reasonable and are indirectly obtainedin Fig. 3(b).
With the partial correlation matrix, an inverse relationship is obtained between juicy and refreshing, while apositive relationship is obtained with the causal model.
Refreshing denotes the sense of coolness, and it is difficult to speculate its relationship with juicy.
Unlike Fig. 3(b), Fig. 3(a) suggests a negative relationship between juicy and sour. This relationship may
reflect the decrease in juicy and the increase in sour inthe middle of the eating phase, as shown in Fig. 2(a).
Regarding the dotted edges in Fig. 3(b), like ispositively correlated with sweet and juicy, which issemantically reasonable for the taste of strawberries. Incontrast, the like-flavorsome relationships obtained withthe partial correlation matrix and the causal model aredifferent. As shown in Fig. 2, in the middle eating phase,the rate of flavorsome instances decreases, while that oflike instances increases. Hence, they may exhibit a negative correlation coefficient. However, psychophysically,
the like and flavorsome responses exhibit a positiverelation [5], as shown in Fig. 3(b).
Regarding watery and sour sensations, the two models
produce very different results. In Fig. 3(a), watery is notconnected with any other responses, whereas in Fig. 3(b),sour does not influence the other responses. For these twotypes of qualities, both models may be incomplete because
the sour and watery qualities seem to be important factors
for the taste of strawberries, and both qualities may belinked with the others.
The above results indicate that the two models derived
from the same TD responses to strawberries partly agreewith each other. Differences between the two models
may be attributed to the differences in their mathematicalframework. If one of the models is semantically more
realistic than the other, then it is possible to conclude
which type of model is more suitable for representing theTD data. However, such a conclusion cannot be madesolely by considering the example of strawberries.

The method presented in this study uses the bootstrap
samples; however, its negative effects on the TD data
have not been investigated. Further, the TD data used inthis study were obtained from a relatively small numberof participants; hence, the generality of the establishedmodel of strawberries is low.
6. CONCLUSION
The structure of several types of subjective qualities in
TD curves was investigated. For this purpose, the curves
were modeled by state-space equations, and the covariancematrix of the contemporary changes in the states wascomputed. The suggested structure partly agrees with themodel used in a previous study. To establish the validity ofthe model, the present method will be applied to TD
responses for several types of foods in a future study.
REFERENCES
1. Pineau, N., Schlich, P., Cordelle, S., Mathonnière, C.,
Issanchou, S., Imbert, A., Rogeaux. M., Etiévant, P.,
and Köster, E.; Temporal dominance of sensations:
Construction of the TDS curves and comparison with
time-intensity, Food Quality and Preference, 20,
pp.450-455, 2009.
2. Cardot, H., Lecuelle, G., Schlich, P., and Visalli, M.;
Estimating ﬁnite mixtures of semi-Markov chains: An
application to the segmentation of temporal sensory
data, Journal of the Royal Statistical Society: AppliedStatistics, 68(5), pp.1281-1303, 2020.
3. Okada, T., Okamoto, S., and Yamada, Y.; Affectivedynamics: Causality modeling of temporally evolving
perceptual and affective responses, IEEE Transactions on Affective Computing, 2019. DOI: 10.1109/
TAFFC.2019.2942931
4. Okamoto, S., Ehara, Y., Okada, T., and Yamada, Y.;
Affective dynamics: Principal motion analysis oftemporal dominance of sensations data, IEEE Transactions on Affective Computing, 2020. DOI: 10.1109/
TAFFC.2020.2971700
5. Schiffman, S.; A psychophysical model for gustatory
quality, Physiology & Behavior, 7, pp.617-633, 1971.
Shogo OKAMOTO (Member)
Shogo Okamoto received the Ph.D. degree in information sciences from Tohoku University, in 2010.
Since 2010, he has been with Nagoya University.
He is currently an Associate Professor with theDepartment of Mechanical Systems Engineering,
Nagoya University. His research interests include
haptics, assistive systems, and affective science and engineering.