State-Space Modeling of Temporal Dominance
Responses to Stimuli: A Case Study Using
Strawberry
Kyoichi Tachi, Shogo Okamoto(&), Yasuhiro Akiyama,
and Yoji Yamada
Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi 464-8601, Japan
shogo.okamoto@mae.nagoya-u.ac.jp

Abstract. Our perceptual and affective responses change in a dynamic manner
upon experiencing some stimuli; however, there are few mathematical models
describing their dynamics. In this study, we propose state-space modeling as amethod to represent their relationships based on time-dependent perceptual andaffective responses acquired by the temporal dominance (TD) method. We usedcanonical variate analysis to compute and deﬁne the state variables. For thispurpose, the TD responses were bootstrap-resampled to generate a sufﬁcientamount of training data. We applied this method to the TD responses to thestrawberries reported in our previous work. The estimated model could represent
the temporal evolution of some affective responses with a good accuracy index.
The proposed model consists of three latent variables, and the meaning of eachof these could be reasonably interpreted.
Keywords: Canonical variate analysis  Affective dynamics  Latent variables

1 Introduction
The perceptual and affective responses in humans are known to evolve in a dynamic
manner upon exposure to stimuli. These dynamic changes in multiple subjective
experiences are recorded by the temporal dominance (TD) method [1–3] in the foodindustry. This sensory evaluation method enables the study of evolution of perceptual(gustatory, osmatic, and textural) and affective responses during food intake. However,since this method has been in use only in recent years, there exists few mathematical
approaches to model the time-series data acquired by the TD method. For instance, thus
far, methods based on Granger causality [4, 5], Markov model [6, 7], or principal
motion analysis [8] have been reported.
In a typical TD task, approximately 10 types of perceptual or affective responses areevaluated. If a TD task for perceptual responses or affective responses is conducted
separately for a single type of food, a maximum of 20 types of responses should be
accounted for in a single model. This large number of variables has deterred ourintuitive understanding of the model and responses to food. Therefore, a method thatrepresents the entire model in a reductionist manner is required. The aforementioned
methods [4–7] are not amenable to the reductionist treatment of time-series data
© Springer Nature Singapore Pte Ltd. 2020
H. Shoji et al. (Eds.): KEER 2020, AISC 1256, pp. 139–148, 2020.
https://doi.org/10.1007/978-981-15-7801-4_14

140

K. Tachi et al.

acquired by the TD method, although the method in [8] was capable of decreasing themodel dimensions by using a few non-time-series latent parameters. In the present
study, by using state variables, we established relationships between perceptual andaffective time-series responses. Usually, the number of state variables is smaller than
the number of perceptual and affective responses evaluated in the TD task. The statevariables could therefore be utilized for a reductionist representation of the entire
model. We then utilized the TD data for strawberries [4, 5] to test the state-spacemodeling method reported here.

Fig. 1. Example of a graphical user interface used in TD tasks.

2 Temporal Evolution of Perceptual and Affective Responses
on Eating Strawberries
2.1

Temporal Dominance (TD) Method

In this study, we used the TD method [1–3] to measure the time evolution of perceptualand affective responses. The TD method enables the simultaneous measurement ofmultiple types of subjective responses while assessors ingest food. Here, we briefly
introduce the method, and more details can be found in previous reports [1–3].
In the TD method, a graphical touch panel interface is used, as shown in Fig. 1. Anassessor presses the start button when he or she puts a piece of food into the mouth.
S/he then selects a button on the touch panel, the label of which corresponds to thefeeling of the assessor on food intake. The selected label is not necessarily the one withthe highest subjective intensity. The assessor selects a corresponding button each timethe dominant feeling changes and presses the stop button when the sensations in theirmouths disappear. The same button can be selected more than once, and not all buttons
need to be selected at least once.
TD tasks record the time at which the buttons are selected in each trial, as shown inFig. 2(a). TD curves are subsequently calculated, as shown in Fig. 2(b), by integrating
the records from all the trials and assessors. These curves are smoothened for latercomputation. The TD curves represent the dominance rate obtained by dividing thetotal number of times each button is selected by the total number of trials. The temporalbase represents the time normalized to the period spanning the beginning of each trial
to the moment of disappearance of all sensations in the mouth.

State-Space Modeling of Temporal Dominance Responses to Stimuli

141(a)
Assessor 1

Sweet

Sweet

Assessor 2
Assessor 3

Sweet

Sour
Sour

Sweet

Sour

TD curve Sweet

TD curve Sour

Dominance rate (%)(b)

0Start of eating

Normalized time (-)

1
End of eating

Fig. 2. Calculation of the TD curves. (a) Binary data obtained from the TD method for each
assessor or trial. (b) TD curves are calculated by accumulating and smoothening the binary
responses shown in (a).

Table 1. Terms used to represent perceptual and affective/evaluative responses in [5].
Perceptual Affective/evaluative
Sweet
LikeSour
Delicious
WateryHappy/satisﬁed
Refreshing Fresh
Juicy
FlavorsomeMelty
NaturalSoft
Elegant

2.2

TD Curves for Strawberries

In this study, we used TD curves obtained on eating strawberries [4, 5]. These studies
employed the sensory and affective/evaluative labels listed in Table 1, and the acquired
TD curves are shown in Fig. 3. For the purpose of the present study, we removed theresponses to melty, soft, happy/satisﬁed, natural, and elegant from the original data
because their dominance rates were small and statistically insigniﬁcant. We then used
the remaining nine types of responses in the present study.

K. Tachi et al.
(a)

Sweet
WateryRefreshing

Dominance rate (%)

10080

Juicy
Sour(b)60
40200010.5
Normalized time

Like
Flavorous

100

Dominance rate (%)

142

Delicious
Fresh

80
60
40
20000.5
Normalized time

1Fig. 3. Temporal dominance (TD) curves for (a) sensory and (b) affective/evaluative responses
to strawberries. Modiﬁed from [4, 5].

3 State-Space Modeling of Temporal Dominance Curves
3.1

State and Observation Equations

The state and observation equations used in this study aremt þ Dt ¼ Amt þ But

ð1Þ

yt ¼ Cmt þ Dut þ etð2Þ

where, yt and ut are the output and input vectors at time t, respectively. We establisheda model that estimates changes in affective responses from those in perceptual
responses; hence, the inputs and outputs represent the dominance rates of perceptualand affective responses at time t, respectively. A, B, C, and D are the coefﬁcientmatrices related to the variables. These coefﬁcients indicate the strength of influenceamong variables. et and mt are the vectors for observation error and state variables attime t, respectively. From (1), the present state variables could be determined using thepast state and input vectors; therefore, the present state variables implicitly contain
information about the past states of the system. Hence, mt is called the vector ofmemory.
3.2

Canonical Variate Analysis (CVA)

To deﬁne the state variables mt , we used canonical variate analysis (CVA) for timedependent inputs and outputs [9]. CVA ﬁnds the relationships between input andoutput variables by computing the canonical variates that are the linear combinations ofinput and output variables.
The past vector pt and future vector f t are deﬁned as follows:Tpt ¼ yTtDt ;    ; yTtlDt ; uTtDt ;    ; uTtlDt

ð3ÞTf t ¼ yTt ; yTtþ Dt ;    ; yTtþ hDt

ð4Þ

State-Space Modeling of Temporal Dominance Responses to Stimuli

143

where, l and h represent time lags for the past and future, respectively. The time lags
determine the temporal orders that should be considered in the model. We determinedl ¼ h ¼ 1 in the latter computation. Dt is the sampling period used to generate discrete
TD curves, with a value of Dt ¼ 0:033 normalized time, corresponding to approximately 1 s.
We calculate the vector of memory mt using the past vector pt :
1mt ¼ W T Rpp2 pt

ð5Þwhere, W is a matrix of the left singular vector obtained by singular value decomposition, as represented in (6). Rpp and Rff are the variance matrices of pt and f t ,respectively, and Rpf is the covariance matrix of pt and f t :
1=2

R1=2
pp Rpf Rff¼ WRV Tð6Þ

where, R is a diagonal matrix of the singular values.
3.3

Bootstrap Resampling

For CVA calculation, the sample size needs to be substantially greater than the numberof variables to be analyzed. Usually, TD tasks produce a single set of TD curves frommultiple assessors. Hence, CVA cannot be directly applied to TD curves. Therefore, weincreased the number of TD curves by bootstrap resampling [10]. The new sample set
was generated by sampling the originally observed data with replacements, asdescribed in [8]. The number of assessors for forming one set of TD curves was eight,
and a total of 40 sets of TD curves were calculated.

4 Result
4.1

Computed State-Space Model

We established a model including three state variables. The number of state variables
was determined based on the ease of interpretation of the state variables. Figures 4, 5, 6and 7 show parts of the model corresponding to each of the three state variables. In
these ﬁgures, each node represents the dominance rate of each attribute and statevariable, and the edges denote the relationships between nodes. Red and blue edges
represent the positive and negative influences, respectively. The values next to theedges are the values of the coefﬁcient matrices corresponding to each edge. Here, theedges whose effect is small are not shown for visual clarity. From (1) and (2), the edges
ending at the state variables and affective labels represent influences from the past andpresent values, respectively.

144K. Tachi et al.

Figure 4 shows the state-space model related to the ﬁrst state variable. The ﬁrst
state variable is mainly affected by juicy and sweet and exhibits a great effect on
delicious. Therefore, the ﬁrst state represents the memory of the deliciousness ofstrawberries.
Figure 5 shows the model for the second-state variable. This state variable isaffected by all the perceptual responses and exhibits a positive effect on delicious and anegative effect on fresh. This state, therefore, represents the memory of comprehensive
sensation, and is the average property of subjective responses evoked on eatingstrawberries.

Flavorsome

Delicious

0.17

0.34
0.39

0.57
Juicy

0.28

-0.21

0.63

Watery

Sour

Sweet

Fig. 4. State-space model related to the ﬁrst state variable: memory of deliciousness.

Delicious

Fresh

0.37

-0.48

0.46

0.21
Juicy

0.28
Watery0.61

0.39

Refreshing

0.39
Sour

Sweet

Fig. 5. State-space model related to the second state variable: memory of average experience.

State-Space Modeling of Temporal Dominance Responses to Stimuli
Like

Flavorsome0.23-0.16
-0.40

0.93
Refreshing

0.49
SourFig. 6. State-space model related to the third state variable: memory of coolness.

Delicious

0.34

0.38

Memory ofdeliciousness

Averageproperties ofstrawberry
0.35
Sweet

Fig. 7. State-space model related to delicious.

145

146

K. Tachi et al.
Table 2. Correlation coefﬁcients between observed and estimated TD curves.
Responses Correlation coefﬁcients
Like
0.81
Delicious
0.84
Fresh0.09
Flavorsome 0.90

As shown in Fig. 6, the third state variable is affected by refreshing and sour;
therefore, it represents the memory of the cool feeling associated with eating strawberries. This state has a positive effect on like, whereas it has a negative effect onflavorsome.
As an example, we show the edges connected to delicious (shown in Fig. 7).
Delicious is positively affected by sweet and the ﬁrst- and second-state variables.
4.2

Estimation Accuracy

Figure 8 shows the observed and estimated TD curves for the four types of affective
responses. The orange and blue lines represent the observed and estimated values,
respectively. The correlation coefﬁcients between the observed and estimated values
are listed in Table 2.
As shown in Table 2, like, delicious, and flavorsome could be predicted with
sufﬁcient accuracy; however, the correlation coefﬁcient of fresh was 0.09, indicatingthat fresh was not well-represented by the present model. Furthermore, despite its large
correlation coefﬁcient, as shown in Fig. 8(a), the differences between the observed andestimated values of like were large. The trends could be predicted accurately; however,
the absolute values were not representative of reality.
The optimization of the past and future time lags and the number of states isnecessary to increase the estimation accuracy of the model.

State-Space Modeling of Temporal Dominance Responses to Stimuli(a) Observed and estimated TD curves for like

Dominance rate

0.6

0.2
0Dominance rate

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(b) Observed and estimated TD curves for delicious

0.6

0.9

1

Estimated
Observed

0.4
0.2
00

Dominance rate

Estimated
Observed

0.4

00.6
0.5
0.4
0.3
0.20.1
0-0.1

0.6
0.5
0.4
0.3
0.2
0.1
0-0.1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1(c) Observed and estimated TD curves for fresh
Estimated
Observed

0Dominance rate

147

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(d) Observed and estimated TD curves for Flavorsome
Estimated
Observed

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Normalized time
Fig. 8. Observed and estimated TD curves for (a) like, (b) delicious, (c) fresh, and (d) flavorsome.

5 Conclusion
In this study, we applied a state-space modeling method to the data acquired by the TD
method. To this end, we employed canonical variate analysis of past and future data on
resampled TD curves generated for the intake of strawberries. Using state-spacemodeling, it was possible to represent the temporal evolutions of affective responses,

148K. Tachi et al.

and the three state variables could sufﬁciently express the relationship among the ninesubjective responses. In the model established here, some attributes could not beestimated with sufﬁcient accuracy. This problem could be solved by optimizing themodel. We also need to conﬁrm the generality of this method by applying it to TD
responses to other foods. Furthermore, the semantic validity of the model calculated bythis method remains to be studied.

References
1. Pineau, N., Schlich, P., Cordelle, S., Mathonnière, C., Issanchou, S., Imbert, A., Rogeaux,M., Etièvant, P., Köster, E.: Temporal dominance of sensations: Construction of the TDS
curves and comparison with time-intensity. Food Qual. Prefer. 20(6), 450–455 (2009)
2. Di Monaco, R.: Temporal dominance of sensations: a review. Trends Food Sci. Technol. 38(2), 104–112 (2014)
3. Schlich, P.: Temporal dominance of sensations (TDS): a new deal for temporal sensory
analysis. Curr. Opin. Food Sci. 15, 38–42 (2017)
4. Okada, T., Okamoto, S., Yamada, Y., Ishikawa, T.: Vector autoregression model of temporalperceptual and affective responses towards food. In: IEEE Global Conference on LifeSciences and Technologies, pp. 43–45 (2019)
5. Okada, T., Okamoto, S., Yamada, Y.: Affective dynamics: causality modeling of temporallyevolving perceptual and affective responses. IEEE Trans. Affect. Comput. (2019). https://doi.
org/10.1109/taffc.2019.2942931
6. Franczak, B.C., Browne, R.P., McNicholas, P.D., Castura, J.C., Findlay, C.J.: A Markov
model for temporal dominance of sensations (TDS) data. In: Proceedings of 11th Pangborn
Sensory Science Symposium, Gothenburg, Sweden (2015)
7. Lecuelle, G., Visalli, M., Cardot, H., Schlich, P.: Modeling temporal dominance ofsensations with semi-Markov chains. Food Qual. Prefer. 67, 59–66 (2018)
8. Okamoto, S., Ehara, Y., Okada, T., Yamada, Y.: Affective dynamics: principal motion
analysis of temporal dominance of sensations data. IEEE Trans. Affect. Comput. (2020).
https://doi.org/10.1109/TAFFC.2020.2971700
9. Larimore, W.E.: Canonical variate analysis in control and signal processing. In: Katayama,T., Sugimoto, S. (eds.) Statistical Methods in Control and Signal Processing, pp. 83–119.
Marcel Dekker Inc., New York (1997)
10. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall, New York(1993)