Journal of Robotics, Networking and Artificial Life, Vol. 2, No. 1 (June 2015), 50-53

Evaluation of a Narrative Discourse Generation System
Based on the Concept of “Norm and Deviation”
Taisuke Akimoto

Graduate School of Informatics and Engineering, the University of Electro-Communications, 1-5-1 Chofugaoka
Chofu, Tokyo 182-8585, Japan
Takashi Ogata

Faculty of Software and Information Science, Iwate Prefectural University, 152-52 Sugo
Takizawa, Iwate 020-0693, Japan
E-mail: t8akimo@yahoo.co.jp, t-ogata@iwate-pu.ac.jp

Abstract
This paper deals with the verification of a narrative discourse system. The system automatically produces a variety
of discourse structures from an inputted story structure through an iterative mutual action between a narrator andnarratee mechanisms. We analyze a series of generated discourse structures according to their structural feature
values by focusing on the diachronic alternation of norms, the narratee’s expectation in receiving discourses. Basedon the results, we discuss achievement and issues to be addressed.
Keywords: narrative generation system, automatic analyses, narrative discourse, norm and deviation, Jauss.

1. Introduction
Automatic narrative generation is a challenging topic in
the field of artificial intelligence. In this topic,
methodology for evaluating system is a difficult issue
and has addressed by several researchers. Each of Roweet al.1 and Zhu2 presented multiple viewpoints toevaluate narrative generation system including theauthorial process or intention, generated texts, andprocess or modes of reading. Each of Pérez y Pérez3 andPeinado et al.4 presented formalization of such
evaluation criteria as coherence, interestingness, andnovelty in generated narratives.
We have been addressing the development of an“integrated narrative generation system” (INGS) basedon an expanded literary theory, an interdisciplinary
approach to narrative generation mechanism in INGS
across informatics and literary theories.5, 6 This paper
deals with the verification of the narrative discourse
system that we have developed as a practice of theexpanded literary theory.7, 8

The system automatically produces a series ofnarrative discourse structures from an inputted story
structure through an iterative mutual action between anarrator mechanism, which generates discourse
structures, and a narratee mechanism, which receives
the generated discourses. This cyclic generation model
continuously produces different discourse structures
through diachronic alternations of norms, the narratee’s
expectation in receiving discourses, caused bydeviations of the norms. A norm corresponds to a fixed
frame at the time of generating discourse structures anddeviation is an action to try to produce a new type of
discourse structure by breaking the norm.
This paper quantitatively analyzes the behavior of
the implemented system based on a conceptual
framework of norm and deviation and considers theachievements, limitations, and issues to be addressed.
2. An Outline of the Narrative Discourse SystemThe narrative discourse system was implemented with
Common Lisp. It automatically produces a series of

Published by Atlantis Press
Copyright: the authors
50

discourse structures from an input story structure. We
outline the system based on our previous papers. 7, 8
2.1. The discourse structure generation
The generation of a discourse structure refers tostructural transformations of a story structure into a
discourse structure. A story is a content of a narrative ora temporal sequence of events. A discourse refers to astructure of how the story is expressed. Each structure iscommonly represented with a tree structure inconceptual description. Each terminal node of a tree isan event and each internal node is a relation among thechild nodes.
A story structure is transformed by using discourse
techniques which define transformational operations ofa part of the tree structure. The techniques are definedby referring a part of the discourse categories byGenette.9 Thirteen types of discourse techniquesincluding temporal ordering, repetition, and so on are
equipped in the system. Different discourse structures
are generated according to what techniques are used andwhere the techniques are applied in the tree structure.
The techniques to be used are determined based on thenarrator’s generative goal which we will describe latter.
On the other hand, since the target of each technique isdecided at random with several conditions, the output
structures by a same generative goal have relatively
small differences.
2.2. The generation cycle
The output discourse structures gradually change
through the iterative mutual actions between thenarrator and narratee. We call the iterative actions
generation cycle. In each step, the narrator generates a
discourse structure from a story structure according to aset of parameters as generative goal or targeting
structural features. On the other hand, the narratee feeds
back an evaluation of the generated discourse according
to a set of parameters as expectation or desiring
structural features. The ten parameters corresponding tostructural features relevant to the 13 discourse
techniques are commonly used in the generative goal
andexpectation:“supplement,”“complexity,”“suspense,” “length,” “hiding,” “descriptiveness,”“repetition,”“diffuseness,”“implication,”and“temporal-independency.” Each parameter takes a valueof 1 (small), 2 (medium), or 3 (large).

The diachronic change of output discourses arises
from the change of parameters in both the generative
goal and expectation. This mechanism is based on areinterpretation of a part of the literary history model byJauss.10 The narrator basically sets the generative
parameters to fit the narratee’s expectation andgenerates discourse structures iteratively. The narratee
increases his satisfaction by receiving the fitted
discourses to the expectation. The process, however,
eventually reaches a point where the narratee gets tiredor his satisfaction begins to fall. The turning point of thesatisfaction is arbitrary set by the variable np. When thishappens, a deviation occurs and the narrator abandons a
portion of the old generative parameters and moves to a
new cycle of discourse grounded on the newly foundstrategy. The narratee’s expectations change according
to the reconstruction. In this generation model, the roleof the narratee’s expectation is to hold a norm fordiscourse generation.
3. The Framework of the Analyses
This section describes the framework of the analyses fora series of discourses produced by the system.
3.1. The structure of a series of discourses
The structure of a series of discourse structures isrepresented as Fig. 1. Each arrow beside “generativegoal” and “expectation” in the figure means the durationof same parameters. A shift in norm means a change ofa parameter in the expectation. We use a subscript
number to indicate a specific norm like “normi.” We calla discourse generated by the generative goal equal to theexpectation normative discourse and a discourse by agenerative goal not equal to the expectation deviateddiscourse. The processing of deviation is to change aparameter in the generative goal at random and it causesa shift in norm in the next step. By segmenting theseries with each point of shift in norm, the tale discourse
in each segment is the deviated discourse and the others
are normative discourses.
Set of normativediscourses
D

Generative
goal
Expectation

Norm1

D

D

Deviated
discourse…

D

Deviation

DD…

D

Norm2

Shift in norm

D =a discourse structure

Fig. 1. Structure of a series of discourse structures.

Published by Atlantis Press
Copyright: the authors51…

3.2. Four aspects of the analyses
The basic idea of the analyses is to treat each discourse
structure as a set of numerical values which represent
structural features corresponding to the ten parameters.
We call the values discourse feature values (DFVs).
Each DFV is automatically calculated from a generated
discourse. For example, “length” and “complexity” arerespectively calculated based on the number of terminal
nodes in a discourse structure and the number ofrelations needed for defining temporal ordertransformation. A set of discourse structures can betreated as a kind of feature space based on their DFVs.
The degree of deviation is calculated as the distance
between a space of normative discourses and thedeviated discourse. The difference between two norms
is also calculated as the difference of their spaces.
Based on the above method, we composed aprogram which consists of the following four analyses.
(i) Local generation space: The role of norm is torestrict the generation space into a certain range.
For confirming the behavior of norm-based
generation, this analyzes characteristics of the set of
normative discourses in each norm.
(ii) Degree of deviation: The deviation is a process totranscend the generation from the local generation
space at the time. For clarify the actual action of thedeviation, this calculates the distance of thedeviated discourse from the local generation space.
(iii) Degree of shift in norm: For clarifying the mannerof actual changes in local generation spaces, thiscalculates the magnitude of difference of each norm
from the last norm.
(iv) Novelty of norm: If local generation spaces arealways different with all the past local spaces, thesystem can continuously produce novel norms anddiscourses. This analysis calculates the magnitudeof difference of each norm from the most similar
norm in all the past norms.
4. Results
We executed the system 10000 steps for preparing anexperimental data. The input story was same with thestory in Ref. 8: A warrior rescues a princess who wasabducted by a snake, the plot consisting of 16 total
events. The value of np, the turning point of thenarratee’s satisfaction, was 200.
The DFVs of each discourse were automaticallycalculated. The program preliminary analyzed the range(the minimum and maximum), average, and standard
deviation of each DFV in all the discourses (see Table1). In total, 8982 patterns of different discourse
structures—as combination of the ten DFVs—werecounted from the 10000 discourse structures.
Next, the series of discourses was divided into 271
segments (i.e., norms). The average of segment length(i.e., number of discourses) was 36.90 and the minimum
and maximum were respectively 17 and 113.
4.1. Local generation space (i)The program analyzed the normative discourses in eachnorm by the same manner with the above analysis.
Table 2 shows two examples of the results. The rangesof DFVs were restricted from the entire set and eachnorm has different characteristics.
As an issue to be considered, although each local
generation space had different characteristics, the timing
in which the narratee’s expectation is saturated wasTable 1. Ranges, averages, and standard deviations of DFVs in all the discourses.

supplement complexity suspense length hiding descriptiveness repetition diffuseness implication temporal-independency
minimum
000
110
00
-900
maximum
450
852
10824
20
62
average
2.74
8.07
0.69
28.97 3.86
3.44
6.78
2.92
1.90
1.22
SD
1.32
4.48
0.99
6.16 2.34
2.17
3.67
4.21
1.29
0.71

Table 2. The local generation spaces in norm1 and norm181.

Norm1 (cycles 1-113)
supplement complexity suspense length hiding descriptiveness repetition diffuseness implication
minimum000
11300-50maximum
000
13
5
00-3
0average
0.00
0.00
0.00
12.30 3.70
0.00
0.00
-3.70
0.00
SD
0.00
0.00
0.00
0.65 0.65
0.00
0.00
0.65
0.00
Norm181 (cycles 6674-6710)
supplement complexity suspense length hiding descriptiveness repetition diffuseness implication
minimum
480
19
5
1
3-51maximum
440
532
8
314
6
3
average
4.00
14.94
1.97
25.67 6.72
2.42
6.50
-0.22
2.61
SD
0.00
7.20
1.26
3.24 0.90
0.72
2.75
2.69
0.54

Published by Atlantis Press
Copyright: the authors
52

temporal-independency
000.000.00
temporal-independency
00
0.00
0.00

arbitrary defined by the value of np. On one hand a lotof overlapped discourses with others appeared ingeneration spaces which have small ranges like norm1.
On the other hand, large generation spaces like norm181
were shifted to the next norm before the space was not
filled sufficiently. A solution is to redefine thesaturation as the filling of the local generation space.
4.2. Degree of deviation (ii)
The program calculated the degree of deviation in eachnorm based on the distance between the local generation
space and deviated discourse. As the result, on one handabout 66% of deviated discourses were positioned
outside of the space at the time. On the other hand,about 34% of deviated discourses were included in thespace—i.e., the narrator failed the deviation in apractical sense. Such failures occurred due to the partial
overlapping between the current local generation space
and the deviated (subsequent) space. A solution is to use
this analyzing method in the narratee mechanism for
judging the success or failure of deviation.
4.3. Degree of shift in norm (iii)
The program calculated the degree of shift in each norm
based on the difference of each local generation spacefrom the last space. We confirmed that the local
generation spaces were gradually shifted. It means thatthe holistic diversity of discourse structures arose
through the restriction of generation space based on thenorm and the accumulation of small shifts in the norms.
4.4. Novelty of norm (iv)
The program calculated the novelty of each norm basedon the difference of each local generation space fromthe most similar norm in the past. We clarified that thenovelty was gradually decreased through the generation
cycle due to the limitation of the possible combinationsof parameters in the generative goal and expectation.
5. Conclusion
In this paper, we analyzed a series of discourse
structures produced by the narrative discourse system.
We used an analyzing program based on a conceptual
framework of norm and deviation. The results
quantitatively showed that the system can produce
diverse discourse structures through the restriction ofgeneration space based on a norm and the accumulationof small shifts in norms.

In addition, mainly the following two issues wereclarified: saturation of the narratee’s expectation wasarbitrary defined regardless the actual reception ofgenerated discourses and the narrator failed thedeviation often in a practical sense. A solution is toembed the analyzing program into the narratee
mechanism for controlling the generation cycle basedon the analyses of actually generated discourses.
The diversity of generable narratives is an importantability for narrative generation systems. The analyzing
method proposed in this paper can be applied to othernarrative generation systems for clarifying the holisticgeneration ability.
References
1. J. P. Rowe, S. W. McQuiggan, J. L. Robison, D. R. Marcey,
and J. C. Lester, STORYEVAL: An empirical evaluation
framework for narrative generation, in Papers from the 2009
AAAI Spring Symposium, TR SS-09-06 (2009), pp. 103–110.
2. J. Zhu, Towards a mixed evaluation approach forcomputational narrative systems, in Proc. 3rd Int. Conf.
Computational Creativity (2012), pp. 150–154.
3. R. Pérez y Pérez, The three layers evaluation model forcomputer-generated plots, in Proc. 5th Int. Conf.
Computational Creativity (2014).
4. F. Peinado, V. Francisco, R. Hervás, and P. Gervás,
Assessing the novelty of computer-generated narratives using
empirical metrics, Minds & Machines, 20(4) (2010) 565–588.
5. T. Ogata, Expanded literary theory for automaticnarrative generation, in Proc. Joint 7th Int. Conf. Soft
Computing and Intelligent Systems and 15th Int. Symp.
Advanced Intelligent Systems (2014), pp. 1558–1563.
6. T. Akimoto and T. Ogata, An information design ofnarratology: The use of three literary theories in anarrative generation system, The International Journal of
Visual Design, 7(3) (2014) 31–61.
7. T. Akimoto and T. Ogata, A narratological approach fornarrative discourse: Implementation and evaluation of thesystem based on Genette and Jauss, in Proc. 34th AnnualConf. the Cognitive Science Society (2012), pp. 1272–1277.
8. T. Akimoto and T. Ogata, Towards a discourse mechanism
in narrative generation system: Proposal of a system
introducing narrative discourse theory and reception theory,
Cognitive Studies, 20(4) (2013) 396–420.
9. G. Genette, Narrative discourse: An essay in method,transl. J. E. Lewin, (Cornell University Press, NY, 1980).
10. H. R. Jauss, Literary history as a challenge to literary theory,in Toward an Aesthetic of Reception, transl. T. Bahti,(University of Minnesota Press, MN, 1982), pp. 3–45.

Published by Atlantis PressCopyright: the authors
53