# Emergentist View on Generative Narrative Cognition: Considering Principles of the Self-Organization of Mental Stories

Authors: Taisuke Akimoto
Journal: Advances in Human-Computer Interaction
 (2018)

---

## Abstract

We consider the essence of human intelligence to be the ability to mentally (internally) construct a world in the form of stories through interactions with external environments. Understanding the principles of this mechanism is vital for realizing a human-like and autonomous artificial intelligence, but there are extremely complex problems involved. From this perspective, we propose a conceptual-level theory for the computational modeling of generative narrative cognition. Our basic idea can be described as follows: stories are representational elements forming an agent’s mental world and are also living objects that have the power of self-organization. In this study, we develop this idea by discussing the complexities of the internal structure of a story and the organizational structure of a mental world. In particular, we classify the principles of the self-organization of a mental world into five types of generative actions, i.e., connective, hierarchical, contextual, gathering, and adaptive. An integrative cognition is explained with these generative actions in the form of a distributed multiagent system of stories.


---

## Body

## 1. Introduction

The computational modeling of an autonomous intelligence that can adapt to external environments (physical and social situations including other humans) is an essential issue for realizing human-like artificial intelligences. In cognitive architecture studies, computational frameworks of autonomous intelligence have been explored with biological inspirations, including psychology and neuroscience [1, 2]. In the early years of artificial intelligence, Schank and his colleagues argued the importance of narrative ability and narrativity-based memory in higher-level cognition and learning. They proposed several significant theories, including script knowledge [3] and a dynamic memory framework [4]. His dynamic memory framework demonstrated a systematic cognitive mechanism of flexible reminding (remembering), reconstruction, generalization, and organization of story-form memories. Although a large part of his idea was not implemented, it provided an important insight into the autonomous development of intelligence.Based on the above background, we assume that generative narrative cognition is an essential aspect of an autonomous intelligence, which develops through interactions with external environments. Here, generative narrative cognition refers to an agent’s mental system of dynamically generating and organizing stories for interacting and adapting to environments. In this study, we use the term “story” to refer to a mental representation of a part of the world of an agent. It is used as a concept unifying episodic memories, autobiographical memories, the contextual structures of current situations, prospective memories, planned or imagined futures, and fictive or virtual stories. On the contrary, a narrative expressed through language or other media of expression is referred to as a “narrative” or a “discourse.”We can state several reasons for the importance of generative narrative cognition in cognitive architectures. First, a story is a universal information format that integrates various informational elements, including events, entities, relationships, abstract concepts, intents, goals, emotions, nonverbal information (e.g., memories of visual images), and hypothetical events. Second, a narrative is a universal way of communicating world information with others. Third, a story forms the contextual structure of an unfolding situation involving temporal reach into the past (i.e., experiences and results of one’s actions) and future (i.e., expectations and plans). In this sense, a story is the basis of a higher-level perception-action system. Fourth, memories of past experiences become reusable knowledge when they are organized as stories. Moreover, the importance of narrative ability in cognitive architectures, artificial agents, and human-computer interaction has been discussed from various perspectives [5–8].However, the computational modeling of generative narrative cognition is an extremely complex problem that has challenged researchers for many years in artificial intelligence studies [9, 10]. Although most previous narrative generation systems have focused on the production of narrative texts such as fairy tales and literary narratives, the basic problem is common: using generative narrative cognition as the foundation of an agent’s mind. There exist several difficult problems in the computational modeling of generative narrative cognition. In particular, a story or a narrative has a complex structure, and human narrative cognition is based on a vast store of experiential knowledge, including informal, tacit, and cultural knowledge. Such a complex problem is difficult to model on the basis of classical symbolic processing. Although connectionist models including deep neural networks are applied to various domains including image-recognition systems and end-to-end natural language processing systems, this type of approach does not fit the essential part of generative narrative cognition. The critical issue of generative narrative cognition is to explore a computational framework and the underlying principles of the generation and organization of stories in the mental system of an agent. Therefore, we must seek an alternative method from a long-term perspective.In this paper, we propose a conceptual-level theory for the computational modeling of generative narrative cognition based on a type of emergentist approach. Our basic idea can be described as follows: stories are living objects having the power of self-organization. This is similar to a multiagent system. Here, an agent is not a character, but a representational element in an agent’s mental system. For example, in Minsky’sThe Society of Mind [11], the cognitive mechanism of a mind is explained as a type of distributed multiagent system based on the collaborative activities of diverse simple functional agents. However, we assume that the central agents forming a mind are stories.The rest of this paper is organized as follows. Sections2 and 3 describe the basic idea of how generative narrative cognition plays a crucial role in an autonomous intelligence. Section 4 discusses the necessity of an emergentist approach for the computational modeling of generative narrative cognition. Based on this idea, Section 5 provides a macroscopic classification of the principles of the self-organization of a story and a mental world formed by many stories. Section 6 contains concluding remarks with future research directions. Although this paper provides only conceptual descriptions, creating a vision for solving this complex problem (generative narrative cognition) is a significant step for the future of artificial intelligence.


## 2. Stories Forming an Agent’s Mental World

The motivation behind this study is based on an assumption that generative narrative cognition is an essential aspect of an autonomous artificial intelligence. From this perspective, we have been addressing the conceptual systematization of a cognitive architecture. The initial concept is presented in [12]. The key concept of our architecture is an agent’s mental world formed as an organization of many stories.


### 2.1. Story

In general terms, a story refers to the information of chronologically and semantically organized events recounted in a narrative. Here, an event refers to a character’s action or a happening (e.g., “Taro eats an apple”).The notion of story is rooted in narratological terminologies (narratology is the discipline of theoretical studies on narrative, inspired by structuralism and semiology). In terms of narratology, a “narrative” basically refers to an expression of events in a real or fictional world based on a language or other sign system [13]. However, a narrative has a close relationship with the form of mental representation of knowledge and memory. To clearly distinguish the representational aspect from an expressed narrative, we introduce the notions of story and discourse based on a reinterpretation of narratological terminology [13, 14]. The terms “story” and “discourse” are generally used to distinguish between the content and expression planes of a narrative. More precisely, a discourse refers to the narrative text itself and a story corresponds to the content, i.e., information of events recounted in a discourse or a text. However, because a story is intangible, the notion of stories is slightly unclear. From a narrative-communication perspective, the relationship between the content and expression planes of a narrative can be reinterpreted as the relationship between a mental representation and the surface expression. A sender (author or teller) writes or tells a discourse based on a story that is remembered or generated inside the mind. A receiver (reader or hearer) mentally constructs a story by interpreting or comprehending the discourse. Stories between the sender and the receiver are not the same objects.Based on the above conception, we use the term “story” as a uniform mental representation involving an episodic memory, an autobiographical memory, the contextual structure of a current situation, a prospective memory, a planned or imagined future, and a fictional or virtual story.


### 2.2. Mental World

An agent’s mental world contains individual meaning and a rich temporal extent with numerous and diverse stories, as illustrated in Figure1. A story corresponds to a piece of the world for an agent. Stories contained in a mental world can be classified from several perspectives. In the relationship with an external world, these stories include past, future, and fictive or hypothetical stories. With respect to the manner of generation, there are stories based on an agent’s own experiences (experience-based stories), imaginative power (imagined stories), and interpretation or comprehension of others’ narratives (narrative-mediated stories). A more systematic description of structural properties and functions of an agent’s mental story are presented in our previous study [15].Figure 1
An agent’s mental world formed by stories.From a perspective of computational knowledge representation, a story itself forms only a relational structure of concrete events and entities. However, we assume that the semantic aspect of a story is underpinned by associations with the following three types of mental elements:(i)
Concept: a concept corresponds to a primitive linguistic element corresponding to meanings of a word. Concepts include general concepts (i.e., nominal, verbal, adjectival, and adverbial concepts) and ontological (or proper) concepts for identical entities.(ii)
Schema: a schema refers to a generalized structure based on one or more concrete stories. A schema is a structured composition of two or more concepts or (sub)schemas. Schemas underpin top-down and abstract-level cognitions of stories. The idea of schemas is rooted in Minsky’s frame theory [16]. Script knowledge [3] and memory organization packets (MOPs) [4] form the schematic knowledge relevant to narrative cognition.(iii)
Mental image: a mental image represents nonsymbolic information, including verbal, visual, auditory, and haptic images.


## 2.1. Story

In general terms, a story refers to the information of chronologically and semantically organized events recounted in a narrative. Here, an event refers to a character’s action or a happening (e.g., “Taro eats an apple”).The notion of story is rooted in narratological terminologies (narratology is the discipline of theoretical studies on narrative, inspired by structuralism and semiology). In terms of narratology, a “narrative” basically refers to an expression of events in a real or fictional world based on a language or other sign system [13]. However, a narrative has a close relationship with the form of mental representation of knowledge and memory. To clearly distinguish the representational aspect from an expressed narrative, we introduce the notions of story and discourse based on a reinterpretation of narratological terminology [13, 14]. The terms “story” and “discourse” are generally used to distinguish between the content and expression planes of a narrative. More precisely, a discourse refers to the narrative text itself and a story corresponds to the content, i.e., information of events recounted in a discourse or a text. However, because a story is intangible, the notion of stories is slightly unclear. From a narrative-communication perspective, the relationship between the content and expression planes of a narrative can be reinterpreted as the relationship between a mental representation and the surface expression. A sender (author or teller) writes or tells a discourse based on a story that is remembered or generated inside the mind. A receiver (reader or hearer) mentally constructs a story by interpreting or comprehending the discourse. Stories between the sender and the receiver are not the same objects.Based on the above conception, we use the term “story” as a uniform mental representation involving an episodic memory, an autobiographical memory, the contextual structure of a current situation, a prospective memory, a planned or imagined future, and a fictional or virtual story.


## 2.2. Mental World

An agent’s mental world contains individual meaning and a rich temporal extent with numerous and diverse stories, as illustrated in Figure1. A story corresponds to a piece of the world for an agent. Stories contained in a mental world can be classified from several perspectives. In the relationship with an external world, these stories include past, future, and fictive or hypothetical stories. With respect to the manner of generation, there are stories based on an agent’s own experiences (experience-based stories), imaginative power (imagined stories), and interpretation or comprehension of others’ narratives (narrative-mediated stories). A more systematic description of structural properties and functions of an agent’s mental story are presented in our previous study [15].Figure 1
An agent’s mental world formed by stories.From a perspective of computational knowledge representation, a story itself forms only a relational structure of concrete events and entities. However, we assume that the semantic aspect of a story is underpinned by associations with the following three types of mental elements:(i)
Concept: a concept corresponds to a primitive linguistic element corresponding to meanings of a word. Concepts include general concepts (i.e., nominal, verbal, adjectival, and adverbial concepts) and ontological (or proper) concepts for identical entities.(ii)
Schema: a schema refers to a generalized structure based on one or more concrete stories. A schema is a structured composition of two or more concepts or (sub)schemas. Schemas underpin top-down and abstract-level cognitions of stories. The idea of schemas is rooted in Minsky’s frame theory [16]. Script knowledge [3] and memory organization packets (MOPs) [4] form the schematic knowledge relevant to narrative cognition.(iii)
Mental image: a mental image represents nonsymbolic information, including verbal, visual, auditory, and haptic images.


## 3. Dynamic Generation and Organization of Mental Stories

How generative narrative cognition plays an essential role in an agent’s intelligence can be explained from its functional generality. In particular, generative narrative cognition forms the common basis of a higher-level perception-action system, linguistic communication about world information, and formation of a self or identity. We will describe the first two aspects later in this section. The thoughts behind the third aspect (formation of a self or identity) can be found in philosophy, psychology, or other disciplines [17–19].In an autonomous intelligence, dynamic generation and organization of stories includes two aspects, i.e., (1) interaction with an environment by generating a story and (2) adaptation to environments by developing the organizational structure of a mental world. Figure2 illustrates these two aspects, and we will describe each aspect in the following two subsections. Here, assumed environments include various social and physical situations over the course of an agent’s activities, e.g., shopping at a supermarket, climbing a mountain, linguistic communication with others, housework, and creation of literary work.Figure 2
Two dynamic aspects of a mental world: (1) the generation of a story and (2) development of the organizational structure of a mental world.


### 3.1. Interaction with an Environment by Generating a Story

An agent interacts with an environment based on a story. This idea is rooted in our previous consideration of the structure of an agent’s subjective world while interacting with an environment [15]. A story for interacting with an environment can be explained through the following two perspectives:(i)
Action and perception: in an agent’s mind, interaction with an environment is based on the continual (re)construction of a story (see Figure 3(a)). In particular, acting in an environment corresponds to performing mentally constructed events placed in the future. Perceiving the movement of an environment, including the results of one’s actions, corresponds to the construction of past episodic events. Both actions and perceptions always occur in the context of a story, i.e., a chain of events across the past (experiences and results) and future (expectations and plans).(ii)
Expression and interpretation (linguistic communication): a narrative is the universal way of exchanging world information, and story generation is the core mental process in this activity (see Figure 3(b)). When an agent expresses a narrative to others, a story for the source of the narrative is remembered or imagined in the mind. The mental process of recalling includes not only memory retrieval but also flexible editing of memories (stories) according to the situation or goal of the narrative communication. On the contrary, the fundamental mental process of interpreting or comprehending another’s narrative is to compose a mental story over the course of hearing/reading it.Figure 3
Action/perception and linguistic communication based on a story.
(a)
Action and perception (b)
Linguistic communicationIt is also important that generative narrative cognition integrates the above two fundamental human activities. A human develops mental stories in various ways, e.g., the organization of one’s own experiences, imagination, and the reception of various narratives. The acquisition of stories from narratives has great significance in constructing rich world knowledge beyond one’s own experiences and imaginative power. In addition, humans cocreate world knowledge by communicating narratives, such as the histories and current states of societies and visions for the future.


### 3.2. Adaptation to Environments by Developing a Mental World

A mental world is a holistic memory system that provides knowledge resources for generating new stories. At the same time, a mental world organizes the pieces required to direct the dynamic generation of stories in various environments. The cognitive mechanism of the autonomous development of a mental world through the accumulation of experiences is the foundation of an environmental adaptability, i.e., the potential to build abilities to generate adequate stories in various environments.The developmental process of a mental world is conceptualized as the formation of a “mental space,” which is the basic organizational unit of a mental world. This concept is inspired by Schank’s dynamic memory framework [4]. In general terms, a mental space corresponds to a generalized structure of similar stories. The basic role of a mental space is to provide a framework that directs and restricts story generation for smoothly interacting with similar type of environments, e.g., shopping in a supermarket and communication about a specific theme (politics, local events, etc.). In addition, a mental space organizes memories of experiences, i.e., previously generated stories in that space, as knowledge resources for generating a new story in a similar situation.The framework of a mental space is formed by a schema, i.e., a general structure covering similar type of stories. In particular, the schema of a mental space provides a thematic and stylistic framework based on a compound of multiple subschemas. A schema adaptively forms or reforms by the generalization of stories and reorganization of existing schemas. When an agent is facing a relatively familiar environment, a story is generated in an existing mental space corresponding to that environment. The schema of this space may be adjusted or reformed according to a generated story (i.e., experience). Moreover, when an agent is facing an unfamiliar environment, the agent’s mental system tries to adapt to that environment by reorganizing one or more existing schemas to create a story with a new mental space (see Figure4). We call this specific mental process of adapting to a new environment “reorganizational differentiation.” (This notion is similar to abduction in C. S. Perce’s theory.)Figure 4
Reorganizational differentiation of a mental space.


## 3.1. Interaction with an Environment by Generating a Story

An agent interacts with an environment based on a story. This idea is rooted in our previous consideration of the structure of an agent’s subjective world while interacting with an environment [15]. A story for interacting with an environment can be explained through the following two perspectives:(i)
Action and perception: in an agent’s mind, interaction with an environment is based on the continual (re)construction of a story (see Figure 3(a)). In particular, acting in an environment corresponds to performing mentally constructed events placed in the future. Perceiving the movement of an environment, including the results of one’s actions, corresponds to the construction of past episodic events. Both actions and perceptions always occur in the context of a story, i.e., a chain of events across the past (experiences and results) and future (expectations and plans).(ii)
Expression and interpretation (linguistic communication): a narrative is the universal way of exchanging world information, and story generation is the core mental process in this activity (see Figure 3(b)). When an agent expresses a narrative to others, a story for the source of the narrative is remembered or imagined in the mind. The mental process of recalling includes not only memory retrieval but also flexible editing of memories (stories) according to the situation or goal of the narrative communication. On the contrary, the fundamental mental process of interpreting or comprehending another’s narrative is to compose a mental story over the course of hearing/reading it.Figure 3
Action/perception and linguistic communication based on a story.
(a)
Action and perception (b)
Linguistic communicationIt is also important that generative narrative cognition integrates the above two fundamental human activities. A human develops mental stories in various ways, e.g., the organization of one’s own experiences, imagination, and the reception of various narratives. The acquisition of stories from narratives has great significance in constructing rich world knowledge beyond one’s own experiences and imaginative power. In addition, humans cocreate world knowledge by communicating narratives, such as the histories and current states of societies and visions for the future.


## 3.2. Adaptation to Environments by Developing a Mental World

A mental world is a holistic memory system that provides knowledge resources for generating new stories. At the same time, a mental world organizes the pieces required to direct the dynamic generation of stories in various environments. The cognitive mechanism of the autonomous development of a mental world through the accumulation of experiences is the foundation of an environmental adaptability, i.e., the potential to build abilities to generate adequate stories in various environments.The developmental process of a mental world is conceptualized as the formation of a “mental space,” which is the basic organizational unit of a mental world. This concept is inspired by Schank’s dynamic memory framework [4]. In general terms, a mental space corresponds to a generalized structure of similar stories. The basic role of a mental space is to provide a framework that directs and restricts story generation for smoothly interacting with similar type of environments, e.g., shopping in a supermarket and communication about a specific theme (politics, local events, etc.). In addition, a mental space organizes memories of experiences, i.e., previously generated stories in that space, as knowledge resources for generating a new story in a similar situation.The framework of a mental space is formed by a schema, i.e., a general structure covering similar type of stories. In particular, the schema of a mental space provides a thematic and stylistic framework based on a compound of multiple subschemas. A schema adaptively forms or reforms by the generalization of stories and reorganization of existing schemas. When an agent is facing a relatively familiar environment, a story is generated in an existing mental space corresponding to that environment. The schema of this space may be adjusted or reformed according to a generated story (i.e., experience). Moreover, when an agent is facing an unfamiliar environment, the agent’s mental system tries to adapt to that environment by reorganizing one or more existing schemas to create a story with a new mental space (see Figure4). We call this specific mental process of adapting to a new environment “reorganizational differentiation.” (This notion is similar to abduction in C. S. Perce’s theory.)Figure 4
Reorganizational differentiation of a mental space.


## 4. Necessity of an Emergentist Approach to Generative Narrative Cognition

Computationally implementing a dynamic mental world described in the previous section is a hugely complex problem. For solving this problem, we argue the necessity of an emergentist approach.


### 4.1. Structural Complexity of a Story

The necessity of an emergentist approach can be described from the following two perspectives:First, from a cognitive perspective, the developmental process of a mental world (or a memory system in general terms) is generally assumed as a type of self-organization phenomenon. The ability to build diverse mental spaces with new intellectual functions is a key aspect of a dynamic mental world. These functions should not be externally embedded but emerge through adaptive interactions with environments.The second reason refers to the structural complexity of a story itself. A story as a world representation can be viewed as a complex structural object in which events (a character’s action or a happening) and entities (an individual existence including a character, object, and place) are organically organized. Although the central elements of a story are events, these events are potentially accompanied by various types of informational elements, e.g., relationships, abstract concepts, intents, goals, emotions, nonverbal information, and hypothetical events. Because of this property, it is assumed that the cognitive process of story generation is based on a flexible collaboration of multiple cognitive modules. In addition, the structure of a story involves interdependencies in the whole-part and part-part relationships. In a well-organized story, for example, a small change in a story’s part may cause incoherence in the story’s whole structure or the lack of contextual coherence over the course of its events. Based on an extensional reinterpretation of our hierarchical graph model of multidimensional narrative structure [20] and a structural conception of an agent’s subjective world [15], we arrange general structural properties of a story as follows (Figure 5 illustrates these notions):(a)
A story has a hierarchical whole-part structure. A higher-level part (e.g., a scene, a semantic or functional unit, and a larger section) of a story is formed from two or more lower-level parts (e.g., events and scenes).(b)
Parts of a story (including events) are mutually connected by temporal, causal, or other types of relationships.(c)
There are two types of interdependencies in a story’s hierarchical structure: (1) vertical interdependencies, including top-down restriction (from a higher-level part to the lower-level parts) and bottom-up abstraction (from lower-level parts to the higher-level part) and (2) horizontal interdependencies based on the contextual coherence between parts.(d)
A story contains a story world, i.e., the organization of entities relevant to the story. It is similar to the setting of the world and there exists an interdependency between a story and the story world.(e)
A story is formed based on the reconstructive reuse of existing mental resources including relationships with other stories. In this sense, a story itself is not an independent mental object.(f)
There exists an interdependency between a story and an external environment. In particular, a story based on one’s own experiences is formed based on perceived environmental information. At the same time, the expectational aspect of the agent’s story directs or restricts the perception of environmental information.Figure 5
Structural complexity of a story. Dot-lines represent interdependencies or relationships among a story’s parts and other mental objects.In the above six items, (e) is derived from the discussion in Section3.2 and (f) is an ordinary notion from a cognitive perspective. To explain notions (a)–(d), the next subsection uses an example story structure.


### 4.2. Example Story Structure

Box1 shows an English expression of a simple story written by the author. An example of how the structure of this story text can be interpreted is shown in Figure 6. In this structural representation, each sentence in Box 1 is simply imagined as an event. From the perspectives of (a)–(d) in the above list, we can examine the cognitive processes of creating and manipulating this fictional story.(i)
Hierarchical structure (a): this story is divided into three scenes (intermediate parts): the setting (daily life), the stealing by Sally, and the revenging of the theft by Lisa.(ii)
Relationship between parts (b): there are anteroposterior and causal relationships between the different parts. The anteroposterior relationships are depicted as arrowed lines. The block arrow from the second scene to the third scene represents the causal relationship, i.e., Lisa’s reason or motivation for taking revenge.(iii)
Vertical interdependency (c): a change in a part may cause a change in the higher- and/or lower-level parts. For example, when event s4 is changed to “Sally put a cookie beside the chocolate,” it will propagate and influence the meaning of the second scene, e.g., “gift to Lisa.”(iv)
Horizontal interdependency (c): a change in a part also propagates in horizontal directions. For example, when the second scene is changed to “gift to Lisa,” the third scene, “revenging of the theft by Lisa,” becomes an unnatural reaction (at least from our common-sense perspective). This inconsistency causes a global reorganization of a story.(v)
Relationship with the story world (d): in the structural representation in Figure 6, the story world contains several entities, i.e., Lisa, Sally, a desk, a chocolate, and a doll, and their relationships such as “Lisa and Sally are sisters” and “Lisa and Sally like chocolate.” A change in the setting of the world propagates to events or the story’s parts. For example, if the setting is changed to a fantasy world (e.g., Lisa is a witch and all children like dried frogs), the chocolate element will become a dried frog and the way Lisa takes revenge will change to become a magical attack.Box 1:English expression of an example story.
(s1) Lisa and Sally are sisters that get along well. (s2) They play together every day. (s3) One day, Sally found a chocolate in Lisa’s desk when Lisa was not in the room. (s4) Sally stole and ate the chocolate. (s5) The next day, Lisa saw that the chocolate was gone. (s6) Lisa assumed that the chocolate was eaten by Sally. (s7) Then, Lisa threw Sally’s doll out a window. (s8) Sally cried.Figure 6
A structural representation of the example story in Box1 (s1–8 in the small circles correspond to sentences in Box 1).


### 4.3. Emergentist Approach

In studies of computational narrative generation, an orthodox approach is to model the process of generating a story (or a narrative) based on a centrally controlled symbolic processing (see Figure7(a)). For example, narrative generation using a story grammar is a traditional story-generation method [21, 22]. The simplest implementation is a pipelined procedure of composing a story using a story grammar in a top-down manner, from the abstract structure to the detailed contents. However, this type of inflexible framework will be limited when modeling a complex and dynamic narrative cognition and does not lead to our objective. An advanced challenge is to model narrative generation as a more flexible procedure. For example, a blackboard architecture (see Figure 7(b)) that shares a processing object, i.e., a story, among various types of cognitive modules has the potential of flexible collaboration of cognitive modules in generating a story (e.g., [23]). However, a blackboard architecture requires knowledge of the principles of directing the collaborative cognitive activities, and how to model it remains a difficult problem.Figure 7
A centrally controlled architecture and a blackboard architecture.
(a)
Centrally controlled architecture (b)
Blackboard architectureTherefore, we propose an alternative approach as an extension of the blackboard architecture. Our basic idea is to build the power of self-organization into stories and mental spaces themselves. Figure8 illustrates this concept. Each part at each level in a story structure has the power of generating one’s own structure in the relationship with other mental objects. The story’s whole structure emerges from distributed collaborative functioning of the parts (see Figure 8a). On the contrary, a mental space forms one’s own schema through interactions with stories and other mental spaces (see Figure 8b). We introduce the term “generative actions” to refer to the basic principles of driving these self-organization activities. A mental world is developed by the distributed collaborative functioning of the generative actions of stories and mental spaces.Figure 8
Powers of the self-organization in a mental world.


## 4.1. Structural Complexity of a Story

The necessity of an emergentist approach can be described from the following two perspectives:First, from a cognitive perspective, the developmental process of a mental world (or a memory system in general terms) is generally assumed as a type of self-organization phenomenon. The ability to build diverse mental spaces with new intellectual functions is a key aspect of a dynamic mental world. These functions should not be externally embedded but emerge through adaptive interactions with environments.The second reason refers to the structural complexity of a story itself. A story as a world representation can be viewed as a complex structural object in which events (a character’s action or a happening) and entities (an individual existence including a character, object, and place) are organically organized. Although the central elements of a story are events, these events are potentially accompanied by various types of informational elements, e.g., relationships, abstract concepts, intents, goals, emotions, nonverbal information, and hypothetical events. Because of this property, it is assumed that the cognitive process of story generation is based on a flexible collaboration of multiple cognitive modules. In addition, the structure of a story involves interdependencies in the whole-part and part-part relationships. In a well-organized story, for example, a small change in a story’s part may cause incoherence in the story’s whole structure or the lack of contextual coherence over the course of its events. Based on an extensional reinterpretation of our hierarchical graph model of multidimensional narrative structure [20] and a structural conception of an agent’s subjective world [15], we arrange general structural properties of a story as follows (Figure 5 illustrates these notions):(a)
A story has a hierarchical whole-part structure. A higher-level part (e.g., a scene, a semantic or functional unit, and a larger section) of a story is formed from two or more lower-level parts (e.g., events and scenes).(b)
Parts of a story (including events) are mutually connected by temporal, causal, or other types of relationships.(c)
There are two types of interdependencies in a story’s hierarchical structure: (1) vertical interdependencies, including top-down restriction (from a higher-level part to the lower-level parts) and bottom-up abstraction (from lower-level parts to the higher-level part) and (2) horizontal interdependencies based on the contextual coherence between parts.(d)
A story contains a story world, i.e., the organization of entities relevant to the story. It is similar to the setting of the world and there exists an interdependency between a story and the story world.(e)
A story is formed based on the reconstructive reuse of existing mental resources including relationships with other stories. In this sense, a story itself is not an independent mental object.(f)
There exists an interdependency between a story and an external environment. In particular, a story based on one’s own experiences is formed based on perceived environmental information. At the same time, the expectational aspect of the agent’s story directs or restricts the perception of environmental information.Figure 5
Structural complexity of a story. Dot-lines represent interdependencies or relationships among a story’s parts and other mental objects.In the above six items, (e) is derived from the discussion in Section3.2 and (f) is an ordinary notion from a cognitive perspective. To explain notions (a)–(d), the next subsection uses an example story structure.


## 4.2. Example Story Structure

Box1 shows an English expression of a simple story written by the author. An example of how the structure of this story text can be interpreted is shown in Figure 6. In this structural representation, each sentence in Box 1 is simply imagined as an event. From the perspectives of (a)–(d) in the above list, we can examine the cognitive processes of creating and manipulating this fictional story.(i)
Hierarchical structure (a): this story is divided into three scenes (intermediate parts): the setting (daily life), the stealing by Sally, and the revenging of the theft by Lisa.(ii)
Relationship between parts (b): there are anteroposterior and causal relationships between the different parts. The anteroposterior relationships are depicted as arrowed lines. The block arrow from the second scene to the third scene represents the causal relationship, i.e., Lisa’s reason or motivation for taking revenge.(iii)
Vertical interdependency (c): a change in a part may cause a change in the higher- and/or lower-level parts. For example, when event s4 is changed to “Sally put a cookie beside the chocolate,” it will propagate and influence the meaning of the second scene, e.g., “gift to Lisa.”(iv)
Horizontal interdependency (c): a change in a part also propagates in horizontal directions. For example, when the second scene is changed to “gift to Lisa,” the third scene, “revenging of the theft by Lisa,” becomes an unnatural reaction (at least from our common-sense perspective). This inconsistency causes a global reorganization of a story.(v)
Relationship with the story world (d): in the structural representation in Figure 6, the story world contains several entities, i.e., Lisa, Sally, a desk, a chocolate, and a doll, and their relationships such as “Lisa and Sally are sisters” and “Lisa and Sally like chocolate.” A change in the setting of the world propagates to events or the story’s parts. For example, if the setting is changed to a fantasy world (e.g., Lisa is a witch and all children like dried frogs), the chocolate element will become a dried frog and the way Lisa takes revenge will change to become a magical attack.Box 1:English expression of an example story.
(s1) Lisa and Sally are sisters that get along well. (s2) They play together every day. (s3) One day, Sally found a chocolate in Lisa’s desk when Lisa was not in the room. (s4) Sally stole and ate the chocolate. (s5) The next day, Lisa saw that the chocolate was gone. (s6) Lisa assumed that the chocolate was eaten by Sally. (s7) Then, Lisa threw Sally’s doll out a window. (s8) Sally cried.Figure 6
A structural representation of the example story in Box1 (s1–8 in the small circles correspond to sentences in Box 1).


## 4.3. Emergentist Approach

In studies of computational narrative generation, an orthodox approach is to model the process of generating a story (or a narrative) based on a centrally controlled symbolic processing (see Figure7(a)). For example, narrative generation using a story grammar is a traditional story-generation method [21, 22]. The simplest implementation is a pipelined procedure of composing a story using a story grammar in a top-down manner, from the abstract structure to the detailed contents. However, this type of inflexible framework will be limited when modeling a complex and dynamic narrative cognition and does not lead to our objective. An advanced challenge is to model narrative generation as a more flexible procedure. For example, a blackboard architecture (see Figure 7(b)) that shares a processing object, i.e., a story, among various types of cognitive modules has the potential of flexible collaboration of cognitive modules in generating a story (e.g., [23]). However, a blackboard architecture requires knowledge of the principles of directing the collaborative cognitive activities, and how to model it remains a difficult problem.Figure 7
A centrally controlled architecture and a blackboard architecture.
(a)
Centrally controlled architecture (b)
Blackboard architectureTherefore, we propose an alternative approach as an extension of the blackboard architecture. Our basic idea is to build the power of self-organization into stories and mental spaces themselves. Figure8 illustrates this concept. Each part at each level in a story structure has the power of generating one’s own structure in the relationship with other mental objects. The story’s whole structure emerges from distributed collaborative functioning of the parts (see Figure 8a). On the contrary, a mental space forms one’s own schema through interactions with stories and other mental spaces (see Figure 8b). We introduce the term “generative actions” to refer to the basic principles of driving these self-organization activities. A mental world is developed by the distributed collaborative functioning of the generative actions of stories and mental spaces.Figure 8
Powers of the self-organization in a mental world.


## 5. Classification of the Generative Actions of a Story and Mental Space

Based on the above concept, we can classify generative actions in a mental world. According to the structural properties of a story and the notion of a mental space, we can classify generative actions into five basic types:connective,hierarchical,contextual,gathering, andadaptive. The last type (adaptive) drives the self-organization of a mental space. The first four types, which drive the self-organization of a story, are derived from the structural complexity of a story; this is described in Sections 4.1 and 4.2. In particular, the connective action is relevant to (b) and (c), the hierarchical action is relevant to (a) and (c), and the contextual and gathering actions are relevant to (c)–(f).In the following subsections, the concept and principles of each basic type are presented. The presented principles are basically rooted in general notions of cognitive science and artificial intelligence studies such as abstraction, generalization, analogy, schema, and relationship. However, the main contribution of the following ideas is to provide a conceptual-level theory of an integrative cognition from an emergentist perspective.


### 5.1. Connective Actions: Relation

A connective action is the most fundamental action of organizing events. The basic principle of making a connection is a relationship between a story’s parts. The major relationship types in a story structure are as follows (see also Figure9):(i)
Temporal relationship: this denotes a relative temporal relationship between two parts. Allen [24] classified temporal relationships between two events or actions into “before,” “equal,” “meets,” “overlaps,” “during,” “starts,” “finishes,” and their inverses, on the basis of anteroposterior relationships and temporal intervals of events. This classification can be adopted to temporal relationships in a story.(ii)
Causal relationship: this denotes a causal relationship between any two parts of a story.(iii)
Other type of relationships: because the computational modeling of relationships in a story structure or narrative is an essential but complex problem, further consideration is required. For example, various types of relationships in a discourse structure are proposed in the area of natural language processing, e.g., Hobbs’s coherence relations [25] and the rhetorical structure theory by Mann and Thompson [26].Figure 9
Connective actions.


### 5.2. Hierarchical Actions: Top-Down Restriction and Bottom-Up Abstraction

Hierarchical actions form or reform upper- and lower-level structures in a story. They are the basis of the vertical interdependency between upper- and lower-level parts. Hierarchical actions can be broadly classified into top-down restriction and bottom-up abstraction (see also Figure10):(i)
Top-down restriction: a higher-level part restricts the lower-level structure by creating the expectation of a blank based on a schema or similar story that is associated with the higher-level part. An expectation refers to the generation of subsequent information in a dynamic story-generation process during interactions with an environment. In particular, an expectation drives the perception of environmental information based on a schema or similar story. A blank denotes a lack of information in the lower-level structure, and it drives the filling of that part using a schema or similar story.(ii)
Bottom-up abstraction: in this action, a higher-level part is formed or reformed from an aggregation of lower-level parts. This type of cognition, which generates higher-level meaning from lower-level elements, is generally referred to as abstraction [27]. In a story, abstractions arise in various structural levels, i.e., temporal segmentation of events from sensory information and the formation of a higher-level meaning from two or more events or parts. Whereas an abstraction is a bottom-up action, it always functions under top-down restriction. In particular, a higher-level structure (the meaning and power of top-down restriction) is formed by matching and reorganizational diversion of associated mental resources, i.e., a schema or another story’s part that is similar to the lower-level structure.Figure 10
Hierarchical actions.


### 5.3. Contextual Actions: Associations of Events and States

Using contextual actions, a part draws contextual information relevant to the organization of one’s own structure. These actions generate horizontal interdependency in a story. Contextual actions can be classified into the following subclasses, from the perspective of the position of source information (see also Figure11):(i)
Past: a part draws relevant past events (e.g., the background or reason behind an action by the agent or another character).(ii)
Future: a part draws relevant future events (e.g., an intent, desire, objective, and goal for the agent’s action).(iii)
State: a part draws relevant stative information from the story world (e.g., a character’s emotion and whether conditions).(iv)
Another’s mental story: a part associates another’s imagined mental story. This is relevant to the ability to imagine others’ mental states, known as the theory of mind. From a structural perspective, an imagined mental story about another person is represented as a nesting of stories (“a story within a story”) [15].Figure 11
Contextual actions.


### 5.4. Gathering Actions: Mental Space, Similarity/Analogy, and Perception

Gathering actions gather mental resources externally (memories including stories, schemas, mental images, and concepts) or environmental information (perception) for generating their own structures of a part. These mental resources are used by other types of generative actions, as materials, general structures, or cases. Gathering actions include the following three subclasses (see also Figure12):(i)
Mental space: a story’s part gathers mental resources from the mental space in which the story is generated. As we described in Section 3.2, a mental space organizes relevant knowledge including (sub)schemas and stories for generating stories in similar environments.(ii)
Similarity/analogy: similarity or analogy is a key principle for the flexible reuse of existing mental resources across boundaries of mental spaces or problem domains. Particularly, analogy is an essential human cognition for reusing mental resources by making a structural correspondence between two different representational elements [28–30]. Case-based reasoning [31] is also rooted in analogical cognition.(iii)
Perception: perception is the action of gathering environmental information. It is basically driven by an expectation as a top-down restriction. When a part gets unexpected information from an environment, reformation of part of the story may be done to maintain coherence.Figure 12
Gathering actions.


### 5.5. Adaptive Actions: Generalization and Differentiation

Adaptive actions form or reform the schema of a mental space to adapt environments. Whereas the above four types are a story’s activities, adaptive actions are the activities of a mental space. Adaptive actions can be classified into the following three types (see also Figure13):(i)
Inductive generalization: a mental space forms or reforms one’s own schema based on structural commonality among stories in that space. Abstraction of a story (forming a higher-level structure of a story) and analogies between stories (creating structural correspondences between stories) provide the basis for this action.(ii)
Failure-based generalization: a mental space adjusts one’s own schema according to a failure in interacting with an environment. A failure refers to a type of negative feedback from an environment. A similar concept is discussed in Schank’s dynamic memory framework [4]. It is also relevant to the general notion of reinforcement learning.(iii)
Reorganizational differentiation: when an agent faces an unfamiliar environment that existing mental spaces do not cover, a new mental space emerges in the agent’s mental world to generate a story for interacting that environment. This new mental space is formed by the reorganization of existing mental spaces or mental resources. We call this mental action reorganizational differentiation. This is the essence of the ability to develop by adapting to environments. At the same time, this is the fundamental principle of creativity, which seeks to construct new ideas, styles, social environments, etc.Figure 13
Adaptive actions.


### 5.6. Integrative View

As we described previously, the key point of the proposed theory is to integrate various types of cognition in the form of a distributed multiagent system of story generation. Because this idea reflects a complex system, it is difficult to show a concrete image of the system’s holistic behavior until the idea is computationally implemented. However, a simplified image of the integrative operations of various cognitive processes can be seen from the example story presented in Section4.2.From an integrative perspective, the following two notions will be the key points for the computational implementation of this theory:(i)
The main agent of the system is an “event” (in a broad sense) formed at various levels of abstraction, e.g., a character’s action (Lisa threw away Sally’s doll), a scene or a semantic segment of actions (Lisa took revenge on Sally), and the story itself.(ii)
Because the connection between agents, i.e., events or stories, is a foundation of various generative actions—connective, contextual, gathering, and adaptive—and the integrated operations of these generative actions, the agents must be associated with each other.


## 5.1. Connective Actions: Relation

A connective action is the most fundamental action of organizing events. The basic principle of making a connection is a relationship between a story’s parts. The major relationship types in a story structure are as follows (see also Figure9):(i)
Temporal relationship: this denotes a relative temporal relationship between two parts. Allen [24] classified temporal relationships between two events or actions into “before,” “equal,” “meets,” “overlaps,” “during,” “starts,” “finishes,” and their inverses, on the basis of anteroposterior relationships and temporal intervals of events. This classification can be adopted to temporal relationships in a story.(ii)
Causal relationship: this denotes a causal relationship between any two parts of a story.(iii)
Other type of relationships: because the computational modeling of relationships in a story structure or narrative is an essential but complex problem, further consideration is required. For example, various types of relationships in a discourse structure are proposed in the area of natural language processing, e.g., Hobbs’s coherence relations [25] and the rhetorical structure theory by Mann and Thompson [26].Figure 9
Connective actions.


## 5.2. Hierarchical Actions: Top-Down Restriction and Bottom-Up Abstraction

Hierarchical actions form or reform upper- and lower-level structures in a story. They are the basis of the vertical interdependency between upper- and lower-level parts. Hierarchical actions can be broadly classified into top-down restriction and bottom-up abstraction (see also Figure10):(i)
Top-down restriction: a higher-level part restricts the lower-level structure by creating the expectation of a blank based on a schema or similar story that is associated with the higher-level part. An expectation refers to the generation of subsequent information in a dynamic story-generation process during interactions with an environment. In particular, an expectation drives the perception of environmental information based on a schema or similar story. A blank denotes a lack of information in the lower-level structure, and it drives the filling of that part using a schema or similar story.(ii)
Bottom-up abstraction: in this action, a higher-level part is formed or reformed from an aggregation of lower-level parts. This type of cognition, which generates higher-level meaning from lower-level elements, is generally referred to as abstraction [27]. In a story, abstractions arise in various structural levels, i.e., temporal segmentation of events from sensory information and the formation of a higher-level meaning from two or more events or parts. Whereas an abstraction is a bottom-up action, it always functions under top-down restriction. In particular, a higher-level structure (the meaning and power of top-down restriction) is formed by matching and reorganizational diversion of associated mental resources, i.e., a schema or another story’s part that is similar to the lower-level structure.Figure 10
Hierarchical actions.


## 5.3. Contextual Actions: Associations of Events and States

Using contextual actions, a part draws contextual information relevant to the organization of one’s own structure. These actions generate horizontal interdependency in a story. Contextual actions can be classified into the following subclasses, from the perspective of the position of source information (see also Figure11):(i)
Past: a part draws relevant past events (e.g., the background or reason behind an action by the agent or another character).(ii)
Future: a part draws relevant future events (e.g., an intent, desire, objective, and goal for the agent’s action).(iii)
State: a part draws relevant stative information from the story world (e.g., a character’s emotion and whether conditions).(iv)
Another’s mental story: a part associates another’s imagined mental story. This is relevant to the ability to imagine others’ mental states, known as the theory of mind. From a structural perspective, an imagined mental story about another person is represented as a nesting of stories (“a story within a story”) [15].Figure 11
Contextual actions.


## 5.4. Gathering Actions: Mental Space, Similarity/Analogy, and Perception

Gathering actions gather mental resources externally (memories including stories, schemas, mental images, and concepts) or environmental information (perception) for generating their own structures of a part. These mental resources are used by other types of generative actions, as materials, general structures, or cases. Gathering actions include the following three subclasses (see also Figure12):(i)
Mental space: a story’s part gathers mental resources from the mental space in which the story is generated. As we described in Section 3.2, a mental space organizes relevant knowledge including (sub)schemas and stories for generating stories in similar environments.(ii)
Similarity/analogy: similarity or analogy is a key principle for the flexible reuse of existing mental resources across boundaries of mental spaces or problem domains. Particularly, analogy is an essential human cognition for reusing mental resources by making a structural correspondence between two different representational elements [28–30]. Case-based reasoning [31] is also rooted in analogical cognition.(iii)
Perception: perception is the action of gathering environmental information. It is basically driven by an expectation as a top-down restriction. When a part gets unexpected information from an environment, reformation of part of the story may be done to maintain coherence.Figure 12
Gathering actions.


## 5.5. Adaptive Actions: Generalization and Differentiation

Adaptive actions form or reform the schema of a mental space to adapt environments. Whereas the above four types are a story’s activities, adaptive actions are the activities of a mental space. Adaptive actions can be classified into the following three types (see also Figure13):(i)
Inductive generalization: a mental space forms or reforms one’s own schema based on structural commonality among stories in that space. Abstraction of a story (forming a higher-level structure of a story) and analogies between stories (creating structural correspondences between stories) provide the basis for this action.(ii)
Failure-based generalization: a mental space adjusts one’s own schema according to a failure in interacting with an environment. A failure refers to a type of negative feedback from an environment. A similar concept is discussed in Schank’s dynamic memory framework [4]. It is also relevant to the general notion of reinforcement learning.(iii)
Reorganizational differentiation: when an agent faces an unfamiliar environment that existing mental spaces do not cover, a new mental space emerges in the agent’s mental world to generate a story for interacting that environment. This new mental space is formed by the reorganization of existing mental spaces or mental resources. We call this mental action reorganizational differentiation. This is the essence of the ability to develop by adapting to environments. At the same time, this is the fundamental principle of creativity, which seeks to construct new ideas, styles, social environments, etc.Figure 13
Adaptive actions.


## 5.6. Integrative View

As we described previously, the key point of the proposed theory is to integrate various types of cognition in the form of a distributed multiagent system of story generation. Because this idea reflects a complex system, it is difficult to show a concrete image of the system’s holistic behavior until the idea is computationally implemented. However, a simplified image of the integrative operations of various cognitive processes can be seen from the example story presented in Section4.2.From an integrative perspective, the following two notions will be the key points for the computational implementation of this theory:(i)
The main agent of the system is an “event” (in a broad sense) formed at various levels of abstraction, e.g., a character’s action (Lisa threw away Sally’s doll), a scene or a semantic segment of actions (Lisa took revenge on Sally), and the story itself.(ii)
Because the connection between agents, i.e., events or stories, is a foundation of various generative actions—connective, contextual, gathering, and adaptive—and the integrated operations of these generative actions, the agents must be associated with each other.


## 6. Concluding Remarks

We proposed a new approach for the computational modeling of an autonomous intelligence based on generative narrative cognition. Throughout this paper, we conceptualized the developmental and generative process of an agent’s mind as a type of self-organization on both levels: the organizational structure of a mental world and the internal structure of a story. Under this concept, as the principles of the self-organization of a mental world, we presented five types of generative actions: connective, hierarchical, contextual, gathering, and adaptive actions. Although the mechanisms of these generative actions are still abstract and implementing the proposed concept remains a distant goal, we showed the total picture of a mental system using the above concept.The basic direction of future work will be to develop a theory of integrating generative actions in the form of a multiagent system. Agents of generative actions (in the mental system of an artificial agent) are stories and mental spaces. Although we listed many generative actions in Section5, there are several key issues relevant to a wide array of mental activities. For example, similarity or analogy of stories is a common principle of reusing existing mental resources in various types of generative actions such as gathering actions, top-down restriction, inductive generalization, and reorganizational differentiation. Moreover, in terms of knowledge representation, how a story represents the integrated world information in an agent’s mind is still unclear. The structure of a story itself, including relationships with other mental elements, needs to be considered more closely.



---

*Source: 6780564-2018-11-12.xml*