CONTRIBUTION OF INTERNAL REFLECTION IN LANGUAGE EMERGENCE WITH AN UNDER-RESTRICTED SITUATION

1 INTRODUCTION . For artificial intelligence studies , it is important to build machines that can handle words and concepts ( Bruner , 1981 ; Mikolov et al. , 2016 ) . Deep learning , especially the attention mechanism , has improved the success of various natural language processing tasks ( Xu et al. , 2018 ; Das et al. , 2018 ; Mao et al. , 2019 ; Das et al. , 2017a ; Huang et al. , 2016 ) . Furthermore , studies on multi-modal information are progressing . Supervised learning based on large datasets regards languages as statistical pattern processing with discrete symbols . Humans , as well as animals , have emerged information and concepts through the exchange of languages ( Lupyan & Bergen , 2016 ; Lupyan & Clark , 2015 ) . Languages enable the understanding of the intentions of others and a socially common system to be built ( Bloom , 2002 ; Frank et al. , 2009 ; Tomasello , 2010 ) . In addition , languages can be used to recursively construct concepts and create new ones ( Berwick & Chomsky , 2016 ; Hauser et al. , 2002 ) . Language intelligence involves more than statistically predicting words and sentences . It is important for the intelligence to formulate and systematize a language from contexts and situations . Introducing the language system using the current datasets has been difficult in machine learning ( Lake et al. , 2017 ; Niven & Kao , 2019 ; Dua et al. , 2019 ) . Researchers have attempted to build agents to create the language system by communicating with each agent instead of language processing using statistical information ( Steels , 2003 ; Mikolov et al. , 2016 ; Rahwan et al. , 2019 ) . Complex tasks could be solved using these generated messages . Formal rules and messages that emerge owing to this communication have a system that differs from that of a human . However , current language emergence studies have not achieved the autonomous grounding of mental images as those of humans and animals ( Kottur et al. , 2017 ; Bouchacourt & Baroni , 2018 ) . Two issues remain : ( 1 ) The correspondence of generated messages to concepts is mediated by the architecture . Messages can not be directly grounded to concepts . In some cases , a meaningless message was generated , and in other cases , a meaningful message was assigned to random noise images . ( 2 ) In current studies , the “ supervisor ” role is essential because agents are rewarded for acquiring the language system when communication is established . The supervisor method provided implicit answers such as “ direction of correct communication for task resolutions ” and “ direction of a correct language system corresponding to a concept. ” Few studies have focused on language systems that emerge from prior knowledge regarding pure machines . These issues appear in language emergence models , which are different from the emergence and learning of human languages . Humans have prior knowledge of the concept and corresponding grounding messages ( Imai & Masuda , 2013 ; Milligan et al. , 2007 ) . They can create language systems independently from the supervisor ( Senghas et al. , 2004 ) . Little has been reported on language emergence under these conditions , in which the agent possesses individual prior knowledge and is independent of the supervisor . The purpose of this study is to confirm a compositional language emergence method , in which machines autonomously ground concepts to each agent in an almost human-like manner . We developed a method and architecture that can generate a language from discrete symbols . In a human-like manner , agents must autonomously generate messages corresponding to cluster images that are not fixed at one point . To create formal rules within these under-restricted situations , we introduce the human cognitive bias ( Brennan & Clark , 1996 ; Lake et al. , 2019 ) . Using the generated messages and the reconstructed images , we compared the characteristics of the agreed formal rules by bias and discovered the emergent parameters necessary for the rules . 2 RELATED WORK . Artificial life studies of language emergence have discovered the dynamism of the generated language and the condition under which language systematization and formality has emerged ( Steels , 2000 ; Martin & White , 2003 ; Nolfi & Mirolli , 2009 ) . Recently , a neural network model has been used to generate a complex language using reinforcement learning or supervised learning . It is classified into ( 1 ) the task solution type and ( 2 ) the concept correspondence type . In the task solution type , multiple agents are prepared , and the tasks are solved by communicating each other ’ s situations and actions . The agents materialize words ( instructions ) to generate appropriate actions ( Mordatch & Abbeel , 2018 ; Foerster et al. , 2016 ) . Additionally , the tasks are solved through combinations of agent communication by generating answers to specific questions , such as VQA methods ( Das et al. , 2017b ; Evtimova et al. , 2018 ; Sukhbaatar et al. , 2016 ) . The concept correspondence type generate messages corresponding to objects recognized through a reinforcement learning ( RL ) -based approach ( Havrylov & Titov , 2017 ; Lazaridou et al. , 2016 ; Cao et al. , 2018 ; Lazaridou et al. , 2018 ) . This approach comprises different modules for generating and learning messages . It converges on a common language because the reward is transmitted to each module . However , the RL-based approach suffers from internal consistency ( Choi et al. , 2018 ; Kottur et al. , 2017 ; Bouchacourt & Baroni , 2018 ; Lowe et al. , 2019 ) . Choi et al . ( 2018 ) proposed the obverter technique to generate compositional languages . The obverter technique , which was proposed by Batali ( 1998 ) , can solve “ the problem of internal consistency in languages ” in the RL-based approach and enables language emergence based on “ the theory of mind ( Premack & Woodruff , 1978 ) . ” However , in Batali ’ s study , experiments were conducted under strong constraints that allowed the corresponding concepts to be uniquely determined . Language emergence under conditions without disentanglement information has not been studied . In Choi et al ’ s study , supervised signal was required for generation . Our study is inspired by the studies of Batali ( 1998 ) and Choi et al . ( 2018 ) . In this study , we extend the abovementioned studies to conceptually generate corresponding massages with formal rules without a human supervisor . The formation process and concept correspondence of human communication are useful as reference . Human language acquisition does not involve a supervisor in generating new words . Furthermore , it is not characterized by not learning concepts and words simultaneously ( Imai & Masuda , 2013 ; Milligan et al. , 2007 ) . The emergence of sign languages in Africa ( Senghas et al. , 2004 ) and the formation of languages such as the Pidgin and Creole languages demonstrate the unsupervised and asynchronous nature in language systems . Additionally , it is known that animals acquire different formalities depending on the species ( Spierings & ten Cate , 2016 ) . 3 METHOD . 3.1 MULTI-AGENTS DESCRIPTION GAME . The description game consists of a teacher agent that generates messages and a student agent that receives and learns the messages ( Lazaridou et al. , 2016 ; Choi et al. , 2018 ) . The student receives the messages from the teacher and associates them with the presented image . Repeating this process creates a language with concepts and message shared between agents . We herein propose the multi-agent description game to establish a language system for multiple agents without a human supervisor . The game proceeds as follows for each epoch . ( 1 ) One teacher is randomly selected from all agents . ( 2 ) The teacher agent randomly selects one image from a list and generates the corresponding messages . ( 3 ) The teacher agent sends the image and message to the student agent , and the student agent associated the message with the image . ( 4 ) ( 1 ) to ( 3 ) are performed a specified number of times . One hundred agents are prepared . No image labels are presented between in-game agents . Therefore , the agent must associate the message with the representation read from the image . There is no guarantee that a message will completely correspond to a label1 . The image object used in this study was the MNIST database . A training dataset was used during learning and generation , and a test dataset was used when evaluating the messages . 3.2 MODEL ARCHITECTURE AND OBVERTER TECHNIQUE . We herein propose an agent architecture for the multi-agent description game . An agent comprises a language module for reading and generating a message , and a visual module for processing an image and associating the image with the messages . Each agent can not exchange information other than messages and images . Figure 1 shows the configuration of the agent module . The visual module uses a variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) and the language module uses a gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) . The visual module has prior knowledge of the MNIST . The language module outputs a numerical value corresponding to the latent space by inputting an arbitrary message that has been converted into a one-hot vector . This structure enables the generation of messages corresponding to the image and reconstruction of a Dream ( Ha & Schmidhuber , 2018 ) that corresponds to the messages . Dream is defined as an output image reconstructed by a message . It is not an output image reconstructed from the VAE input 1It is noteworthy that “ image ” means raw pixel data that are provided to the agent , and “ label ” means “ numbers ( common concepts understood by humans ) ” written to the image . ( the output image is expressed as Reconstructed Image ) . The details of the model architecture are described in Appendix A . This module architecture differs from the preexisting architecture in terms of three characteristics : ( 1 ) The proposed architecture enables language emergence by agent interaction without using human-prepared supervised signals and rewards . ( 2 ) The concept corresponding to the generated message can be directly confirmed . ( 3 ) We can explore the structure and characteristics of messages organized by agent interaction . An agent learns and generates a language using the obverter technique ( Batali , 1998 ) . In the obverter technique , at the time of generation , assuming that a teacher has a mind similar to that of a student , the teacher generates a message that maximizes his own understanding . In the teacher agent , the model parameters are fixed . When a target image is input , the visual module outputs a latent space value z corresponding to the image . After initializing the GRU hidden layer , the language module evaluates the output of all possible symbols . During this evaluation , the symbol with the minimum value between latent space z and the output of the language module is selected . The GRU hidden vector derived by the selected symbol is used by the next GRU unit . This procedure is repeated until the minimal value between z and the output value of the language module is less than a predefined threshold or until the max message length is attained . In the student agent , the GRU model parameters are updated . The parameters are updated by backpropagating the mean square error loss between the output of the GRU unit and the latent space z . These algorithms are described in the Appendix B . The abovementioned procedure corresponds to self-supervised learning from the viewpoint of one agent . An agent set takes an emergent behavior that forms a conceptual pact without supervised signals .

Summary: This paper proposes a framework for emergent communication system using internal reflection, similar to obverter technique in Batali 1998 and Choi et al., 2018, while not requiring explicit supervision. Despite the nice idea, the dire quality of writing makes it extremely difficult if not impossible to understand the proposed model and the results. Also, some of the experimental results seem unnecessary. In Table 3., for example, (1) in §4.4.1 the authors merely list the rules without giving an overall conclusion. (2) I'm not certain what's the significant of the results in Table 3., as these are messages from one trained instance, and I'm sure a different training run will yield different messages.