Abstract: Human thinking requires the brain to understand the meaning of language expression and to properly organize the thoughts flow using the language. However, current natural language processing models are primarily limited in the word probability estimation. Here, we proposed a Language guided imagination (LGI) network to incrementally learn the meaning and usage of numerous words and syntaxes, aiming to form a human-like machine thinking process. LGI contains three subsystems: (1) vision system that contains an encoder to disentangle the input or imagined scenarios into abstract population representations, and an imagination decoder to reconstruct imagined scenario from higher level representations; (2) Language system, that contains a binarizer to transfer symbol texts into binary vectors, an IPS (mimicking the human IntraParietal Sulcus, implemented by an LSTM) to extract the quantity information from the input texts, and a textizer to convert binary vectors into text symbols; (3) a PFC (mimicking the human PreFrontal Cortex, implemented by an LSTM) to combine inputs of both language and vision representations, and predict text symbols and manipulated images accordingly. LGI has incrementally learned eight different syntaxes (or tasks), with which a machine thinking loop has been formed and validated by the proper interaction between language and vision system. The paper provides a new architecture to let the machine learn, understand and use language in a human-like way that could ultimately enable a machine to construct fictitious 'mental' scenario and possess intelligence.

Introduction: Human thinking is regarded as ‘mental ideas flow guided by language to achieve a goal’. For instance, after seeing heavy rain, you may say internally ‘holding an umbrella could avoid getting wet’, and then you will take an umbrella before leaving. In the process, we know that the visual input of ‘water drop’ is called rain, and can imagine ‘holding an umbrella’ could keep off the rain, and can even experience the feeling of being wet. This continual thinking capacity distinguishes us from the machine, even though the latter can also recognize images, process language, and sense rain-drops. Continual thinking requires the capacity to generate mental imagination guided by language, and extract language representations from a real or imagined scenario. Modern natural language processing (NLP) techniques can handle question answering etc. tasks, such as answering that ‘Cao Cao’s nickname is Meng De’ based on the website knowledge [1]. However, the NLP network is just a probability model [2] and does not know whether Cao Cao is a man or cat. Indeed, it even does not understand what is a man. On the other hand, human being learns Cao Cao with his nickname via watching TV. When presented the question ‘what’s Cao Cao’s nickname?’, we can give the correct answer of ‘Meng De’ while imagining the figure of an actor in the brain. In this way, we say the machine network does not understand it, but the human does. Human beings possess such thinking capacity due to its cumulative learning capacity accompanying the neural developmental process. Initially, parent points to a real apple and teaches the baby ‘this is an apple’. After gradually assimilating the basic meanings of numerous nouns, children begin to learn some phrases and finally complicated syntaxes. Unlike the cumulative learning, most NLP techniques normally choose to learn by reading and predicting target words. After consuming billions of words in corpus materials [2], the NLP network can predict ‘Trump’ following ‘Donald’, but it is merely a probability machine. The human-like thinking system often requires specific neural substrates to support the corresponding functionalities. The most important brain area related to thinking is the prefrontal cortex (PFC), where the working memory takes place, including but not confined to, the maintenance and manipulation of particular information [3]. With the PFC, human beings can analyze and execute various tasks via ‘phonological loop’ and ‘visuospatial scratchpad’ etc. [4,5]. Inspired by the human-like brain organization, we build a ‘PFC’ network to combine language and vision streams to achieve tasks such as language controlled imagination, and imagination based thinking process. Our results show that the LGI network could incrementally learn eight syntaxes rapidly. Based on the LGI, we present the first language guided continual thinking process, which shows considerable promise for the human-like strong machine intelligence.

Question: In what was does an LSTM mimic the prefrontal cortex?
Answer: It combines language and vision streams similar to the human prefrontal cortex.