A Formal Model of the Mechanismof Semantic Analysis in the Brain
BICA 20182018-08-23
Yuuji Ichisugi and Naoto Takahashi
National Institute of Advanced Industrial
Science and Technology (AIST), Japan

Our research goals• Long term goal : Human-like intelligenceby imitating the architecture ofthe whole brain• Short term goals:– Implement a cerebral cortex model

http://www.irasutoya.com/2015/05/ai.html• Our working hypothesis :The cerebral cortex is a kind of Bayesian network– Implement visual areas, language areas, motorareas, prefrontal areas, etc. using the cerebralcortex model

Models of cerebral cortex
based on Bayesian networks• Various functions, illusions, neural responses andanatomical structure of the visual cortex werereproduced by Bayesian network models.
– [Tai Sing Lee and Mumford 2003]– [Dileep George and Hawkins 2005]– [Rao 2005]– [Ichisugi 2007]– [Litvak and Ullman 2009]– [Chikkerur, Serre, Tan and Poggio 2010]– [Hosoya 2012]– ...

The cerebral cortex seems to be a huge
Bayesian network with layered structure like
Deep Neural Networks.

Motivation of this work
• If the cerebral cortex is a kind of Bayesiannetwork, we can build a system that
reproduces the behavior of the humanlanguage areas using a Bayesian network.
• This will become a new evidence for theBayesian network hypothesis of thecerebral cortex.

Previous work: Bayesian networkparser for context free grammar[Takahashi, Ichisugi 2017]

Φ1,2-4

S

Φ

2,3-4

Φ

1

NP

time

Naoto Takahashi and Yuuji Ichisugi,
Restricted Quasi Bayesian Networks as a Prototyping Tool for
Computational Models of Individual Cortical Areas, In Proc. ofAMBN 2017, Proceedings of Machine Learning Research,
Vol.73, pp.188--199, 2017.

VPΦ2VfliesPP3Plike4NP

an-arrowPrevious work: Bayesian networkparser for context free grammar[Takahashi, Ichisugi 2017]

Φ1,2-4SΦ

2,3-4

Naoto Takahashi and Yuuji Ichisugi,
Restricted Quasi Bayesian Networks as a Prototyping Tool for
Computational Models of Individual Cortical Areas, In Proc. ofAMBN 2017, Proceedings of Machine Learning Research,
Vol.73, pp.188--199, 2017.

VP

CFG is NOT
goodmodelofΦΦPP
natural language.
So we use CCG, which is betterVP
1 NP of natural234 NP
modellanguage.
timeflies

likean-arrow

Combinatory Categorial Grammar(CCG) [Steedman 2000]• CCG is one of the most successful
frameworks of grammar description intheoretical linguistics.
– CCG successfully explains many language
phenomena.

• We consider that CCG is a promising theoryof information processing of the brain.

An example of CCG parsing

Semantic analysis in CCG
・ The meaning of each word is represented as lambda term.
・ The meaning of the whole sentence is composed byfunction applications and function compositions.

The meaning of the sentence

Problems as a model of language areas:
Because CCG uses variable-length data structure,
it is hard to be implemented by neural networks in the brain.

Hierarchical address
representation
Address： (C,R,F)
Clause:C ∈{sconj, c1, c2}
Semantic Role:R ∈{action, agent, patient,...}
Feature:F ∈ {entity, color, size, ...}The meaning of a sentence is represented asa set of pairs of addresses and semanticrepresentations.
-> flat and fixed-size structureThis model does not use
variable-length data structuresuch as lambda terms.

"if small mice areChasedBy bigdogs black cats may eat mice"Modules of the proposed model
Lexicon

Words

ParserAddresses
Gates
Syntactic Categories

Semantic Representations

Modules are reciprocally (bidirectionally) connected
like Bayesian networks, and like cerebral cortex.

Parsing
Input words:"black cats eat mice"
Words

Lexicon

21

Parser22

4Addresses

3
Syntactic Categories

6Gates57
Semantic Representations

Output:Utterance(Speech)
Output words:"black cats eat mice"or " mice areEatenBy black cats "Words

Lexicon45

3

ParserAddresses4
Syntactic Categories

22Gates31
Semantic Representations

Input:

Correspondence tocortical areas

Language Areas
Broca’s area（BA 44,45）

https://en.wikipedia.org/wiki/Brodmann_area

Wernicke’s area (BA22)

A possible correspondence
between the modules in themodel and cortical areas
Lexicon
Words

BA44,45
Addresses,Categories
ParserParserAddresses
Gates

Syntactic Categories

BA41,42

Semantic
Representations

Words
LexiconBA22

Semantic Representations

Proposed model

Cortical areas

We can reproduce utterance of aphasia by disabling some modules.Broca’s area and Wernicke’s area
• Broca’s area : Grammar processing– Symptoms of Broca’s aphasia :Utterance consists of scattering words that do
not constitute sentences• Wernicke’s area : Association between
speech sounds and concepts– Symptoms of Wernicke’s aphasia :Utterance that is fluent but does not makesense because of mistakenly selected words.

Normal utterance
Input semantic representation:

BA44,45
Addresses,
Categories
ParserBA41,42

Semantic
Representations

Words
Lexicon

BA22・ The proposed model has been
implemented in the Prolog language.
・ Simplified English with many limitations.
・ A semantic representation is given.
・ The model infers all possible sentencesthat consist of four words?- M=[[[c1,agent,color],black],[[c1,agent,entity],cats], [[c1,action,-],eat],[[c1,patient,entity],mice]],
Ws=[W1,W2,W3,W4], maplist(lexicalItem,Ws, Cs, As, Ds), parse([], Cs, s(c1)),
maplist(bind(M), As, Ds), print(Ws), nl, fail.
Output:[black,cats,eat,mice][mice,areEatenBy,black,cats]Reproduction of Broca's aphasia
BA44,45
Addresses,
Categories
ParserBA41,42

Semantic
Representations

Words
Lexicon

BA22

Sentences are syntactically incorrect.
Selected words are semantically correct.
Resembles Broca's aphasia

Disable parser module?- M=[[[c1,agent,color],black],[[c1,agent,entity],cats], [[c1,action,-],eat],[[c1,patient,entity],mice]],
Ws=[W1,W2,W3,W4], maplist(lexicalItem, Ws,Cs, As, Ds), /* parse([], Cs, s(c1)),*/
maplist(bind(M), As, Ds), print(Ws), nl, fail.
Output:[black,black,black,black][black,black,black,cats][black,black,black,mice][black,black,black,eat][black,black,black,areEatenBy][black,black,cats,black][black,black,cats,cats][black,black,cats,mice]...

Reproduction ofWerniche's aphasia
Infer sentences without
giving concrete semantic
representation

BA44,45
Addresses,Categories
ParserBA41,42

Semantic
Representations

Words
Lexicon

BA22

Sentences are syntactically correct.
Selected words are semantically incorrect.
Resembles Werniche's aphasia?- /* M=[[[c1,agent,color],black],[[c1,agent,entity],cats], [[c1,action,-],eat],[[c1,patient,entity],mice]],*/
Ws=[W1,W2,W3,W4], maplist(lexicalItem,Ws, Cs, As, Ds), parse([], Cs, s(c1)),
maplist(bind(M), As, Ds), print(Ws), nl, fail.
Output:[white,dogs,eat,dogs][white,dogs,eat,cats][white,dogs,eat,mice][white,dogs,chase,dogs][white,dogs,chase,cats][white,dogs,chase,mice][white,dogs,areEatenBy,dogs][white,dogs,areEatenBy,cats]...

Conclusion• We proposed a model of the mechanism of
the semantic analysis that does not use
variable-length data structure such as
lambda terms.
• The model must be realized as a cortexlike Bayesian network in the future.
• Utterance of aphasia is reproduced.
• This research will connect computational
neuroscience and theoretical linguistics.