On the Word Boundaries of Emergent
Languages

Ryo Ueda
2023/06/29 @ SNL2023
The University of TokyoSelf-introductionSelf-introduction
nameafffield

Ryo Ueda (D1)
Miyao Lab, The University of Tokyo
Emergent Communication
my homepage

Interest
How statistical propertiesof languages emerge?
Newspre-print• Organized a session at JSAI2023• Released a pre-print
https://psyarxiv.com/rz5ng
1/20

Paper I introduce todayPaper I introduce today
Today, I would like to introduce this paper:

accepted to ICLR2023 (held in Kigali, Rwanda)
2/20

Pictures in Rwanda

3/20

Overview of our paper 1/2
Research Question
Harris’s articulation scheme (HAS)
holds for emergent languages
Emergent languages have meaningful words
Emergent Language

HAS

communication protocol
emerging among agents

statistical property onwords in natural language

Results Implication
HAS does :nothold for emergent languages::::
Emergent languages lack meaningful words

4/20

Overview of our paper 2/2
Research Question
Harris’s articulation scheme (HAS)
holds for emergent languages
Emergent languages have meaningful words

HAS is (anyway) applicable to emergent languages
to obtain segments
But does it really make senseThat’s the problem 5/20

Background:
Emergent Language

Background: Emergent Language
Emergent Language
Commpunication protocol emerging among agentsa.k.a Emergent Communication
Typical motivations in this area
1. Develop interactive AI[e.g., Lazaridou et al., 2018]
2. Simulate language evolution[e.g., Kirby, 2001]
3. :Clarify
diff between emergent/human languages:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::[e.g., Chaabouni et al., 2020]
6/20

Signaling Game
Typical Environmental Setting in this area
Signaling Game [Lewis, 1969]

Attribute-Value Setting#attributes natt ∈ N, #values nval ∈ N

7/20

Background: Harris’sArticulation Scheme (HAS)Next character uncertainty and word boundaries
Example: “natural”
n...
nat...
natura...
natural...

I have no idea on the next
The next may be “u” or “i”
The next must be “l”

Harris’s hypothesis [Harris, 1955]
Next character gets uncertain again
there tends to be a word boundary there
Harris’s Articulation Scheme (HAS) [Tanaka-Ishii, 2021]
Information-theoretic re-formulation
Uncertainty as Entropy
8/20

Conditional Entropy & Branching Entropy 1/2
A set of character sequences Σ∗ ,a sequence x1:n = x1 · · · xn ∈ Σ∗ ,a random variable X1:n = X1 · · · Xn over Σ∗
Conditional Entropy H(n) Branching Entropy h(x1:n )H (Xn+1 |X1:n )Its intuition

H(Xn+1 |X1:n = x1:n )Its intuition

next character uncertainty next character uncertainty
ina specific contextonaverage::::::::::::::::::::::::::::::::::::::::::
Conditional Entropy is the average of Branching EntropyP∵ H(n) = x1:n ∈Σ∗ p(x1:n )h(x1:n )9/20

Conditional Entropy & Branching Entropy 2/2
Conditional Entropy

Branching Entropy

Their behavior in natural languages
Conditional Entropy H(n) monotonically decreases
vsBranching Entropy h(x1:n ) repeatedly falls and rises
In natural languages, uncertainty decreases onaverage, but jitters in specific contexts

10/20

Harris’s Articulation Scheme (HAS)

Harris’s Articulation Scheme (HAS) [Tanaka-Ishii, 2021]
There tend to be word boundaries
at the increasing points of Branching Entropy hBranching Entropy h decreases on average(∵ Conditional Entropy H monotonically decreases),
but sometimes increases
There tend to be a word boundary

11/20

Problem Definition

Problem Definition 1
Re: Research Question
HAS holds for emergent languages?

12/20

Problem Definition 2
Re: Research Question
HAS holds for emergent languages?
Three questions to answer:
In emergent languages...
Q1 Conditional Entropy monotonically decreases
Q2 Branching Entropy jitters
Q3 Tentatively obtained segments are meaningful
Entropies behave like natural languages?
Obtained segments are really “words”?

Q1, Q2
Q3
13/20

Problem Definition 3
Tentatively obtained segments are meaningful?
Difficulty of Q3
No ground truth data of word segmentation
Emergent languages have words in the first place?
To mitigate this issue,
replace Q3 with the following questions:
Additional questions alternative to Q3
Q3-1 #attributes ∝ the number of boundaries
Q3-2 #values ∝ the number of distinct segments
Q3-3 segment-level compositionality >character-level compositionality

14/20

Problem Definition 4
Intuition of Q3-1
#attributes ∝ the number of boundaries

For Q3-2 & Q3-3, refer to our paper!
15/20

Experimental Results

Experimental Results
To Q1, Q2, & Q3
Q1 Conditional Entropy H monotonically decreases?
YES
Q2 Branching Entropy h jitters?

YES

Q3 Tentatively obtained segments are meaningful?
Probably NO
Alternatives to Q3
Q3-1 #attributes ∝ the number of boundaries?
Q3-2 #values ∝ the number of distinct segments?
Q3-3 segment-level compositionality >char-level compositionality?
NO for all
16/20

SummarySummary
Research Question
Harris’s articulation scheme (HAS)
holds for emergent languages
Emergent languages have meaningful words
Emergent Language

HAS

communication protocol
emerging among agents

statistical property onwords in natural language

Results Implication
HAS does :nothold for emergent languages::::
Emergent languages lack meaningful words

17/20

Additional Remarks

Entanglement and Disentanglement“Emergent languages have meaningful segments”
implies...
Disentanglement of Segments
1-to-1 correspondence between attributes/segments
Segments are disentangled
Entanglement of Characters
Segments can be obtained via characters’ entropy
Characters are entangled (statistically correlated)

18/20

References iRahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, andMarco Baroni. Compositionality and generalization in emergent languages. In Dan
Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedingsof the 58th Annual Meeting of the Association for Computational Linguistics, ACL
2020, Online, July 5-10, 2020, pages 4427–4442. Association for ComputationalLinguistics, 2020. doi: 10.18653/v1/2020.acl-main.407.
Zellig S. Harris. From phoneme to morpheme. Language, 31(2):190–222, 1955. ISSN
00978507, 15350665.
Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learningmodel of the emergence of regularity and irregularity. IEEE Trans. Evol. Comput., 5(2):102–110, 2001. doi: 10.1109/4235.918430.

19/20

References ii

Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergenceof linguistic communication from referential games with symbolic and pixel input.
In 6th International Conference on Learning Representations, ICLR 2018, Vancouver,BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net,2018.
David K. Lewis. Convention: A Philosophical Study. Wiley-Blackwell, 1969.
Kumiko Tanaka-Ishii. Articulation of Elements, pages 115–124. Springer International
Publishing, Cham, 2021.

20/20