Infinite SCAN:意味の数とその時間変化を同時に推定する
統計モデル
持橋大地
情報・システム研究機構 統計数理研究所
daichi@ism.ac.jp
日本英語学会 “深層学習時代の言語研究” シンポジウム
2023-11-4 (土)今日の話
l 自然言語処理の分野で、言語の分析のために開発した二つの統計モデル
l Infinite SCAN (EMNLP 2022):
各単語の意味の数と、その時間変化を同時に推定することができる統計モデル(Inoue+ 2022)
l Holographic CCG (ACL 2023):
潜在的なベクトル空間において、CCGの合成演算を
二つのベクトルの再帰的な合成として表し、文の木構造をベクトル空間上のグラフとして表す
研究(Yamaki+ 2023)

言語の時間的変化“Language is a dynamic system, constantly evolving andadapting to the needs of its users and their environment”(Aitchison, 2001)
l 音韻変化: 単語の発音の変化
l 文法的変化: 使われる統語構造の変化
l 意味的変化: 単語の意味の時間的変化– “cute”: 賢い (18世紀)à悪賢い (19世紀)
àかわいい (現在)
– 意味の変化した単語は多数あり、その変化は未知– どうやって自動的にこうした変化を推定すればよい？

単語の意味とは
l 単語の「意味」とは何か？

à 分布仮説 (Harris 1954)に従うとする(単語の意味は、前後の文脈に同時に現れる単語の分布に反映される)
l 通時コーパスから、目的の単語を含むスニペットをまず抽出するスニペットの抽出
目標単語: “coach”
通時コーパス
タイムスタンプ＋前後のスニペット
1853 The driver made room for the trunk on the top of the coach.
1900 The chair passed the coach, the horses proceeding at a walk.
1949 Tell him if I start coaching, it’ll be as a head coach at a top school.
1995 available for inspection, like the Coach bags offered on Canal Street
2003 Football coach and other top school officials have been interviewed.

単語の意味変化の既存研究(Hamilton+ 2016)

l 単語の前後の文脈(スニペット)から、単語の埋め込みベクトルを計算 (word2vecなど)
à時間的な変動を観察する
l 変化したことはわかるが、何がどう変化したかはわからない！

得られた“coach”のスニペット

z<latexit sha1_base64="THGOQFHHcnTOPxGombNG5gFOrzk=">AAAHY3ichZXNbtNAEMenLTSlBfoBBySEVBG14hRtAhKIUwtUompV5YO0qdqq8rrrYMWxLdtJm1h5AbiCOHACiQPiMbjwAhz6ABwQxyJx4cB47chJdlocJbs78/tPZnbXu9y1TD9g7HRsfOLS5cnM1JXpmavXrs/OzS9s+07L00VVdyzHq3HNF5Zpi2pgBpaouZ7QmtwSO7zxNPLvtIXnm479Iui44qCp1W3TMHUtQFOpeziXZTkmn0W1k086WUieojM/sQz7cAQO6NCCJgiwIcC+BRr4+NmDPDBw0XYAIdo87JnSL6AH06htISWQ0NDawN86jvYSq43jKKYv1Tr+i4VfD5WLsMS+s8/sjH1jX9hP9vfcWKGMEeXSwZbHWuEezr66VfnzX1UT2wBepqoLFBzpi2sKwIBHshYTa3OlJapSj+O3u+/OKo/LS+Ey+8h+YX0f2Cn7ihXa7d/6p5Iov5fRj1BjYNvGVocsznCIo36mHH2htPaSbFL+hCBPCK5DcB2C6xJcl+COCe6Y4AyCMwiuQXANglsluFWCWye4dYLbILgNgqsRXI3gdglul+AYwTGF48n6RvuQWleerGvsV9eTy13c9wvC38QdmxL7cqxSLuZoDnGxhSJ9hfRJMsotwPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcQWtdbmSg3RqVRWbQ1mlms2RnAZVW0n+OvosPFm2FGJtZEdxHK8pVJmgygr1TJ42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8vdzhdKD7MqT5CaagttwF+5h/IewAs9x/quYg4DX8AbeTv7IzGQWMjdjdHws0dyAoSdz5x/ns4GT</latexit>

1853
1900
1949
1995
2003

[driver, make, room, trunk]
à馬車[chair, pass, horse, proceed, walk]à馬車[tell, start, coach, head, top, school]
àスポーツ[available, inspection, bags, offer]
àブランド[football, top, school, official, interview] àスポーツ

l 各スニペットは、いずれかの意味(トピック) z に対応<latexit sha1_base64="THGOQFHHcnTOPxGombNG5gFOrzk=">AAAHY3ichZXNbtNAEMenLTSlBfoBBySEVBG14hRtAhKIUwtUompV5YO0qdqq8rrrYMWxLdtJm1h5AbiCOHACiQPiMbjwAhz6ABwQxyJx4cB47chJdlocJbs78/tPZnbXu9y1TD9g7HRsfOLS5cnM1JXpmavXrs/OzS9s+07L00VVdyzHq3HNF5Zpi2pgBpaouZ7QmtwSO7zxNPLvtIXnm479Iui44qCp1W3TMHUtQFOpeziXZTkmn0W1k086WUieojM/sQz7cAQO6NCCJgiwIcC+BRr4+NmDPDBw0XYAIdo87JnSL6AH06htISWQ0NDawN86jvYSq43jKKYv1Tr+i4VfD5WLsMS+s8/sjH1jX9hP9vfcWKGMEeXSwZbHWuEezr66VfnzX1UT2wBepqoLFBzpi2sKwIBHshYTa3OlJapSj+O3u+/OKo/LS+Ey+8h+YX0f2Cn7ihXa7d/6p5Iov5fRj1BjYNvGVocsznCIo36mHH2htPaSbFL+hCBPCK5DcB2C6xJcl+COCe6Y4AyCMwiuQXANglsluFWCWye4dYLbILgNgqsRXI3gdglul+AYwTGF48n6RvuQWleerGvsV9eTy13c9wvC38QdmxL7cqxSLuZoDnGxhSJ9hfRJMsotwPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcQWtdbmSg3RqVRWbQ1mlms2RnAZVW0n+OvosPFm2FGJtZEdxHK8pVJmgygr1TJ42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8vdzhdKD7MqT5CaagttwF+5h/IewAs9x/quYg4DX8AbeTv7IzGQWMjdjdHws0dyAoSdz5x/ns4GT</latexit>

していると考えられる
l 生成モデル：– ある確率分布 (z) に従って、z を選択– トピック z から、トピック毎の単語分布 (w|z)
に従って実際のスニペットが生成される<latexit sha1_base64="THGOQFHHcnTOPxGombNG5gFOrzk=">AAAHY3ichZXNbtNAEMenLTSlBfoBBySEVBG14hRtAhKIUwtUompV5YO0qdqq8rrrYMWxLdtJm1h5AbiCOHACiQPiMbjwAhz6ABwQxyJx4cB47chJdlocJbs78/tPZnbXu9y1TD9g7HRsfOLS5cnM1JXpmavXrs/OzS9s+07L00VVdyzHq3HNF5Zpi2pgBpaouZ7QmtwSO7zxNPLvtIXnm479Iui44qCp1W3TMHUtQFOpeziXZTkmn0W1k086WUieojM/sQz7cAQO6NCCJgiwIcC+BRr4+NmDPDBw0XYAIdo87JnSL6AH06htISWQ0NDawN86jvYSq43jKKYv1Tr+i4VfD5WLsMS+s8/sjH1jX9hP9vfcWKGMEeXSwZbHWuEezr66VfnzX1UT2wBepqoLFBzpi2sKwIBHshYTa3OlJapSj+O3u+/OKo/LS+Ey+8h+YX0f2Cn7ihXa7d/6p5Iov5fRj1BjYNvGVocsznCIo36mHH2htPaSbFL+hCBPCK5DcB2C6xJcl+COCe6Y4AyCMwiuQXANglsluFWCWye4dYLbILgNgqsRXI3gdglul+AYwTGF48n6RvuQWleerGvsV9eTy13c9wvC38QdmxL7cqxSLuZoDnGxhSJ9hfRJMsotwPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcQWtdbmSg3RqVRWbQ1mlms2RnAZVW0n+OvosPFm2FGJtZEdxHK8pVJmgygr1TJ42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8vdzhdKD7MqT5CaagttwF+5h/IewAs9x/quYg4DX8AbeTv7IzGQWMjdjdHws0dyAoSdz5x/ns4GT</latexit><latexit sha1_base64="/8e5N8aVjJSFb3ef7nsIHYEYGNs=">AAAHaXichZXPTxNBFMcfoBTxB1QvRC/EDQQvZFpMNJ5AJZFASGktLaGE7CyzZdPt7mZ3W2ib/gPePJnISRMPxj/Di/+AB/4EwxETLx58O7tNf8wDt2ln5r3P9/W9ndm33LOtIGTsfGx84sbNydTUrenbd+7em5lN398N3IZviKLh2q5f5nogbMsRxdAKbVH2fKHXuS1KvPYq8peawg8s13kbtjxxUNerjmVahh6iqVTxjq2l9pPDWY0tM3nNq5NMMtEguXJuemIRKnAELhjQgDoIcCDEuQ06BPjZhwww8NB2AB20+TizpF9AF6ZR20BKIKGjtYa/VVztJ1YH11HMQKoN/Bcbvz4q52GB/WRf2SX7wb6xX+zvlbE6MkaUSwtHHmuFdzjzbq7w57+qOo4hHPdV1yg40tfXFIIJz2UtFtbmSUtUpRHHb7Y/XBZe5Bc6i+wzu8D6PrFz9h0rdJq/jS87In8mox+hxsSxiaMBGt7hDq56mXL0daS1m2TT508J8pTgWgTXIrg2wbUJ7oTgTgjOJDiT4GoEVyO4NYJbI7gNgtsguE2C2yS4MsGVCW6P4PYIjhEcUzie7G90Dql95cm+xn51P7k8xT2/IPx1PLF9oiLXKuVhjtYQF1soMlDIgCSj3EKsf5Dt2VQ6uks9flDRt3cJlS2feY6WUd2gh1LmlIpzV1RcQGtV7uQg3beqiq2hrPqarZGcBlXbSf4G+mzsLNsKsT5yojiu1xUqT1B5hXotu40GWUmb2J+jvhfvZq/jx3nEXYn2ZKO4+P7JjL5t1MludjmzspzdeaqtvkzeRFPwCB7DEsZ/BqvwBu9/Ufbe9/ARziYvUunUXOphjI6PJZoHMHSltH9u+4O9</latexit><latexit sha1_base64="THGOQFHHcnTOPxGombNG5gFOrzk=">AAAHY3ichZXNbtNAEMenLTSlBfoBBySEVBG14hRtAhKIUwtUompV5YO0qdqq8rrrYMWxLdtJm1h5AbiCOHACiQPiMbjwAhz6ABwQxyJx4cB47chJdlocJbs78/tPZnbXu9y1TD9g7HRsfOLS5cnM1JXpmavXrs/OzS9s+07L00VVdyzHq3HNF5Zpi2pgBpaouZ7QmtwSO7zxNPLvtIXnm479Iui44qCp1W3TMHUtQFOpeziXZTkmn0W1k086WUieojM/sQz7cAQO6NCCJgiwIcC+BRr4+NmDPDBw0XYAIdo87JnSL6AH06htISWQ0NDawN86jvYSq43jKKYv1Tr+i4VfD5WLsMS+s8/sjH1jX9hP9vfcWKGMEeXSwZbHWuEezr66VfnzX1UT2wBepqoLFBzpi2sKwIBHshYTa3OlJapSj+O3u+/OKo/LS+Ey+8h+YX0f2Cn7ihXa7d/6p5Iov5fRj1BjYNvGVocsznCIo36mHH2htPaSbFL+hCBPCK5DcB2C6xJcl+COCe6Y4AyCMwiuQXANglsluFWCWye4dYLbILgNgqsRXI3gdglul+AYwTGF48n6RvuQWleerGvsV9eTy13c9wvC38QdmxL7cqxSLuZoDnGxhSJ9hfRJMsotwPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcQWtdbmSg3RqVRWbQ1mlms2RnAZVW0n+OvosPFm2FGJtZEdxHK8pVJmgygr1TJ42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8vdzhdKD7MqT5CaagttwF+5h/IewAs9x/quYg4DX8AbeTv7IzGQWMjdjdHws0dyAoSdz5x/ns4GT</latexit><latexit sha1_base64="ehAhocO08C98cafCmGH43IgGTQ8=">AAAHa3ichZXPTxNBFMcfoBTxByAXox6ITQlemmk10XgClUQCIaW10IYSsrvM1k33V3a3LW3tP+DVgwf1oIkH45/hxX/AA3+CMZ4w8eLBt7PbbNt54DbtzLz3+b6+NzM7o7qm4QeMnUxMTl24OJ2auTR7+crVa3PzC9d3fafpabysOabjVVTF56Zh83JgBCavuB5XLNXke2rjSejfa3HPNxz7edBx+YGl1G1DNzQlQFO15vrGSvtl9+7hfJplmXiW5E4u7qQhfgrOwtQy1OAIHNCgCRZwsCHAvgkK+PjZhxwwcNF2AD20edgzhJ9DH2ZR20SKI6GgtYG/dRztx1Ybx2FMX6g1/BcTvx4qlyDDvrPP7JR9Y1/YD/b3zFg9ESPMpYOtGmm5ezj36kbpz39VFrYBvEhU5yhUpM+vKQAdHopaDKzNFZawSi2K3+q+OS09KmZ6y+wj+4n1fWAn7CtWaLd+a592ePGtiH6EGh3bFrYapHGGezgaZKqiryes/TibhD8myGOC6xBch+C6BNcluDbBtQlOJzid4BoE1yC4NYJbI7gNgtsguE2C2yS4CsFVCK5KcFWCYwTHJE6N1zfch9S6qvG6Rn55PVWxiwd+Tvgt3LEJURNjmXIxR2OEiywU6UukT5JhbgHWP8wObDIdztKAH1Yk9j6hMsU7r6JlXDfsoZQFqeLCGRWX0FoXKzlMJ1ZZsTWSVaLZGstpWLUd56+hz8STZVsi1sd2lIrjdYkqElRRop6K0yYNeUHreD6H5160moMTP8ojOpVoTz6Mi/dPbvy2kTu7+WzuXja/cz+9+ji+iWbgFtyBFYz/AFbhGc5/GXOw4DW8g/fTv1KLqZup2xE6ORFrFmHkSWX+AbtnhM8=</latexit>

トピックの教師なし学習l 各スニペットの意味(トピック)は書かれていない！

à 実は、自動的に学習できる
l アルゴリズム：– 各スニペットを、確率的にK個のトピック z のどれかに割り当てる ( (z) がわかる)– 各トピック z に割り当てられたスニペットの文章全体から、 (w|z) を更新– 以上を繰り返すl スニペットのクラスタリングに相当している<latexit sha1_base64="THGOQFHHcnTOPxGombNG5gFOrzk=">AAAHY3ichZXNbtNAEMenLTSlBfoBBySEVBG14hRtAhKIUwtUompV5YO0qdqq8rrrYMWxLdtJm1h5AbiCOHACiQPiMbjwAhz6ABwQxyJx4cB47chJdlocJbs78/tPZnbXu9y1TD9g7HRsfOLS5cnM1JXpmavXrs/OzS9s+07L00VVdyzHq3HNF5Zpi2pgBpaouZ7QmtwSO7zxNPLvtIXnm479Iui44qCp1W3TMHUtQFOpeziXZTkmn0W1k086WUieojM/sQz7cAQO6NCCJgiwIcC+BRr4+NmDPDBw0XYAIdo87JnSL6AH06htISWQ0NDawN86jvYSq43jKKYv1Tr+i4VfD5WLsMS+s8/sjH1jX9hP9vfcWKGMEeXSwZbHWuEezr66VfnzX1UT2wBepqoLFBzpi2sKwIBHshYTa3OlJapSj+O3u+/OKo/LS+Ey+8h+YX0f2Cn7ihXa7d/6p5Iov5fRj1BjYNvGVocsznCIo36mHH2htPaSbFL+hCBPCK5DcB2C6xJcl+COCe6Y4AyCMwiuQXANglsluFWCWye4dYLbILgNgqsRXI3gdglul+AYwTGF48n6RvuQWleerGvsV9eTy13c9wvC38QdmxL7cqxSLuZoDnGxhSJ9hfRJMsotwPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcQWtdbmSg3RqVRWbQ1mlms2RnAZVW0n+OvosPFm2FGJtZEdxHK8pVJmgygr1TJ42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8vdzhdKD7MqT5CaagttwF+5h/IewAs9x/quYg4DX8AbeTv7IzGQWMjdjdHws0dyAoSdz5x/ns4GT</latexit><latexit sha1_base64="/8e5N8aVjJSFb3ef7nsIHYEYGNs=">AAAHaXichZXPTxNBFMcfoBTxB1QvRC/EDQQvZFpMNJ5AJZFASGktLaGE7CyzZdPt7mZ3W2ib/gPePJnISRMPxj/Di/+AB/4EwxETLx58O7tNf8wDt2ln5r3P9/W9ndm33LOtIGTsfGx84sbNydTUrenbd+7em5lN398N3IZviKLh2q5f5nogbMsRxdAKbVH2fKHXuS1KvPYq8peawg8s13kbtjxxUNerjmVahh6iqVTxjq2l9pPDWY0tM3nNq5NMMtEguXJuemIRKnAELhjQgDoIcCDEuQ06BPjZhwww8NB2AB20+TizpF9AF6ZR20BKIKGjtYa/VVztJ1YH11HMQKoN/Bcbvz4q52GB/WRf2SX7wb6xX+zvlbE6MkaUSwtHHmuFdzjzbq7w57+qOo4hHPdV1yg40tfXFIIJz2UtFtbmSUtUpRHHb7Y/XBZe5Bc6i+wzu8D6PrFz9h0rdJq/jS87In8mox+hxsSxiaMBGt7hDq56mXL0daS1m2TT508J8pTgWgTXIrg2wbUJ7oTgTgjOJDiT4GoEVyO4NYJbI7gNgtsguE2C2yS4MsGVCW6P4PYIjhEcUzie7G90Dql95cm+xn51P7k8xT2/IPx1PLF9oiLXKuVhjtYQF1soMlDIgCSj3EKsf5Dt2VQ6uks9flDRt3cJlS2feY6WUd2gh1LmlIpzV1RcQGtV7uQg3beqiq2hrPqarZGcBlXbSf4G+mzsLNsKsT5yojiu1xUqT1B5hXotu40GWUmb2J+jvhfvZq/jx3nEXYn2ZKO4+P7JjL5t1MludjmzspzdeaqtvkzeRFPwCB7DEsZ/BqvwBu9/Ufbe9/ARziYvUunUXOphjI6PJZoHMHSltH9u+4O9</latexit><latexit sha1_base64="THGOQFHHcnTOPxGombNG5gFOrzk=">AAAHY3ichZXNbtNAEMenLTSlBfoBBySEVBG14hRtAhKIUwtUompV5YO0qdqq8rrrYMWxLdtJm1h5AbiCOHACiQPiMbjwAhz6ABwQxyJx4cB47chJdlocJbs78/tPZnbXu9y1TD9g7HRsfOLS5cnM1JXpmavXrs/OzS9s+07L00VVdyzHq3HNF5Zpi2pgBpaouZ7QmtwSO7zxNPLvtIXnm479Iui44qCp1W3TMHUtQFOpeziXZTkmn0W1k086WUieojM/sQz7cAQO6NCCJgiwIcC+BRr4+NmDPDBw0XYAIdo87JnSL6AH06htISWQ0NDawN86jvYSq43jKKYv1Tr+i4VfD5WLsMS+s8/sjH1jX9hP9vfcWKGMEeXSwZbHWuEezr66VfnzX1UT2wBepqoLFBzpi2sKwIBHshYTa3OlJapSj+O3u+/OKo/LS+Ey+8h+YX0f2Cn7ihXa7d/6p5Iov5fRj1BjYNvGVocsznCIo36mHH2htPaSbFL+hCBPCK5DcB2C6xJcl+COCe6Y4AyCMwiuQXANglsluFWCWye4dYLbILgNgqsRXI3gdglul+AYwTGF48n6RvuQWleerGvsV9eTy13c9wvC38QdmxL7cqxSLuZoDnGxhSJ9hfRJMsotwPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcQWtdbmSg3RqVRWbQ1mlms2RnAZVW0n+OvosPFm2FGJtZEdxHK8pVJmgygr1TJ42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8vdzhdKD7MqT5CaagttwF+5h/IewAs9x/quYg4DX8AbeTv7IzGQWMjdjdHws0dyAoSdz5x/ns4GT</latexit><latexit sha1_base64="ehAhocO08C98cafCmGH43IgGTQ8=">AAAHa3ichZXPTxNBFMcfoBTxByAXox6ITQlemmk10XgClUQCIaW10IYSsrvM1k33V3a3LW3tP+DVgwf1oIkH45/hxX/AA3+CMZ4w8eLBt7PbbNt54DbtzLz3+b6+NzM7o7qm4QeMnUxMTl24OJ2auTR7+crVa3PzC9d3fafpabysOabjVVTF56Zh83JgBCavuB5XLNXke2rjSejfa3HPNxz7edBx+YGl1G1DNzQlQFO15vrGSvtl9+7hfJplmXiW5E4u7qQhfgrOwtQy1OAIHNCgCRZwsCHAvgkK+PjZhxwwcNF2AD20edgzhJ9DH2ZR20SKI6GgtYG/dRztx1Ybx2FMX6g1/BcTvx4qlyDDvrPP7JR9Y1/YD/b3zFg9ESPMpYOtGmm5ezj36kbpz39VFrYBvEhU5yhUpM+vKQAdHopaDKzNFZawSi2K3+q+OS09KmZ6y+wj+4n1fWAn7CtWaLd+a592ePGtiH6EGh3bFrYapHGGezgaZKqiryes/TibhD8myGOC6xBch+C6BNcluDbBtQlOJzid4BoE1yC4NYJbI7gNgtsguE2C2yS4CsFVCK5KcFWCYwTHJE6N1zfch9S6qvG6Rn55PVWxiwd+Tvgt3LEJURNjmXIxR2OEiywU6UukT5JhbgHWP8wObDIdztKAH1Yk9j6hMsU7r6JlXDfsoZQFqeLCGRWX0FoXKzlMJ1ZZsTWSVaLZGstpWLUd56+hz8STZVsi1sd2lIrjdYkqElRRop6K0yYNeUHreD6H5160moMTP8ojOpVoTz6Mi/dPbvy2kTu7+WzuXja/cz+9+ji+iWbgFtyBFYz/AFbhGc5/GXOw4DW8g/fTv1KLqZup2xE6ORFrFmHkSWX+AbtnhM8=</latexit>ただし...
l 単語の意味は変化するà 時刻tでの意味の割合 t (z) は、少しずつ変化する
時刻tでのトピック-単語確率 t (w|z) も変化する
l 観測値がない年もあるので、うまく平滑化する必要がある
l 意味の総数Kは、事前にはわからない
à 本研究で、ディリクレ過程によって自動推定<latexit sha1_base64="6dCIYajfTbf7s26Bl0rqfJjRBTY=">AAAHa3ichZXPTxNBFMcfoBTxByAXox6ITQlemmk10XgClUQCIaW10IaSZmeZLZvur+xuC23Tf8CrBw/qQRMPxj/Di/+AB/4EYzxh4sWDb2e32bb7wG3amXnv8319b2Z2hjuG7vmMnU5MTl26PJ2auTJ79dr1G3PzCzd3PbvlqqKs2obtVrjiCUO3RNnXfUNUHFcoJjfEHm8+C/x7beF6um299DuOODCVhqVruqr4aKrWnCO97q9079fn0yzL5LOU7OSiThqip2AvTC1DDQ7BBhVaYIIAC3zsG6CAh599yAEDB20H0EObiz1d+gX0YRa1LaQEEgpam/jbwNF+ZLVwHMT0pFrFfzHw66JyCTLsO/vMztg39oX9YH/PjdWTMYJcOtjyUCuc+tyrW6U//1WZ2PpwFKsuUHCkL67JBw0ey1p0rM2RlqBKNYzf7r45Kz0pZnrL7CP7ifV9YKfsK1ZotX+rn3ZE8a2MfogaDds2tiqkcYZ7OBpkytHXk9Z+lE3MnxDkCcF1CK5DcF2C6xLcMcEdE5xGcBrBNQmuSXBrBLdGcBsEt0FwmwS3SXAVgqsQXJXgqgTHCI4lOB6tb7APqXXl0bqG/uR6crmLB35B+E3csTFRk+Mk5WCO+ggXWijSS5AeSQa5+Vj/MDuwJelglgb8sCK29wmVId95jpZx3bCHUhYSFRfOqbiE1oZcyWE6tiYVWyNZxZqtsZyGVdtR/ir6DDxZthPE+tiO4jheT1BFgiomqOfytElDXtIans/BuReu5uDED/MITyXakw/i4v2TG79tkp3dfDb3IJvfeZhefRrdRDNwB+7BCsZ/BKvwAue/jDmY8BrewfvpX6nF1O3U3RCdnIg0izDypDL/AI5shKQ=</latexit><latexit sha1_base64="dmNHiKqdeTbGmw87dEsMNC7YrCM=">AAAHbXichZXPaxNBFMdfW21q/dFWEQRFiqG1XsIkCoqnVi1YWkqamDYlKWFnOxuXbDbL7iZpEvMPeBcPgqLgQfwzvPgPeOifIB48VPDiwbezGzbNvLYbkpl57/N9eW9mdoY7lun5jB2OjU+cOz+ZmLowffHS5Sszs3NXt71G09VFQW9YDbfINU9Ypi0Kvulboui4QqtzS+zw2tPAv9MSrmc27Bd+xxF7da1qm4apaz6aSmXHMyv+UvtV915lNslSTD7zaicddZIQPdnG3MQilGEfGqBDE+ogwAYf+xZo4OGnBGlg4KBtD3poc7FnSr+APkyjtomUQEJDaw1/qzgqRVYbx0FMT6p1/BcLvy4q52GB/WBf2BH7zr6yn+zfibF6MkaQSwdbHmqFU5l5fSP/90xVHVsfXsaqUxQc6dNr8sGAR7IWE2tzpCWoUg/jt7pvj/KPcwu9RfaJ/cL6PrJD9g0rtFt/9M9bIvdORt9HjYFtC1sdkjjDPRwNMuXo60lrP8om5g8I8oDgOgTXIbguwXUJrk1wbYIzCM4guBrB1QhuheBWCG6N4NYIbp3g1gmuSHBFgtsluF2CYwTHFI5H6xvsQ2pdebSuoV9dTy538cAvCH8dd2xMlOVYpRzM0TzGhRaK9BTSI8kgNx/rH2YHNpUOZmnADytie59QWfKd52gZ1Q17KGVWqTh7QsV5tFblSg7TsVVVbBzLKtZsjOQ0rNqM8tfRZ+HJsqkQqyM7iuN4VaFyBJVTqGfytElCRtIGns/BuReu5uDED/MITyXakwni4v2THr1t1M52JpW+n8psPUguP4luoim4CXdgCeM/hGV4jvNfkPfFG3gPHyZ/J64nbiVuh+j4WKS5BseexN3/3JyFtg==</latexit>先行研究: SCAN (Frermann+ TACL 2016)l SCAN: Dynamic Bayesian Model of Sense ChANge– スニペット時系列の動的トピックモデル
各スニペット[chair, pass,horse,
proceed,walk]意味分布の時間発展
l 意味分布<latexit sha1_base64="8DRQuvlzL76adimdRO3GE91/An8=">AAAHaHichZXPTxNBFMcfoBTxB0UParwQGoinZlpNNJ5AJZFASGltKaGk2Vlmy9rt7mZ3W2ib/gOevBn1pIkH45/hxX/AA3+CesTEiwffzm7TH/OAbdqZee/zfX1vZnaGu5bpB4ydTExOXbo8nZi5Mnv12vUbc8n5myXfaXq6KOqO5XhlrvnCMm1RDMzAEmXXE1qDW2KH15+F/p2W8HzTsV8GbVfsN7SabRqmrgVoKlXcQ7MaVJMplmbyWVA7mbiTgvjJOfNTy1CBA3BAhyY0QIANAfYt0MDHzx5kgIGLtn3oos3Dnin9Anowi9omUgIJDa11/K3haC+22jgOY/pSreO/WPj1ULkAS+wH+8JO2Xf2lf1k/86M1ZUxwlza2PJIK9zq3Os7hb8XqhrYBnA4UJ2j4EifX1MABjyWtZhYmystYZV6FL/VeXtaeJJf6i6zT+w31veRnbBvWKHd+qN/3hb5DzL6AWoMbFvY6pDCGe7iqJ8pR19XWntxNgP+mCCPCa5NcG2C6xBch+COCO6I4AyCMwiuTnB1glsluFWCWye4dYLbILgNgisTXJngdglul+AYwTGF4/H6hvuQWlcer2vkV9eTy13c9wvC38AdOyAqcqxSLuZojnCRhSJ9hfRJMswtwPqH2b5NpcNZ6vPDioG9R6gs+c5ztIzrhj2UMqdUnDuj4gJaa3Ilh+mBVVVsjmQ10GyO5TSs2orz19Fn4cmypRBrYzuK43hNofIElVeo5/K0SUFW0gaez+G5F61m/8SP8ohOJdqTDePi/ZMZv23UTimbzjxIZ7cfplaexjfRDNyDRbiP8R/BCrzA+S9iDq/gDbyD99O/EsnE7cTdCJ2ciDW3YORJLP4HeoKDuw==</latexit>t と意味-単語分布<latexit sha1_base64="puIUV1rBidnvw7jvBvzYEtauJj8=">AAAHaHichZXNbtNAEMenLTSlfDSFAyAuVaNWnKJNQAJxaoFKVK2qNCFpqqaKvO46mDi2ZbtpkygvwIkbAk4gcUA8BhdegEMfATgWiQsHxmtHTrLT4ijZ3ZnffzKzu97lrmX6AWMnE5NTFy5Op2YuzV6+cvXaXHr+esV3Dj1dlHXHcrwq13xhmbYoB2ZgiarrCa3FLbHDm09C/05beL7p2M+Djiv2W1rDNg1T1wI0VWqub9aDejrDskw+C2onF3cyED8FZ35qGWpwAA7ocAgtEGBDgH0LNPDxswc5YOCibR96aPOwZ0q/gD7MovYQKYGEhtYm/jZwtBdbbRyHMX2p1vFfLPx6qFyAJfadfWan7Bv7wn6wv2fG6skYYS4dbHmkFW597tWt0p//qlrYBvAiUZ2j4EifX1MABjyUtZhYmystYZV6FL/dfXNaelRc6i2zj+wX1veBnbCvWKHd/q1/2hbF9zL6AWoMbNvY6pDBGe7haJApR19PWvtxNgl/TJDHBNchuA7BdQmuS3BHBHdEcAbBGQTXJLgmwa0S3CrBrRPcOsFtENwGwVUJrkpwuwS3S3CM4JjC8Xh9w31IrSuP1zXyq+vJ5S4e+AXhb+GOTYiaHKuUizmaI1xkoUhfIX2SDHMLsP5hdmBT6XCWBvywIrH3CZUl33mOlnHdsIdSFpSKC2dUXEJrQ67kMJ1YVcXmSFaJZnMsp2HVVpy/jj4LT5YthVgb21Ecx2sKVSSookI9ladNBvKSNvB8Ds+9aDUHJ36UR3Qq0Z58GBfvn9z4baN2Kvls7l42v30/s/I4volm4A4swl2M/wBW4BnOfxlzeAmv4S28m/6ZSqdupm5H6ORErLkBI09q8R/HmIPG</latexit>t は両方とも、時間に従って変化する– ガウス分布のマルコフ確率場

意味の数の推定
l SCANでは、“意味の総数” K は単語ごとにあらかじめ指定する必要がある– 一般には未知！
l 時間によって変わる意味の総数を、どうやって推定するか？
– 使えるデータは、下のようなスニペット集合だけ
1853
1900
1949
2003

[driver, make, room, trunk][chair, pass, horse, proceed, walk][tell, start, coach, head, top, school][football, top, school, official, interview]

ロジスティック棒折り過程 (Ren+ 2011)1qk =1 + e xk
<latexit sha1_base64="trRnjCparmOUD2uTDSEVhdU7l7s=">AAAHfnichZXPaxNBFMdfWzW1/miqF9FLMDQI0jgbBYsgtGrB0lLSxLQpTQ2729m4ZJOsu5s0ybLg2X/AgycFBfGg/4MX/wEP/RPEYwUvHnw7uyFp5rVuSGbmvc/35b2Z2RnNtkzXY+xwYnLqzNlzienzMxcuXro8m5y7suW22o7OS3rLajllTXW5ZTZ5yTM9i5dth6sNzeLbWv1x6N/ucMc1W81nXs/mew211jQNU1c9NFWT119W66mHqYrhqLqvBL5ymz/3F7rVehBUk2mWZeJJyR0l7qQhfvKtuakMVGAfWqBDGxrAoQke9i1QwcXPLijAwEbbHvhoc7BnCj+HAGZQ20aKI6GitY6/NRztxtYmjsOYrlDr+C8Wfh1UpmCe/WCf2BH7zj6zn+zvibF8ESPMpYetFmm5XZ19fa3457+qBrYevBiqTlFoSJ9ekwcGLIpaTKzNFpawSj2K3+m/OSo+KMz7Gfae/cL63rFD9g0rbHZ+6x82eeGtiL6PGgPbDrY6pHGGfRwNMtXQ5wtrEGcz5LsE2SW4HsH1CK5PcH2COyC4A4IzCM4guDrB1QlumeCWCW6V4FYJbo3g1giuTHBlgtshuB2CYwTHJE6L1zfch9S6avG6Rn55PTWxiwd+TvgbuGOHREWMZcrGHM1jXGShSFciXZIMc/Ow/lF2YJPpcJYG/KhiaA8IlSXeeQ0t47pRD6XMSxXnT6i4iNaaWMlRemiVFevHshpq1sdyGlVtxPnr6LPwZNmQiJWxHaXheEWiCgRVkKgn4rRJQ07QBp7P4bkXrebgxI/yiE4l2pML4+L9o4zfNnJnK5dV7mZzm/fSS4/im2gabsBNuIXx78MSPMX5L2EOr+AjfIGvCUhkEguJOxE6ORFrrsKxJ7H4D35cipw=</latexit>(シグモイド関数で確率に変換)

breakqk<latexit sha1_base64="h5XFGSiHoNnCvUgwf1mV6ofsQ3A=">AAAHZXichZVBTxNBFMcfoBRRBNQYEw8SG4inZlpJMJ5AJZFASGktlABpdpbZuul2d93dFtqmX8DEqxw8aeLB+DG8+AU88AmM8YiJFw++nd1m284Dtmln5r3f//W9mdkZ7lqmHzB2OjI6duXqeGri2uT1G1M3p2dmb237TsPTRUl3LMcrc80XlmmLUmAGlii7ntDq3BI7vPY89O80heebjv0qaLnioK5VbdMwdS1AU/FNpVaZSbMMk8+c2snGnTTET96ZHVuAfTgEB3RoQB0E2BBg3wINfPzsQRYYuGg7gA7aPOyZ0i+gC5OobSAlkNDQWsPfKo72YquN4zCmL9U6/ouFXw+VczDPfrAv7Ix9Z1/ZL/bv3FgdGSPMpYUtj7TCrUy/vVf8e6mqjm0ArxPVBQqO9MU1BWDAE1mLibW50hJWqUfxm+2Ts+LTwnxngX1iv7G+j+yUfcMK7eYf/fOWKHyQ0Q9RY2DbxFaHNM5wB0e9TDn6OtLajbNJ+GOCPCa4FsG1CK5NcG2COyK4I4IzCM4guBrB1QhuheBWCG6N4NYIbp3g1gmuTHBlgtsluF2CYwTHFI7H6xvuQ2pdebyukV9dTy53cc8vCH8dd2xC7MuxSrmYoznARRaK9BXSJ8kwtwDr72d7NpUOZ6nH9ysSe5dQWfKd52gZ1vV7KGVeqTh/TsVFtFblSvbTiVVVbAxklWg2hnLqV23G+evos/Bk2VSI1aEdxXG8qlAFgioo1At52qQhJ2kDz+fw3ItWs3fiR3lEpxLtyYVx8f7JDt82amc7l8k+zuS2FtPLz+KbaALuw0N4hPGXYBle4vyXMIcqvIP3cDL+MzWVupO6G6GjI7HmNgw8qQf/AYRNgmg=</latexit>

l 時間変化する意味の割合(の対数) xk から、各時刻ごと<latexit sha1_base64="YITVBHH+BDVKAyw4G+yWWI+Rt9I=">AAAHZXichZXPTxNBFMcfoBQRBNQYEw8SGoinZlpNNJ5AJZFASGktlABpdpbZuul2d7O7Lf2R/gMmXuXgSRMPxj/Di/+AB/4CYzxi4sWDb2e32bbzwG3amXnv8319b2Z2hruW6QeMnY2NT1y5OpmaujZ9fWb2xtz8ws1d32l4uijpjuV4Za75wjJtUQrMwBJl1xNanVtij9eeh/69pvB807FfBW1XHNW1qm0apq4FaCq2KrXKfJplmHwW1U427qQhfvLOwsQKHMIxOKBDA+ogwIYA+xZo4OPnALLAwEXbEXTR5mHPlH4BPZhGbQMpgYSG1hr+VnF0EFttHIcxfanW8V8s/HqoXIRl9p19ZufsG/vCfrK/F8bqyhhhLm1seaQVbmXuzd3in/+q6tgG8DpRXaLgSF9eUwAGPJG1mFibKy1hlXoUv9k5PS8+LSx3V9hH9gvr+8DO2Fes0G7+1j/tiMJ7Gf0YNQa2TWx1SOMMd3HUz5SjryutvTibhG8RZIvg2gTXJrgOwXUI7oTgTgjOIDiD4GoEVyO4NYJbI7gNgtsguE2C2yS4MsGVCW6f4PYJjhEcUzger2+4D6l15fG6Rn51PbncxX2/IPx13LEJcSjHKuVijuYQF1ko0ldInyTD3AKsf5Dt21Q6nKU+P6hI7D1CZcl3nqNlVDfooZR5peL8BRUX0VqVKzlIJ1ZVsTWUVaLZGslpULUd56+jz8KTZVsh1kd2FMfxukIVCKqgUC/kaZOGnKQNPJ/Dcy9azf6JH+URnUq0JxfGxfsnO3rbqJ3dXCb7MJPbeZRefRbfRFNwD5bgAcZ/DKvwEue/hDlU4S28g9PJH6nZ1O3UnQgdH4s1t2DoSd3/B7VUgm8=</latexit>に無限次元の分布を仮定して推定

推論l All the inference is done by Gibbs sampling.
l Because Gaussian is not conjugate to multinomial,

we employ Pólya-Gamma auxiliary variables ! tosample ↵t (and transform into t ):<latexit sha1_base64="hPsSDiVv82QOA65/P/BoWmU0/m8=">AAAHaHichZXPTxNBFMcfoBTxB0UParwQGoinZlpNNJ5AJZFASGltKaGE7Cyzde3+yu620Db9Bzx5M+pJEw/GP8OL/4AH/gT1iIkXD76d3aY/5oHbtDPz3uf7+t7M7Az3LDMIGTuZmJy6cHE6NXNp9vKVq9fm0vPXK4Hb9HVR1l3L9atcC4RlOqIcmqElqp4vNJtbYoc3nkT+nZbwA9N1nodtT+zbWt0xDVPXQjRVaq4t6tpBOsOyTD4LaieXdDKQPAV3fmoZanAILujQBBsEOBBi3wINAvzsQQ4YeGjbhy7afOyZ0i+gB7OobSIlkNDQ2sDfOo72EquD4yhmINU6/ouFXx+VC7DEvrPP7JR9Y1/YD/b3zFhdGSPKpY0tj7XCO5h7dav0578qG9sQXgxU5yg40ufXFIIBD2UtJtbmSUtUpR7Hb3XenJYeFZe6y+wj+4X1fWAn7CtW6LR+65+2RfG9jH6IGgPbFrY6ZHCGuzjqZ8rR15XWXpLNgD8myGOCaxNcm+A6BNchuCOCOyI4g+AMgmsQXIPgVgluleDWCW6d4DYIboPgqgRXJbhdgtslOEZwTOF4sr7RPqTWlSfrGvvV9eRyF/f9gvDbuGMHRE2OVcrDHM0RLrZQZKCQAUlGuYVY/zDbt6l0NEt9flgxsPcIlSXfeY6Wcd2wh1IWlIoLZ1RcQmtdruQwPbCqis2RrAaazbGchlVbSf46+iw8WbYUYm1sR3EcrylUkaCKCvVUnjYZyEvawPM5Ovfi1eyf+HEe8alEe/JRXLx/cuO3jdqp5LO5e9n89v3MyuPkJpqBO7AIdzH+A1iBZzj/ZczhJbyGt/Bu+mcqnbqZuh2jkxOJ5gaMPKnFfy2Yg7A=</latexit><latexit sha1_base64="af48wrsxJxlqaPsuZHUDwtt++x4=">AAAHanichZXPTxNBFMcfoBTxBz+8aLgQK8RTMy0mGk+gkkggpLQWSihpdpbZsul2d7O7LbRN/wGPXjzgRRMPxj/Di/+AB/4EozdMvHjw7ew223YeuE07M+99vq/vzczOcNcy/YCx87HxiWvXJ1NTN6Zv3rp9Z2Z2bn7Xd5qeLkq6YzlemWu+sExblAIzsETZ9YTW4JbY4/UXoX+vJTzfdOzXQdsVhw2tZpuGqWsBmsoVzXKPtWpQnU2zDJPPotrJxp00xE/emZtYhgocgQM6NKEBAmwIsG+BBj5+DiALDFy0HUIXbR72TOkX0INp1DaREkhoaK3jbw1HB7HVxnEY05dqHf/Fwq+HykVYYt/ZZ3bBvrEv7Af7e2msrowR5tLGlkda4VZn3twr/vmvqoFtAMeJ6goFR/rqmgIw4KmsxcTaXGkJq9Sj+K3Ou4vis8JSd5l9ZD+xvg/snH3FCu3Wb/3TjiicyehHqDGwbWGrQxpnuIujfqYcfV1p7cXZJPwpQZ4SXJvg2gTXIbgOwZ0Q3AnBGQRnEFyd4OoEt0ZwawS3QXAbBLdJcJsEVya4MsHtE9w+wTGCYwrH4/UN9yG1rjxe18ivrieXu7jvF4S/gTs2ISpyrFIu5mgOcZGFIn2F9EkyzC3A+gfZvk2lw1nq84OKxN4jVJZ85zlaRnWDHkqZVyrOX1JxEa01uZKDdGJVFVtDWSWarZGcBlXbcf46+iw8WbYVYn1kR3EcrytUgaAKCvVSnjZpyEnawPM5PPei1eyf+FEe0alEe3JhXLx/sqO3jdrZzWWyK5nczuP06vP4JpqCBXgAjzD+E1iFVzj/JXnLvIUzeD/5KzWfup9aiNDxsVhzF4ae1MN/NvWElA==</latexit><latexit sha1_base64="8DRQuvlzL76adimdRO3GE91/An8=">AAAHaHichZXPTxNBFMcfoBTxB0UParwQGoinZlpNNJ5AJZFASGltKaGk2Vlmy9rt7mZ3W2ib/gOevBn1pIkH45/hxX/AA3+CesTEiwffzm7TH/OAbdqZee/zfX1vZnaGu5bpB4ydTExOXbo8nZi5Mnv12vUbc8n5myXfaXq6KOqO5XhlrvnCMm1RDMzAEmXXE1qDW2KH15+F/p2W8HzTsV8GbVfsN7SabRqmrgVoKlXcQ7MaVJMplmbyWVA7mbiTgvjJOfNTy1CBA3BAhyY0QIANAfYt0MDHzx5kgIGLtn3oos3Dnin9Anowi9omUgIJDa11/K3haC+22jgOY/pSreO/WPj1ULkAS+wH+8JO2Xf2lf1k/86M1ZUxwlza2PJIK9zq3Os7hb8XqhrYBnA4UJ2j4EifX1MABjyWtZhYmystYZV6FL/VeXtaeJJf6i6zT+w31veRnbBvWKHd+qN/3hb5DzL6AWoMbFvY6pDCGe7iqJ8pR19XWntxNgP+mCCPCa5NcG2C6xBch+COCO6I4AyCMwiuTnB1glsluFWCWye4dYLbILgNgisTXJngdglul+AYwTGF4/H6hvuQWlcer2vkV9eTy13c9wvC38AdOyAqcqxSLuZojnCRhSJ9hfRJMswtwPqH2b5NpcNZ6vPDioG9R6gs+c5ztIzrhj2UMqdUnDuj4gJaa3Ilh+mBVVVsjmQ10GyO5TSs2orz19Fn4cmypRBrYzuK43hNofIElVeo5/K0SUFW0gaez+G5F61m/8SP8ohOJdqTDePi/ZMZv23UTimbzjxIZ7cfplaexjfRDNyDRbiP8R/BCrzA+S9iDq/gDbyD99O/EsnE7cTdCJ2ciDW3YORJLP4HeoKDuw==</latexit>l Precision parameter 

topic k:<latexit sha1_base64="UQxKIGCFFQRxgexKVgDYDgQg/3s=">AAAHdXichZXNbtNAEMenLTSlfDSBCxJCqgityiVsAhKIUwtUompVpQlpUzUlst11atmxLdtJmlh5AV6AA+IAEiDEY3DhBTj0ERDHIvXCgfHakZN42jpKdnfm95/M7K53ZdvQXI+x44nJqUuXp1MzV2avXrt+Yy6dubntWi1H4RXFMiynKksuNzSTVzzNM3jVdrjUlA2+I+svAv9OmzuuZpmvva7N95tSw9RUTZE8NNXTmZou2bZUr9mH2ht/SX/Qr6ezLMfEM5/s5KNOFqKnaGWmFqEGB2CBAi1oAgcTPOwbIIGLnz3IAwMbbfvgo83Bnib8HPowi9oWUhwJCa06/jZwtBdZTRwHMV2hVvBfDPw6qJyHBfaLfWMn7Cf7zn6zf2fG8kWMIJcutnKo5XZ97u3t8umFqia2HhzGqnMUMtLn1+SBCk9FLRrWZgtLUKUSxm/33p2Un5UW/EX2if3B+j6yY/YDKzTbf5XPW7z0XkQ/QI2KbRtbBbI4wz6OBpnK6POFtR9lE/NHBHlEcF2C6xJcj+B6BNchuA7BqQSnEpxOcDrBrRDcCsGtEdwawa0T3DrBVQmuSnC7BLdLcIzgWIKTo/UN9iG1rnK0rqE/uZ6y2MUDPyf8TdyxMVET4yRlY47aCBdaKNJNkC5JBrl5WP8wO7Al6WCWBvywIrb3CZUh3nkZLeO6YQ+lLCYqLp5RcRmtDbGSw3RsTSo2RrKKNRtjOQ2rNqP8FfQZeLJsJojVsR0l43g1QZUIqpSgXorTJgsFQat4PgfnXriagxM/zCM8lWhPIYiL909+/LZJdrYLufyjXGHrcXb5eXQTzcAduAdLGP8JLMMrnP8K5tCBD/AFvk6fpu6m7qcWQ3RyItLcgpEn9fA/zYeIYQ==</latexit>(k)should be estimated for each

実際のテキストでの実験
l データ: Corpus of Historical American English (COHA)– 先行研究に基づいて、分析に使う単語を選択

l 数値的評価:– ベースライン: SCAN, HDP-LDA, BERT
– 時間変化するトピック分布のcoherenceとdiversity
を調べて評価実験結果 (1)l “coach” : estimated senses=2
Infinite SCAN

SCAN
Trainer

Carriage

Semantic shift: “carriage” →“trainer”

実験結果 (2)l “power” : estimated senses=6
Infinite SCAN
Legal

EnergyAuthority
MentalSense birth: “energy”

SCAN

実験結果 (3)l モデルは意味の数を正しく発見できているのか？
l OntoNotesデータセットからランダムに120語を選んでモデルを計算し、正解率を計算
Model
HDP-LDABERT + K-means
BERT + DBSCANSCAN (K = 5)
SCAN (K = 8)
Infinite SCAN

Accuracy
0.258
0.217
0.125
0.158
0.000
0.358

PCC
0.019
0.026
-0.070
0.141
0.087
0.474

Prediction results for the number of senses.

Correlation between actual and estimatednumber of senses by Infinite SCAN.

構文解析と自然言語処理
l 構文解析は、自然言語処理の中心的テーマの一つ
l 構文解析：文が与えられたとき、その木構造を可能な無数の組み合わせの中から自動的に求める
S

×

○

VP

S

NP
VP

PP
NP
N

NP
V

DT

VP

NP
N

P DT NI saw the girl with a hat

NP

V

N

NP
N

PP
P

DT DT VI saw the girl with a hat

構文解析の難しさ
l 動的計画法を使っても、文長NについてO(N3)の計算時間 (CYKアルゴリズム)
l 与えられた文に対して可能な膨大な木構造の中から、正解の一つだけを選ぶタスク
詳しくは
こうした
教科書を
参照

組み合わせ範疇文法 (CCG)l 語を構文木に沿って組み合わせることで、句構造に対応する表現を生成
l 最終的に、Sが得られれば解析終了
l 途中の句構造に、その下にある単語の情報が反映されている

CCGの問題点

l 単語につけられた (S＼N)/Nのようなタグが離散的à 埋め込みベクトル化したい– 上位の句構造のベクトルが、合成された単語のベクトルを反映できる
l すべての単語に、あらかじめ人手でこうしたタグを付与する必要がある (教師なし学習; 今回は範囲外)

Holographic CCG
l 基本的なアイデア: CCGをベクトル合成と組み合わせられないか？
l CVGのように合成毎にパラメータがあると、厖大なパラメータが必要で過学習してしまう
l 何か、2つのベクトルを合成する見通しのよい

数学的な方法はないか？
à Holographic composition (Plate 1995)

Holographic compositionl 知識グラフの埋め込み手法として、Nickel+(AAAI 2016)

で提案 (最初の提案はPlate(1995))l Circular correlation (巡回相関)

c=a?b
Pd 1
[c]k = i=0 ai b(k+i)mod d<latexit sha1_base64="betML5pmy0QecyJ0yEHjqPFjNTM=">AAAHinichZXNbtNAEMenLeBSCk3hgsQlImrEKdoEJL6E1EIrUbWq2oS0qZoq8rrr1IqdWLaTNonyAog7B04gcUDcuMKNCy/AoY+AOBaJCwfGa1tO4mnrKNmdmd9/MuNdr7ltGq7H2MnE5NSly1eU6asz12av35hLzd/cdlttRxNlrWW2nApXXWEaTVH2DM8UFdsRqsVNscMbL/z4Tkc4rtFqvvK6tti31HrT0A1N9dBVS2Wrluodcr2vDdLP0pGhDqqupzqRyQe1VIblmLzSyUk+nGQgvDZb81NZqMIBtECDNlggoAkezk1QwcXPHuSBgY2+feijz8GZIeMCBjCD2jZSAgkVvQ38raO1F3qbaPs5XanW8F9M/DqoTMMC+8k+sVP2g31mv9i/M3P1ZQ6/li6OPNAKuzb3+nbp74UqC0cPDmPVOQqO9Pk9eaDDI9mLgb3Z0uN3qQX5O723p6UnxYV+ln1gv7G/9+yEfccOm50/2sctUXwnsx+gRsexg6MGGbzDfbSiSjnG+tI7CKuJ+WOCPCa4LsF1Ca5HcD2COyK4I4LTCU4nuAbBNQhuieCWCG6V4FYJbo3g1giuQnAVgtsluF2CYwTHEhwP19ffh9S68nBdg3hyPbncxVFcEHELd2xMVKWdpGys0RjhAg9FugnSJUm/Ng/7H2YjX5L271LEDyti/4BQmfKZ5+gZ1w1HKOVmouPNMzouobcuV3KYjr1JxfpIVbFmfaymYdVGWL+GMRNPlo0EsTK2ozjaKwmqSFDFBLUsT5sMFCSt4/nsn3vBakYnflBHcCrRkYKfF98/+fG3TXKyXcjl7+cKWw8yi8/DN9E03IG7cA/zP4RFeIn3v4w1vIEv8BW+KbNKQXmsPA3QyYlQcwtGLmX5PwfZkMg=</latexit>

l 定義:<latexit sha1_base64="HGh733nN5Yn4WsVr8Q7JUjEqxCs=">AAAHoXichZXNbtNAEMenLZBSPprCBYmLRdSqCFptAhIIqVILVKJqVaUpbVMlxfI6m2DFji3bSZtafgFegAMnQBwQj8EFriAOfQTEsUhcODBeO0oaT4ujZHdnfv/JzO56lzum4fmMHY2Mjp07fyEzfnHi0uUrVyezU9e2Pbvt6mJLt03bLXPNE6bRElu+4Zui7LhCs7gpdnjzSeTf6QjXM+zWc7/riD1La7SMuqFrPprU7EJF31ObyoJS9cWB7/ldU0xUvbalBsYCC18Etbl8qGiqoXA1mG3eMW4HVddSLLsWVu/WQjWbY/NMPkq6k086OUieoj01NgNVqIENOrTBAgEt8LFvggYefiqQBwYO2vYgQJuLPUP6BYQwgdo2UgIJDa1N/G3gqJJYWziOYnpSreO/mPh1UanANPvBPrJj9oV9Yj/Z31NjBTJGlEsXWx5rhaNOvrqx+ee/KgtbH172VWcoONJn1+RDHR7KWgyszZGWqEo9jt85fH28+ag0Hcywd+wX1veWHbHPWGGr81v/sCFKb2T0Gmrq2Haw1SGHMxzgqJcpR18grWGSTZ8/IMgDgusSXJfgDgnukOD2CW6f4OoEVye4JsE1CW6J4JYIboXgVghuleBWCa5McGWC2yW4XYJjBMdSHE/WN9qH1LryZF1jf3o9udzFPb8g/Bbu2D5RleM05WCOxgkutlCklyI9koxy87H+QbZnS9PRLPX4QUXfHhIqU77zHC3DukEPpSymKi6eUvEmWhtyJQfpvjWtWDuRVV+zNpTToGo9yV9Hn4kny3qKWB7aURzHyymqRFClFPVUnjY5KEi6judzdO7Fq9k78eM84lOJ9hSiuHj/5Idvm3RnuzCfvzdf2LifW3yc3ETjcBNuwSzGfwCL8AznfwtzeA9f4Rt8z+QyK5liphSjoyOJ5jqceDKVf4/jmEg=</latexit>

a
b 0
b 1
b 2
b 3
<latexit sha1_base64="oiesVcH+dhxk+byjfr5IEcTBvdY=">AAAHaXichZXNbtNAEMenLTSlfLSBSwWXCqsVp2gTkECcWqASVauqTUiTqqkqr7sOVpzYsp20iZUX4MYJiZ5A4oB4DC68AIc+AuqxSFw4MF47cpKdFkfJ7s78/pMZ73rMXdvyA8bOJianrl2fzszcmL156/adufns3V3faXuGKBuO7XhVrvvCtlqiHFiBLaquJ/Qmt0WFN15G/kpHeL7ltN4EXVccNPV6yzItQw/QVKl1hBHq/cN5jeWYvBbVST6ZaJBc2052ahlqcAQOGNCGJghoQYBzG3Tw8bMPeWDgou0AQrR5OLOkX0AfZlHbRkogoaO1gb91XO0n1hauo5i+VBv4LzZ+PVQuwhL7yb6yC/aDfWO/2N9LY4UyRpRLF0cea4V7OPduofTnv6omjgG8TVVXKDjSV9cUgAnPZC0W1uZKS1SlEcfv9D5clJ4Xl8Jl9pmdY32f2Bn7jhW2Or+NLzuieCqjH6HGxLGDowEa3uEQV4NMOfpCae0n2aT8CUGeEFyX4LoE1yO4HsEdE9wxwZkEZxJcg+AaBLdKcKsEt05w6wS3QXAbBFcluCrB7RHcHsExgmMKx5P9jc4hta882dfYr+4nl6d44BeEv4knNiVqcq1SLuZojXCxhSJ9hfRJMsotwPqH2YFNpaO7NOCHFam9T6hs+cxztIzrhj2UclupePuSiktorcudHKZTq6rYHMkq1WyO5TSs2kryN9BnY2fZUoi1sRPFcb2mUEWCKirUK9ltNChI2sT+HPW9eDcHHT/OI+5KtKcQxcX3T378baNOdgu5/ONcYeeJtvIieRPNwAN4CI8w/lNYgdd4/8uy976Hj3A6fZ7JZhYy92N0ciLR3IORK6P9Azw5hEg=</latexit><latexit sha1_base64="j6Dmaf07Gg/R402UbafYOlQBc9g=">AAAHe3ichZXNbtNAEMenLTSlfDSFA0hcIqxWCKFoE0AgTi1QiapVlSakTdVUkdddByt2bNlO2sTKC/ACHDgB4oCAp+DCC3DoIyCOReICEuO1I6fxtDhKdnfm95/MeNdj7piG5zN2NDE5de78dGbmwuzFS5evzGXnr255dsfVRFWzTdutcdUTptEWVd/wTVFzXKFa3BTbvPU09G93hesZdvuF33PEnqU224ZuaKqPpkb2er0rtIAPGkFQ9yzVNOvN5iDHBo2swvJMXrn0pBBPFIivkj0/tQh12AcbNOiABQLa4OPcBBU8/OxCARg4aNuDAG0uzgzpFzCAWdR2kBJIqGht4W8TV7uxtY3rMKYn1Rr+i4lfF5U5WGDf2Ud2zL6xT+wH+3NqrEDGCHPp4cgjrXAac69uVH7/V2Xh6MPLRHWGgiN9dk0+6PBI1mJgbY60hFVqUfxu//Vx5XF5IVhk79hPrO8tO2JfscJ295f2YVOU38jo+6jRceziqIGCdzjA1TBTjr5AWgdxNgl/SJCHBNcjuB7B9QmuT3AHBHdAcDrB6QTXIrgWwS0T3DLBrRLcKsGtEdwawdUIrkZwOwS3Q3CM4FiK4/H+hueQ2lce72vkT+8nl6d46BeE38ITmxB1uU5TDuZonOAiC0V6KdIjyTA3H+sfZYe2NB3epSE/qkjsA0Jlymeeo2VcN+qhlKVUxaVTKq6gtSl3cpROrGnF+omsEs36WE6jqo04fw19JnaWjRSxMnaiOK5XUlSZoMop6pnsNgoUJa1jfw77XrSbw44f5RF1JdpTDOPi+6cw/rZJT7aK+cK9fHHzvrL0JH4TzcBNuAW3Mf5DWILneP+rsqu+h8/wZfpvRsncydyN0MmJWHMNTlyZB/8AHLWLWA==</latexit><latexit sha1_base64="FUy6smyVQ2UWmjvcfUxkR36puK8=">AAAHe3ichZXNbtNAEMenLTSlfDSFA0hcIqxWCKFoE0AgTi1QiapVlSakTdVUkdfdBCt2bNlO2sTyC/ACHDgB4oCAp+DCC3DoIyCOReICEuO1I6fxtDhKdnfm95/MeNdjbhu66zF2NDE5de78dGbmwuzFS5evzGXnr265VtfRRFWzDMupcdUVht4RVU/3DFGzHaGa3BDbvP009G/3hOPqVueF17fFnqm2OnpT11QPTY3s9XpPaD4PGr5fd03VMOqtVpArBI2swvJMXrn0pBBPFIivkjU/tQh12AcLNOiCCQI64OHcABVc/OxCARjYaNsDH20OznTpFxDALGq7SAkkVLS28beFq93Y2sF1GNOVag3/xcCvg8ocLLDv7CM7Zt/YJ/aD/Tk1li9jhLn0ceSRVtiNuVc3Kr//qzJx9OBlojpDwZE+uyYPmvBI1qJjbba0hFVqUfze4PVx5XF5wV9k79hPrO8tO2JfscJO75f2YVOU38jo+6hp4tjDUQMF77CPq2GmHH2+tAZxNgl/SJCHBNcnuD7BDQhuQHAHBHdAcE2CaxJcm+DaBLdMcMsEt0pwqwS3RnBrBFcjuBrB7RDcDsExgmMpjsf7G55Dal95vK+RP72fXJ7ioV8QfhNPbELU5TpN2ZijfoKLLBTppkiXJMPcPKx/lB3a0nR4l4b8qCKxB4TKkM88R8u4btRDKUupikunVFxBa0vu5CidWNOK9RNZJZr1sZxGVRtx/hr6DOwsGyliZexEcVyvpKgyQZVT1DPZbRQoSrqJ/Tnse9FuDjt+lEfUlWhPMYyL75/C+NsmPdkq5gv38sXN+8rSk/hNNAM34RbcxvgPYQme4/2vyq76Hj7Dl+m/GSVzJ3M3QicnYs01OHFlHvwDI7WLWQ==</latexit><latexit sha1_base64="ykMAEAcPlH4FkxmMymQ8kmgM7eU=">AAAHe3ichZXNbtNAEMenLTSlfDSFA0hcIqJWCKFoE0AgTi1QiapVlSakTdVUkdddByv+ku2kTay8AC/AgRMgDgh4Ci68AIc+AuJYJC4gMV47chpPi6Nkd2d+/8mMdz3mjqF7PmNHE5NT585PZ2YuzF68dPnKXHb+6pZnd1xV1FTbsN06Vzxh6Jao+bpviLrjCsXkhtjm7aehf7srXE+3rRd+zxF7ptKydE1XFR9Nzez1RleoAR80g6DhmYphNFqtQa40aGbzrMDklUtPivEkD/FVtuenFqEB+2CDCh0wQYAFPs4NUMDDzy4UgYGDtj0I0ObiTJd+AQOYRW0HKYGEgtY2/rZwtRtbLVyHMT2pVvFfDPy6qMzBAvvOPrJj9o19Yj/Yn1NjBTJGmEsPRx5phdOce3Wj+vu/KhNHH14mqjMUHOmza/JBg0eyFh1rc6QlrFKN4nf7r4+rjysLwSJ7x35ifW/ZEfuKFVrdX+qHTVF5I6Pvo0bDsYujCnm8wwGuhply9AXSOoizSfhDgjwkuB7B9QiuT3B9gjsguAOC0whOI7g2wbUJbpnglgluleBWCW6N4NYIrk5wdYLbIbgdgmMEx1Icj/c3PIfUvvJ4XyN/ej+5PMVDvyD8Jp7YhGjIdZpyMEf9BBdZKNJLkR5Jhrn5WP8oO7Sl6fAuDflRRWIfECpDPvMcLeO6UQ+lLKcqLp9ScRWtLbmTo3RiTSvWT2SVaNbHchpVbcT5q+gzsLNspIiVsRPFcb2SoioEVUlRz2S3yUNJ0hr257DvRbs57PhRHlFXoj2lMC6+f4rjb5v0ZKtUKN4rlDbv55eexG+iGbgJt+A2xn8IS/Ac739NdtX38Bm+TP/N5DN3MncjdHIi1lyDE1fmwT8qtYta</latexit><latexit sha1_base64="4c/k9OCRGA2Jnu7EwzAS4e1lvLc=">AAAHe3ichZXNbtNAEMenLTSlfDSFA0hcIqJWCKFok4JAnFqgElWrKk1Im6qpIq+7CVb8pdhJm1h5AV6AAydAHBDwFFx4AQ59BMSxSFxAYrx2lI+dto6S3Z35/Scz3vWYu6bh+YwdT0xOXbg4nZi5NHv5ytVrc8n569ue02rqoqQ7ptMsc80TpmGLkm/4pii7TaFZ3BQ7vPEs9O+0RdMzHPul33HFvqXVbaNm6JqPpmryZqUt9ID3qkFQ8SzNNCv1ei+11Ksm0yzD5JVSJ9l4kob4yjvzU4tQgQNwQIcWWCDABh/nJmjg4WcPssDARds+BGhr4syQfgE9mEVtCymBhIbWBv7WcbUXW21chzE9qdbxX0z8NlGZggX2g31iJ+w7+8x+sr+nxgpkjDCXDo480gq3Ovf6VvHPuSoLRx9eDVRnKDjSZ9fkQw0ey1oMrM2VlrBKPYrf7r45KT4pLASL7D37hfW9Y8fsG1Zot3/rH7dE4a2MfoCaGo5tHHVI4x0OcNXPlKMvkNZenM2APyLII4LrEFyH4LoE1yW4Q4I7JLgawdUIrkFwDYJbIbgVglsjuDWCWye4dYIrE1yZ4HYJbpfgGMExhePx/obnkNpXHu9r5Ff3k8tT3PcLwm/hiR0QFblWKRdzNEa4yEKRnkJ6JBnm5mP9w2zfptLhXerzw4qBvUeoTPnMc7SM64Y9lDKvVJw/peIiWutyJ4fpgVVVbIxkNdBsjOU0rNqM89fRZ2Jn2VSI1bETxXG9qlAFgioo1HPZbdKQk3QN+3PY96Ld7Hf8KI+oK9GeXBgX3z/Z8beNOtnOZbJLmdzWg/Ty0/hNNAO34Q7cxfiPYBle4P0vya76Ab7A1+l/iXTiXuJ+hE5OxJobMHIlHv4HMbWLWw==</latexit>

c0
c1
c2
c3
<latexit sha1_base64="b5IvOCc6K6PJhEJw1UDNMoBpW2U=">AAAHZXichZXPTxNBFMcfoBRRBNQYEw8SG4inZlpNNJ5AJZFASGktlABpdpbZuun+yu620Db9B0y8ysGTJh6Mf4YX/wEP/AXGeMTEiwffzm6zbeeB27Qz897n+/rezOwM9ywzCBk7HRufuHR5MjN1ZfrqtZnrs3PzN7YDt+nroqK7lutXuRYIy3REJTRDS1Q9X2g2t8QObzyP/Dst4Qem67wK2544sLW6YxqmroVoKus1VpvLshyTz4LaySedLCRP0Z2fWIJ9OAQXdGiCDQIcCLFvgQYBfvYgDww8tB1AF20+9kzpF9CDadQ2kRJIaGht4G8dR3uJ1cFxFDOQah3/xcKvj8oFWGTf2Wd2xr6xL+wn+3turK6MEeXSxpbHWuHVZt/cKf/5r8rGNoTXqeoCBUf64ppCMOCJrMXE2jxpiarU4/itzslZ+WlpsbvEPrJfWN8Hdsq+YoVO67f+aUuU3svoh6gxsG1hq0MWZ7iLo36mHH1dae0l2aT8MUEeE1yb4NoE1yG4DsEdEdwRwRkEZxBcg+AaBLdCcCsEt0ZwawS3TnDrBFcluCrB7RLcLsExgmMKx5P1jfYhta48WdfYr64nl7u47xeE38YdmxL7cqxSHuZoDnGxhSIDhQxIMsotxPoH2b5NpaNZ6vODitTeI1SWfOc5WkZ1gx5KWVQqLp5TcRmtdbmSg3RqVRUbQ1mlmo2RnAZVm0n+OvosPFk2FWJ1ZEdxHK8qVImgSgr1Qp42WShI2sDzOTr34tXsn/hxHvGpRHsKUVy8f/Kjt43a2S7k8g9zha1H2eVnyU00BXfhPjzA+I9hGV7i/Fcwhzq8hXdwMvkjM5O5lbkdo+NjieYmDD2Ze/8AhVyCHw==</latexit><latexit sha1_base64="1tpSYsr1RV/Lnjt/wubcNh6mc28=">AAAHZXichZVBTxNBFMcfoBRRBNQYEw8SG4inZraaaDyBSiKBkNJaKAHS7CyzddPt7mZ3W2ibfgETr3LwpIkH48fw4hfwwCcwxiMmXjz4dnabbTsP3Kadmfd+/9f3ZmZnuGdbQcjY6dj4xKXLk5mpK9NXr81cn52bv7EduE3fEGXDtV2/wvVA2JYjyqEV2qLi+UJvcFvs8PrzyL/TEn5guc6rsO2Jg4ZecyzTMvQQTSWjqlXnsizH5LOgdrSkk4XkKbjzE0uwD4fgggFNaIAAB0Ls26BDgJ890ICBh7YD6KLNx54l/QJ6MI3aJlICCR2tdfyt4WgvsTo4jmIGUm3gv9j49VG5AIvsO/vMztg39oX9ZH/PjdWVMaJc2tjyWCu86uybO6U//1U1sA3hdaq6QMGRvrimEEx4ImuxsDZPWqIqjTh+q3NyVnpaXOwusY/sF9b3gZ2yr1ih0/ptfNoSxfcy+iFqTGxb2BqQxRnu4qifKUdfV1p7STYpf0yQxwTXJrg2wXUIrkNwRwR3RHAmwZkEVye4OsGtENwKwa0R3BrBrRPcOsFVCK5CcLsEt0twjOCYwvFkfaN9SK0rT9Y19qvryeUu7vsF4W/gjk2JfTlWKQ9ztIa42EKRgUIGJBnlFmL9g2zfptLRLPX5QUVq7xEqW77zHC2jukEPpSwoFRfOqbiE1ppcyUE6taqKjaGsUs3GSE6Dqs0kfwN9Np4smwqxOrKjOI5XFapIUEWFeiFPmyzkJW3i+Ryde/Fq9k/8OI/4VKI9+Sgu3j/a6G2jdrbzOe1hLr/1KLv8LLmJpuAu3IcHGP8xLMNLnP8y5lCDt/AOTiZ/ZGYytzK3Y3R8LNHchKEnc+8fjFuCIA==</latexit><latexit sha1_base64="zIQKyfP+VMLBgSpAADtOKKk8A7M=">AAAHZXichZVBTxNBFMcfoBRRBNQYEw8SG4inZrqaaDyBSiKBkNJaKAHS7CyzddPt7mZ3W2ibfgETr3LwpIkH48fw4hfwwCcwxiMmXjz4dnabbTsP3Kadmfd+/9f3ZmZnuGdbQcjY6dj4xKXLk5mpK9NXr81cn52bv7EduE3fEGXDtV2/wvVA2JYjyqEV2qLi+UJvcFvs8PrzyL/TEn5guc6rsO2Jg4ZecyzTMvQQTSWjqlXnsizH5LOgdvJJJwvJU3DnJ5ZgHw7BBQOa0AABDoTYt0GHAD97kAcGHtoOoIs2H3uW9AvowTRqm0gJJHS01vG3hqO9xOrgOIoZSLWB/2Lj10flAiyy7+wzO2Pf2Bf2k/09N1ZXxohyaWPLY63wqrNv7pT+/FfVwDaE16nqAgVH+uKaQjDhiazFwto8aYmqNOL4rc7JWelpcbG7xD6yX1jfB3bKvmKFTuu38WlLFN/L6IeoMbFtYWtAFme4i6N+phx9XWntJdmk/DFBHhNcm+DaBNchuA7BHRHcEcGZBGcSXJ3g6gS3QnArBLdGcGsEt05w6wRXIbgKwe0S3C7BMYJjCseT9Y32IbWuPFnX2K+uJ5e7uO8XhL+BOzYl9uVYpTzM0RriYgtFBgoZkGSUW4j1D7J9m0pHs9TnBxWpvUeobPnOc7SM6gY9lLKgVFw4p+ISWmtyJQfp1KoqNoaySjUbIzkNqjaT/A302XiybCrE6siO4jheVagiQRUV6oU8bbKgSdrE8zk69+LV7J/4cR7xqUR7tCgu3j/50dtG7WxrufzDnLb1KLv8LLmJpuAu3IcHGP8xLMNLnP8y5lCDt/AOTiZ/ZGYytzK3Y3R8LNHchKEnc+8fk1qCIQ==</latexit><latexit sha1_base64="w2bl39ATPsmx16fG1/Zz1uAmEPg=">AAAHZXichZVBTxNBFMcfoBRRBNQYEw8SG4inZlpMIJ5AJZFASGktlABpdpbZuul2d7O7LbRNv4CJVzl40sSD8WN48Qt44BMY4xETLx58O7vNtp0HbNPOzHu//+t7M7Mz3LVMP2DsbGR07Nr18dTEjcmbt6ZuT8/M3tnxnYani5LuWI5X5povLNMWpcAMLFF2PaHVuSV2ee1F6N9tCs83Hft10HLFYV2r2qZh6lqApqJeWazMpFmGyWdO7WTjThriJ+/Mji3AARyBAzo0oA4CbAiwb4EGPn72IQsMXLQdQgdtHvZM6RfQhUnUNpASSGhoreFvFUf7sdXGcRjTl2od/8XCr4fKOZhnP9gXds6+s6/sF/t3YayOjBHm0sKWR1rhVqbfPij+vVJVxzaAN4nqEgVH+vKaAjBgWdZiYm2utIRV6lH8Zvv0vPisMN9ZYJ/Yb6zvIztj37BCu/lH/7wtCh9k9CPUGNg2sdUhjTPcwVEvU46+jrR242wS/oQgTwiuRXAtgmsTXJvgjgnumOAMgjMIrkZwNYJbJbhVglsnuHWC2yC4DYIrE1yZ4PYIbo/gGMExhePx+ob7kFpXHq9r5FfXk8td3PMLwl/HHZsQB3KsUi7maA5wkYUifYX0STLMLcD6+9meTaXDWerx/YrE3iVUlnznOVqGdf0eSplXKs5fUHERrVW5kv10YlUVmwNZJZrNoZz6VVtx/jr6LDxZthRibWhHcRyvKVSBoAoK9VKeNmnISdrA8zk896LV7J34UR7RqUR7cmFcvH+yw7eN2tnJZbKLmdz20/TK8/gmmoCH8BieYPwlWIFXOP8lzKEK7+A9nI7/TE2l7qXuR+joSKy5CwNP6tF/mlmCIg==</latexit>

Holographic composition (3)l 性質:– Non-commutative: 方向を保存
a ? b 6= b ? a<latexit sha1_base64="b4z3f7MXT9/NusZ36TG5Psmu7tU=">AAAHnHichZXNbtNAEMenLeASPprCBQkJRUStOFWbgETFqYVWokqp0oSkqZoq8rrr1IrjGNtJm0R5AV6AAxISSBwQjwEHLnDj0EdAHIvEhQPjtSMn8bR1lOzOzO8/mfGu19w2Dddj7GRqeubS5SvK7NXEtes3bs4l52+V3Vbb0URJa5ktp8JVV5iGJUqe4ZmiYjtCbXJT7PDGMz++0xGOa7Ssl17XFvtNtW4ZuqGpHrpqyeVqU/UOud5XB1XXU52hyQepqiVeJSJ7LKwOask0W2LySsUnmXCShvDKt+ZnFqEKB9ACDdrQBAEWeDg3QQUXP3uQAQY2+vahjz4HZ4aMCxhAArVtpAQSKnob+FtHay/0Wmj7OV2p1vBfTPw6qEzBAvvJPrFT9o19Zr/YvzNz9WUOv5YujjzQCrs29/pO8e+FqiaOHhxGqnMUHOnze/JAh2XZi4G92dLjd6kF+Tu9N6fFJ4WF/iL7wH5jf+/ZCfuCHVqdP9rHbVF4K7MfoEbHsYOjBmm8w320hpVyjPWldxBWE/HHBHlMcF2C6xJcj+B6BHdEcEcEpxOcTnANgmsQ3CrBrRLcBsFtEFyO4HIEVyG4CsHtEtwuwTGCYzGOh+vr70NqXXm4rkE8vp5c7uJhXBDxJu7YiKhKO07ZWKMxxgUeinRjpEuSfm0e9j/KDn1x2r9LQ35UEfkHhMqUzzxHz6RuNEIp87GO82d0XERvXa7kKB1544rNsaoizeZETaOqrbB+DWMmnixbMWJ9YkdxtNdjVIGgCjFqTZ42achKWsfz2T/3gtUcnvhBHcGpREeyfl58/2Qm3zbxSTm7lHm4lN1+lF55Gr6JZuEu3IcHmP8xrMBzvP8lrOEdfIXv8EO5p6wpOeVFgE5PhZrbMHYp5f8hppkF</latexit>– Non-associative: 構造を保存(a ? b) ? c 6= a ? (b ? c)
– 構文構造を保存: たとえば “loves to eat”のベクトルはv“loves to eat” = vloves ? v“to eat” = vloves ? (vto ? veat )<latexit sha1_base64="COWYPVcsXmuAhlhz86G/Hn287Ho=">AAAHwHichZXfT9NQFMcPoA7nD4a+mPiyuEDghdxNE41PgJJIIAQ2ByOMkN5yuzXr2tKWwbb0H/Af8MEnTXww/hf64j/gA/+BxifFxBcfPL1t0209YMnovd/z+Z6d03t3y21Ddz3GzsbGJ65cvZaZvJ69cfPW7anc9J1t1zp2VFFVLcNyalxxhaGbourpniFqtiOUNjfEDm89C+I7HeG4umW+9Lq22G8rDVPXdFXxUDrI1efqbcVrcq2v+HXXU5x4yv35vBTysaL62bopjrIjhrnEMZRA9ecPcgW2wOSVTw+K0aAA0bVpTU/MQh0OwQIVjqENAkzwcGyAAi7+7UERGNio7UMfNQdHuowL8CGL3mOkBBIKqi3838DZXqSaOA9yutKt4rcY+HHQmYcZ9pV9YOfsC/vIvrO/F+bqyxxBLV2889Ar7IOpV/cqf/7rauPdg2biusTBkb68Jw80eCJ70bE3WypBl2qYv9N7fV55Wp7pz7J37Af295adsc/Yodn5rb7fEuU3MvshejS8d/CuQgGfcB9ncaUcY32p+lE1CX9KkKcE1yW4LsH1CK5HcCcEd0JwGsFpBNciuBbBLRHcEsGtEtwqwa0R3BrB1QiuRnC7BLdLcIzgWIrj0foG+5BaVx6taxhPryeXuziOCyLexh2bEHU5T1M21qgPcaFCkW6KdEkyqM3D/gfZWEvTwVOK+UFHovuEy5C/eY7KqG8wQjk3Ux1vXtBxBdWGXMlBOlHTjvWhqhLP+khNg66NqH4VYwaeLBspYmVkR3Gcr6SoMkGVU9RzedoUoCRpDc/n4NwLVzM+8cM6wlOJjpSCvPj+KY6+bdKD7dJC8eFCaetRYXE5ehNNwn14AHOY/zEswgt8/lWs4RN8g5/wK7OcaWaszFGIjo9FnrswdGV6/wAXXKe3</latexit><latexit sha1_base64="IwjFc5ntcnCxR18R5wzTfKSOCpI=">AAAIEXicjZVLb9NAEMcnLZASHm3pBYlLRNQHl2oTkHhISH1QiapV1aakTdVUie1uihUntuyN29TyF+DKgQMnkDggxAfghuCCxJlDPwLiWCQuHBivHSW1py2Okt2d+f0nM7vrXdUydEcwdpQaGLxw8VJ66HLmytVr14dHRm9sOGbb1nhJMw3TLquKww29xUtCFwYvWzZXmqrBN9XGfODfdLnt6GbrmehYfKep7LX0uq4pAk3VkY8Vl2ue61e9iuAHwqvVDNPlTlaYWa6IyUnfzzzOxhhJ+H624gjFjjtrtf+XZqZibmGeFhYj+v6d6kiOTTP5ZJOdfNTJQfSsmqODE1CBXTBBgzY0gUMLBPYNUMDBzzbkgYGFth3w0GZjT5d+Dj5kUNtGiiOhoLWBv3s42o6sLRwHMR2p1vBfDPzaqMzCOPvB3rNj9o19YD/Z31NjeTJGkEsHWzXUcqs6/OLm+p9zVU1sBTzvqc5QqEifXZOAOjyQtehYmyUtQZVaGN89fHW8/qg47k2wt+wX1veGHbGvWGHL/a29W+PF1zL6Lmrq2LrYapDDGfZw1M1URZ8nrX6UTY8/IMgDgusQXIfgDgnukOD2CW6f4OoEVye4BsE1CG6W4GYJbpHgFgluieCWCK5McGWC2yK4LYJjBMcSnBqtb7APqXVVo3UN/cn1VOUu7vo54W/iju0RFTlOUhbmqJ/gQgtFOgnSIckgN4H197NdW5IOZqnL9yt6dp9QGfKdV9ES1/V7KOVqouLVUypeR+ueXMl+umdNKpZPZNXTLMdy6letRPlr6DPwZFlJEAuxHaXieCFBFQmqmKCeyNMmBwVJ1/F8Ds69cDW7J36YR3gq0Z5CEBfvn3z8tkl2NgrT+bvThbV7uZm56CYagltwG6Yw/n2Ygac4/yXQUmOph6m51Hz6ZfpT+nP6S4gOpCLNGJx40t//AYLQx1A=</latexit>l これは (vloves ? vto ) ? veat とは異なる<latexit sha1_base64="kPHIxt1IBDXt0aaHSFY7+sIcoT0=">AAAHsnichZXNbtNAEMenLZASPprCBYlLRNSqXKJNqATi1AKVqFpFaULalKaKbHcTrDixZW/cJJZfgBfgwAkkDogrb8CFF+BQrnBBHIvEhQPjtaN8eNo6SnZ35vefzOyud1XL0B3B2MnM7Nyly1cS81eT167fuLmQWry165hdW+MVzTRMu6oqDjf0Dq8IXRi8atlcaasG31NbTwP/nsttRzc7L0Tf4odtpdnRG7qmCDTVU4WVmss1z/XrXk3wnvAM0+WO76drjlDsdHLKK0zfvx/5plxcEb5fT2VYlsknHe/kok4GoqdoLs4tQw2OwAQNutAGDh0Q2DdAAQc/B5ADBhbaDsFDm409Xfo5+JBEbRcpjoSC1hb+NnF0EFk7OA5iOlKt4b8Y+LVRmYYl9o19ZKfsK/vEfrF/Z8byZIwglz62aqjlVn3h9Z3y3wtVbWwFvBqpzlGoSJ9fk4AGPJK16FibJS1BlVoY3x28OS0/Li15y+w9+431vWMn7AtW2HH/aB92eOmtjH6Emga2LrYaZHCGPRwNM1XR50mrH2Uz4nsE2SO4PsH1CW5AcAOCOya4Y4JrEFyD4FoE1yK4dYJbJ7hNgtskuC2C2yK4KsFVCW6f4PYJjhEci3FqtL7BPqTWVY3WNfTH11OVu3jo54S/jTt2RNTkOE5ZmKM+wYUWinRipEOSQW4C6x9nh7Y4HczSkB9XjOw+oTLkO6+iZVo37qGUxVjFxTMqLqO1KVdynB5Z44rtiaxGmu2pnMZVhSh/DX0GniyFGLExtaNUHG/EqBJBlWLUM3naZCAv6Qaez8G5F67m8MQP8whPJdqTD+Li/ZObvm3ind18Nvcgm99Zzaw9iW6iebgL92AF4z+ENXiO81/BHD7Dd/gBPxOriZcJJaGF6OxMpLkNE0/C+A8DC6MC</latexit>

Holographic CCG
l Hol-CCGのイメージ– 単語ベクトル自体を最適化して、合成して最終的に文Sのベクトルが得られるようにする

実際の学習の様子 “My sister loves to eat”
“My sister loves
to eat”“sister”“eat”“loves”“My”“to”

l 単語のベクトルを構文構造に従って合成して、句構造のベクトルを再帰的に計算する
l 最終的に、文全体を表すベクトルが得られる

実際の学習の様子 “My sister loves to eat”
Phrase Category
Classifier
Span Existence
Discriminator“sister”“eat”
Word Category
Classifier“loves”“My”“to”

l 合成した結果ベクトルがそれぞれの句に正しく分類されるよう、逆伝播で最初の単語ベクトルを最適化
l 最終的には、文SとなればよいTransformer(深層学習)から始める場合
l 各単語の入力埋め込みとしてRoBERTaを使用

実験結果
l CCGBankというWall Street Journalのデータで評価
l 現在、ヒューリスティックな方法を抜いて世界最高性能

実際のベクトルの合成の様子

l 768次元のベクトル空間をPCAで3次元に可視化

文法に即した穴埋め
l 「句」のベクトルから、ランダムに単語に戻すことが可能 à 文法に準拠した穴埋めが可能に

まとめ
l 言語の意味と文法に関する、講演者の最新の研究を一つずつ紹介l 適切な統計モデルを立てることで、従来は自動的にはできないと思われていた分析が可能になる
l 計算機でモデル化することで、網羅的なモデル化が可能
à どんな言葉や文に、どんな特徴があるのかをデータから抽出することができる
l 言語学と自然言語処理の境目は、どんどん縮まりつつある統計的テキストモデル (教科書)l 岩波書店より、2024年頃に発売予定
l 言語の統計モデル化の基礎について、丁寧に説明
l 類書と異なり、パッケージに頼らず、数学的背景もその場で説明しています
l http://chasen.org/~daiti-m/
textmodel/
で原稿の一部を公開中

34/44