Meta-Llama-3-8Bは英語Universal Dependenciesの係り受けリンクをFew-Shot Promptingで生成できるか

Ercong Nie, Shuzhou Yuan, Bolei Ma, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze『[Decomposed Prompting: Unveiling Multilingual Linguistic Structure Knowledge in English-Centric Large Language Models](https://arxiv.org/abs/2402.18397)』を横目に、[Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)を使って、英語Universal Dependenciesの係り受けリンクでDecomposed Promptingを試してみた。Google Colaboratory (GPU版)だと、こんな感じ。

```
!pip install -U transformers accelerate sentencepiece
ex="""
###text: And if you send me a story, that would be great to!
In Universal Dependencies of the text, ‘if’ is linked from ‘send’ with the label of mark.
###text: I'll admit I wasn't expecting much from this place, but they really did do a good job.
In Universal Dependencies of the text, ‘good’ is linked from ‘job’ with the label of amod.
###text: "I don't know." The girl shrugged once again.
In Universal Dependencies of the text, ‘girl’ is linked from ‘shrugged’ with the label of nsubj.
###text: The dancers were falling back around a Polish agriculturalist who was teaching a gangling Englishman and two young Africans an Eastern European peasant dance.
In Universal Dependencies of the text, ‘around’ is linked from ‘agriculturalist’ with the label of case.
###text: Antigua was awesome.
In Universal Dependencies of the text, ‘was’ is linked from ‘awesome’ with the label of cop.
###text: The food is fresh and taste great.
In Universal Dependencies of the text, ‘The’ is linked from ‘food’ with the label of det.
###text: Now I have wife and son.
In Universal Dependencies of the text, ‘Now’ is linked from ‘have’ with the label of advmod.
###text: However, this fruitful period was short-lived, as Greece suffered badly under the Ottoman Empire, only to recover in the 19th century as the capital of independent Greece.
In Universal Dependencies of the text, ‘suffered’ is linked from ‘lived’ with the label of advcl.
###text: I survived it without a problem.
In Universal Dependencies of the text, ‘.’ is linked from ‘survived’ with the label of punct.
###text: The food is fresh and taste great.
In Universal Dependencies of the text, ‘and’ is linked from ‘taste’ with the label of cc.
###text: you can view at dresscod.com
In Universal Dependencies of the text, ‘dresscod.com’ is linked from ‘view’ with the label of obl.
###text: "I don't know." The girl shrugged once again.
In Universal Dependencies of the text, ‘shrugged’ is linked from ‘shrugged’ with the label of root.
###text: I'll admit I wasn't expecting much from this place, but they really did do a good job.
In Universal Dependencies of the text, ‘n't’ is linked from ‘expecting’ with the label of advmod.
###text: Antigua was awesome.
In Universal Dependencies of the text, ‘Antigua’ is linked from ‘awesome’ with the label of nsubj.
###text: The dancers were falling back around a Polish agriculturalist who was teaching a gangling Englishman and two young Africans an Eastern European peasant dance.
In Universal Dependencies of the text, ‘two’ is linked from ‘Africans’ with the label of nummod.
###text: Yes, the Cyclone is almost certain to lose strength as it surges over land.
In Universal Dependencies of the text, ‘Yes’ is linked from ‘certain’ with the label of discourse.
###text: —-== Posted via Newsfeed.Com - Unlimited - Uncensored - Secure Usenet News ==—-
In Universal Dependencies of the text, ‘—-==’ is linked from ‘Posted’ with the label of punct.
"""
from transformers import pipeline
tgn=pipeline("text-generation","meta-llama/Meta-Llama-3-8B",max_new_tokens=128,torch_dtype="auto",device_map="auto",token="hf_this_token_is_dummy_use_your_own_one")
nlp=lambda x,y=None:"\n".join(tgn(ex+f"###text: {x}\nIn Universal Dependencies of the text"+(f", ‘{y}’" if y else ","))[0]["generated_text"][len(ex):].split("\n")[0:2])
print(nlp("I have a pen.","I"))
print(nlp("I have a pen.","have"))
print(nlp("I have a pen.","a"))
print(nlp("I have a pen.","pen"))
print(nlp("I have a pen.","."))
```

「I have a pen.」を係り受け解析してみたところ、私(安岡孝一)の手元では以下の結果が出力された。

> ###text: I have a pen.
> In Universal Dependencies of the text, ‘I’ is linked from ‘have’ with the label of nsubj.
> ###text: I have a pen.
> In Universal Dependencies of the text, ‘have’ is linked from ‘have’ with the label of root.
> ###text: I have a pen.
> In Universal Dependencies of the text, ‘a’ is linked from ‘pen’ with the label of det.
> ###text: I have a pen.
> In Universal Dependencies of the text, ‘pen’ is linked from ‘have’ with the label of obj.
> ###text: I have a pen.
> In Universal Dependencies of the text, ‘.’ is linked from ‘have’ with the label of punct.

完璧だ。素晴らしい。じゃあ、続けて、もう少し難しい英文に挑戦してみよう。

```
print(nlp("Who did you wait for?","Who"))
print(nlp("Who did you wait for?","did"))
print(nlp("Who did you wait for?","you"))
print(nlp("Who did you wait for?","wait"))
print(nlp("Who did you wait for?","for"))
print(nlp("Who did you wait for?","?"))
```

「[Who did you wait for?](http://kanji.zinbun.kyoto-u.ac.jp/~yasuoka/publications/2024-03-31.pdf#page=17)」を係り受け解析してみたところ、私の手元では以下の結果が出力された。

> ###text: Who did you wait for?
> In Universal Dependencies of the text, ‘Who’ is linked from ‘wait’ with the label of nsubj.
> ###text: Who did you wait for?
> In Universal Dependencies of the text, ‘did’ is linked from ‘wait’ with the label of aux.
> ###text: Who did you wait for?
> In Universal Dependencies of the text, ‘you’ is linked from ‘wait’ with the label of nsubj.
> ###text: Who did you wait for?
> In Universal Dependencies of the text, ‘wait’ is linked from ‘wait’ with the label of root.
> ###text: Who did you wait for?
> In Universal Dependencies of the text, ‘for’ is linked from ‘wait’ with the label of mark.
> ###text: Who did you wait for?
> In Universal Dependencies of the text, ‘?’ is linked from ‘wait’ with the label of punct.

うーむ、残念、「Who」と「for」が間違っているところを見ると、英語の構文を理解できてるわけでは無さそうだ。複雑な構文を係り受け解析するのは、まだまだ無理かなあ。