The 61st Annual Meeting of the Association for Computational Linguistics

Holographic CCG Parsing
112
Ryosuke Yamaki , Tadahiro Taniguchi , Daichi Mochihashi
1Ritsumeikan University，2The Institute of Statistical Mathematics➢ Span-based Parsing➢ Explicit modeling of phrase structure•

Recent CCG supertagging and parsing models demonstrate highperformance yet rely on non-explicit modeling of dependencies
between words through neural networks.

1. Store word-level representations.
2. Recursively compose phrase-level representations.
3. Directly evaluates category assignment to phrases.

👉Explicit modeling of phrase structure with neural networks.

➢ Syntactic phrase-level representation
👉Compose syntactically rich phrase-level representations while
maintaining training efficiency.

➢ Holographic Embeddings (Nickel et al., 2016)• Embedding knowledge graphs into vector spacefor statistical modeling• Vector composition using circular correlation to capture
dependencies between entities• Similarity of knowledge graphs and phrase structures that need tocapture dependencies between components(Nickel et al., 2016)➢ Holographic CCG (Hol-CCG)••

Formulate CCG as a recursive compositional operation between
distributed representations in a vector space.
Applicable to Supertagging and Span-based Parsing.

• Dataset: CCGbank (Hockenmaier et al., 2007)• Calculate the model's prediction error by cross entropy➢ Category assignment to words and phrases: ℒ𝑤 , ℒ𝑝
➢ Existence of span: ℒ𝑠• Compare models by changing the combination of back-propagatingerrors (ℒ)➢ Baseline: ℒ= ℒ𝑤➢ Hol-CCG: ℒ= ℒ𝑤 + ℒ𝑝 + ℒ𝑠
• Supertagging by Baseline and Hol-CCG• Parsing using C&C Parser (Clark and Curran, 2007) and Hol-CCG’s
span-based parsing• Hol-CCG outperforms baseline.
• Span-based Parsing outperforms C&C Parser.
• Explicit modeling of word/phrase dependencies throughcomposition of phrase representations is effective for bothsupertagging and parsing.
Training Objectivesℒ𝑤 (baseline)➢Model Structure
1. Encode word sequence into distributed representations.
2. Recursively compose phrase-level representations.
3. Predict CCG categories and span existence.

ℒ𝑤 + ℒ𝑝 + ℒ𝑠 (Hol-CCG)

ParserC&C
C&C
SpanAcc
LF
96.41 ± 0.03 91.77 ± 0.03
𝟗𝟔. 𝟓𝟗 ± 𝟎. 𝟎𝟐 𝟗𝟐. 𝟎𝟑 ± 𝟎. 𝟎𝟒
𝟗𝟐. 𝟔𝟏 ± 𝟎. 𝟎𝟑• Hol-CCG achieved SoTA in supertagging accuracy and LF with C&CParser.
• Hol-CCG’s span-based parsing is competitive with current SoTA
performance.
Model
Vaswani et al., 2016
Bhargava and Penn, 2020
Prange et al., 2021
Clark, 2021
Hol-CCG

Parser
C&C
C&C
C&C
C&C
Span
C&C
Span

Acc
94.5
96.00
96.22
96.60
-Nickel, Maximilian, Lorenzo Rosasco, and Tomaso Poggio. "Holographic embeddings of knowledge graphs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 30. No. 1. 2016.
Julia Hockenmaier, and Mark Steedman. "CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn Treebank." Computational Linguistics 33.3 (2007): 355-396.
Stephen Clark, and James R. Curran. "Wide-coverage efficient statistical parsing with CCG and log-linear models." Computational Linguistics 33.4 (2007): 493-552.
Vaswani, Ashish, et al. "Supertagging with lstms." Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.
Bhargava, Aditya, and Gerald Penn. "Supertagging with CCG primitives." Proceedings of the 5th Workshop on Representation Learning for NLP. 2020.
Prange, Jakob, Nathan Schneider, and Vivek Srikumar. "Supertagging the long tail with tree-structured decoding of complex categories." Transactions of the Association for Computational Linguistics 9 (2021): 243-260.
Clark, Stephen. "Something old, something new: Grammar-based CCG parsing with transformer models." arXiv preprint arXiv:2109.10044 (2021).

LF
88.32
90.9
90.91
91.9
92.9
92.12
92.67