On the Transferability of Visually Grounded PCFGs
Authors: Yanpeng Zhao
Ivan Titov
Venue: EMNLP 2023 Findings (2023)
Keywords: unsupervised grammar induction
grounded language learning
syntactic parsing
transfer learning
TL;DR: We propose a method that relies on pre-trained word emebddings to enable visually grounded PCFGs to transfer across text domains.

Abstract: There has been a significant surge of interest in visually grounded grammar induction in recent times. While a variety of models have been developed for the task and have demonstrated impressive performance, they have not been evaluated on text domains that are different from the training domain, so it is unclear if the improvements brought by visual groundings are transferable. Our study aims to fill this gap and assess the degree of transferability. We start by extending VC-PCFG (short for Visually-grounded Compound PCFG [[Zhao and Titov, 2020](https://aclanthology.org/2020.emnlp-main.354/)]) in such a way that it can transfer across text domains. We consider a zero-shot transfer learning setting where a model is trained on the source domain and is directly applied to target domains, without any further training. Our experimental results suggest that: the benefits from using visual groundings transfer to text in a domain similar to the training domain but fail to transfer to remote domains. Further, we conduct data and result analysis; we find that the lexicon overlap between the source domain and the target domain is the most important factor in the transferability of VC-PCFG.

On the Transferability of Visually Grounded PCFGs
Yanpeng Zhao ILLC, University of Edinburgh yanp. zhao@ed. ac. uk
Ivan Titov
"ILLC, University of Amsterdam ititoveinf. ed. sed.ac.. ac.uk
Abstract
in-domain text; unclear if the improvements transfer across domains or, equivalently, if the mod- els acquire a general grammar or only a grammar suitable for a certain domain.
There has been significant surge interest in visually grounded grammar induction in recent times. While a variety of models have been de- veloped for the task and have demonstrated im- pressive performance, they have not been eval- uated on text domains that are different from the training domain, provements brought by visual groundings are transferable. Our study aims to fill this gap and assess the degree of transferability. We start by extending vc-PCFG (short for Visually- grounded Compound PCFG (Zhao and Titov, 2020)) in such way that it can transfer across text domains. We consider a zero-shot transfer learning setting where a model trained on the source domain and is directly applied to tar- get domains, without any further training. Our experimental results suggest that: the benefits from using visual groundings transfer text a domain similar to the training domain but fail to transfer remote domains. Further, conduct data and result amalysis; we find that the lexicon overlap between the source domain and the target domain is the most important factor in the transferability of vC-PCFG.
In this work, we bridge the gap by studying the transferability of vc-PCFG, performant visually grounded PCFG model (Zhao and Titov, 2020). To enable vc-PCFG to transfer across different domains, we extend by using pre-trained word embeddings and obtain transferrable vc-PCFG (model dubbed Tvc-PCFG). This modification allows for directly applying vc-PCFG to tar- get domain, without requiring training any data from the target domain.
unclear if the im-
We learn Tvc-PCFG on the MSCOCO image- caption pairs (Lin et al., 2014; Chen et al., 2015) and consider two evaluation setups: (1) proximate-domain transfer evaluates the model on the captions from another image-captioning data set; and (2) remote-domand transfer evaluates the model on sentences from 14 non-caption text do- mains, ranging from news, biography, and fiction to informal web text. Our experiments show that vc-PCFG can transfer to similar text domains but struggle on text domains that are different from the training domain.
1 Introduction
Research in unsupervised grammar induction has long focused learning grammar models from pure text (Lari and Young. 1990; Klein and Man- ning, 2002; Jiang et al., 2016; Kim et al., 2019). Observing that human language learning largely grounded perceptual experiences, a new trend in the area has been inducing grammars of language with visual groundings (Shi et al., 2019; Zhao and Titov, 2020; Hong et al., 2021; Zhang et al., 2021). These grounded models typically trained par- allel images and text such image captioning data. Once trained, they can be directly used to parse text without access to the aligned images. Though these studies have concluded that visual grounding is beneficial, they evaluated their models only on
To investigate the factors that influence the trans- ferability of Tyc-PCFG, we further conduct an analysis the source- and target-domain data and the parsing results. We propose to measure the distance between the source domain and the target domain via the overlaps of phrasal labels, grammar rules, and words. Our findings suggest that lexicon overlap plays important role in the transferabil- ity of Tvc-PCFG. For example, when tested on Wall Street Journal data (Mareus al., 1999), we observe that the fewer words from a test sentence are present in the MSCOCO data, the lower the performance tends to be.

---

2 Transferable vc-PCFG
To enable vc-PCFG transfer across domains, we extend by using pre-trained word embeddings and sharing them between the source domain and the target domain. Following the definition by Zhao and Titov (2020), a vc-PCFG consists of three types of grammar rules:
2.1 Visually-grounded Compound PCFG
To motivate transfer learning of vc-PCFG, we first reiterate the learning objective of vc-PCFG. Sup- pose a captioning data set D {(v(0) (()] < S M) consists of N pairs of image and cap- tion w, the loss function of vc-PCFG consists of a language modeling loss defined on raw text
Start Rule: S A,
(2)
Binary Rule: A -> BC;
(3)
log p( ()) and a contrastive learning loss defined on image-text pairs svv, w). Formally,
Preterminal Rule: T - w
(4)
The start rules and the binary rules are domain- independent because they composed of domain- agnostic grammar symbols (e.g., S and A), but the preterminal rules, which generate a word condition- ing on a preterminal, are domain-dependent since they rely on the domain-specific vocabulary (i.e., w). Thus, the key to transferring vc-PCFG from the source domain to the target domain is to share preterminal rules equivalently, vocabulary be- tween the source and target domains.
C Σ - logp(w s(y(), w)), (I)
where the hyperparameter a controls the relative importance of the two loss terms.
The language modeling loss from each sentence is defined as the negative log probability of the sentence. As with PCFG G the sentence prob- ability p(w) is formulated as the marginalization of all parse trees (w) that yield w: p(w) EteTg(un) p(4). can be tractably computed using the inside algorithm (Baker, 1979).
Still, sharing the same set of grammar rules between the source and target domains does not guarantee that a learned model transfers to un- seen preterminal rules. This is because the target- domain vocabulary is not necessarily subsumed by the source-domain vocabulary. To make it more clear, we first note that vc-PCFG generates rule probabilities conditioning on grammar symbols. Take preterminal rules of the form T w.
The contrastive learning loss s(v, w) is defined as hinge loss. Intuitively, it is optimized to score higher for an aligned pair (v()) ()) than for any un-aligned pair (i.e., (v(), w)) or (v()), w()) with # j) by positive margin. Since su,, w) is computed with respect to (t w)), the conditional probability distribution of the parse trees of the sen- tence w under the PCFG G. optimizing s(u, w) will backpropagate the learning signals derived from image-text pairs to the parser G (see Zhao and Titov (2020) for technical details).
P(T + w) x g(ur, ew, 8)
(5)
where go neural network, u and e indicate preterminal-symbol embeddings and word embed- dings, respectively, and z is a centence-depender latent vector. Since we train vC-PCF G only on the source domain, for preterminal rules that contain words outside of the source domain, their rule prob- abilities and, specifically, the word embeddings that are used to compute the rule probabilities, will never be learned.
Essentially, the loss function defined in Equa- tion I corresponds to multi-objective learning and can be applied text alone or image-text pairs. Specifically, for sentences that are paired with aligned images, both the LM loss and the hinge loss are minimized; for sentences without aligned images, only the LM loss minimized. By treating images as the labels of the aligned text, this type of multi-objective learning can be seen as semi- supervised learning.
To resolve this issue, we use pre-trained word embeddings, namely GloVe (Pennington et al., 2014), and refer to the resulting model as Tvc-PCFG. Pre-trained word embeddings have encoded similarities among words, i.e., similar words are generally closer in the learned vector space. We keep pre-trained word embeddings frozen during training: thus, test time, for words (preterminal rules) that are unseen during training, our Tvc-PCFG can exploit similarities in the em- bedding space to estimate rule probabilities.
2.2 Transfer Learning Model
In this work, we consider zero-shot transfer learn- ing setting: we directly apply and transfer a pre- trained vc-PCFG to the target domain. This set- ting viable because vc-PCFG can be learned solely on text and does not rely on images to parse text at inference time.

---

Model
NP
VP
SBAR
ADJP
ADVP
C-F1
S-F1
Left Branching
33.2
0.0
0.0
0.0
15.1
15.7
Right Branching
23.8
91.5
63.0
96.0
18.3
76.7
42.4
42.8
Random Trees
32.8
18.440.4
24.4
17.7+1.7
26.8-2.6
20.9
24.2
24.6=0.2
c-PCFG T
43.0+s.6
85.0+2.6
78.4
90.6
36.6-21
87.4
53.6+4.7
53.7-4.6
vc-PCFG 1
14.9
83.213.9
80.9
89.0
38.8_25
86.3
59.3
59.4=8.3
TC-PCFG
31.8
60.0125.5
54.5
73.0
39.3_23.0
59.5
38.742.6
38.8_2.6
PERM
26.0+2.8
36.5+12.9
36.042.3
32.6+10.9
25.0-1.8
31.2
27.0+2.0
27.5-2.1
Tvc-PCFG 79.1
67.8
71.48.55
80.7
59.1_17.9
84.9
65.7
66.3
PERM
42.012.7
34.215.2
41.411.
33.224.4
25.4_2.3
35.9
35.2+1.0
36.0_1.1
Table 1:
Parsing performance on MSCOCO. indicates the results from Zhao and Titov (2020) and indicates models with pre-trained GloVe word embeddings.
Model
NP
VP
SBAR
ADJP
ADVP
C-F1
S-F1
Left Branching
32.9
0.0
0.3
0.5
0.5
0.0
14.4
16.4
Right Branching
27.9
88.0
56.0
92.5
13.3
66.9
44.3
48.0
Random Trees
30.6g0.2
17.440.5
21.940.6
15.8
25.5
19.6_6.1
22.0go.3
24.2
c-PCFG
35.6
64.649.8
63.3
55. +33.0
10.2+44.4
58.6-36.2
43 43.0
45.8g17.3
vc-PCFG
33.7
61.618.2
46.8
40.6
12.7
39.2
38.0g15.4
40.9215.4
TC-PCFG
29.6115.5
18.3
58.0
66.719.1
38.6
55,8
38.5
40.5
PERM
22.313.2
35.419.0
32.312.0
27.644.4
20.5
28.5
24.4
27.321.6
TCC-PCFG
76.3
64.8
72.745.6
69. 1:.1
55.1
70.0
a2.0
66.622.3
PERM
37.4+2.9
32.8+4.6
37.2+1.6
30.0+2.6
24.842.0
34.5=2.4
31.5
35.5+114
Table 2: Parsing performance on Flick. indicates the results obtained by running (V)c-PCFG Flick and indicates the best models (w/ pre-trained GloVe word embeddings) trained on MSCOCO but evaluated on Flickr.
3 Experiments
MSCOCO captions (Chen et al., 2015) are inspired by those of Flickr (Hodosh al., 2013; Young et al., 2014), due the differences in the instruc- tions, the statistics of the collected captions tend be different, c.g., Flickr test captions are slightly longer than MSCOCO training captions (i.e., 12.4 vs 10.5 tokens on average). Nevertheless, Flickr is close to MSCOCO and thus we choose Flickr cap- tions for proximate-domute transfer. Since Flickr does not contain gold phrase structures of captions, follow Zhao and Titov (2020) and parse all the captions with Benepar (Kitaev and Klein, 2018).
3.1 Data Sets and Evaluation
We use MSCOCO captioning data set (Lin et al., 2014; Chen et al., 2015) as the source domain and conduct proximate-domain transfer and remote-domain transfer experiments. For proximate-domain transfer, we consider Flickr30k (Flickr; Young et (2014)). For remote-domain transfer, we consider Wall Street Journal (WSJ) and Brown portions of the Penn Treebank (Mar- cus et al., 1999), and the English Web Treebank (Enweb: Bies et al. (2012)). Note that Brown and Enweb consist of 8 and 5 subdomains, respectively, will be actually performing remote-domain transfer 14 text domains (see below for details).
WSJ is news corpus and the central part of the Penn Treebank resource (Marcus et al., 1999). Sen- tences in WSJ have been manually annotated with phrase structures. We use WSJ for demate-domain transfer; sentences in newswire and image captions are very different as evident, for example, from the divergences in distributions of tokens, syntac- tic fragments, and sentence lengths (20.4 vs 10.5 tokens on average).
Flickr is an image captioning dataset. While the images of Flickr and MSCOCO are all sourced from Flickr, they focus on different aspects, so do their captions. Though the guidelines for collecting
Flickr images focus on people and animals that perform some actions (Hodosh et al., 2013; Young et al., 2014; Plum- mer et al., 2015) while MSCOCO covers more diverse ob- ject categories (up to 80) and focuses on multiple-object im- ages (Lin et al., 2014; Chen et al., 2015).
Brown is also part of the Penn Treebank resource and consists of manually parsed sentences from

---

Model
NP
VP
PP
SBAR
ADJP
ADVP
C-F1
S-F1
Left Branching
10.4
0.5
5.0
5.3
2.5
8.0
6.0
8.7
Right Branching
24.1
71.5
42.4
68.7
27.7
38.1
36.1
39.5
Random Trees
22.5
12.3=0.3
19.0
9.340.6
24.3
26.9
15.3
18.1
c-PCFG 1
76.7=2.0
40.7=5.5
71.3
53.8
45.9
64.2+2..
53,5+1.4
55.7
L10c-PCFG 1
67.1
31.0 99.8
61.3
45.9
36.7
41.346.00
45.5
4..
TC-PCFG
30.9
23.6=7.3
36.429.0
27.2
24.7.1.6
34.2+1.7
24.4
28.0
PERM
20.7+0.3
18.0=3.3
21.1+1.2
13.842.3
23.0+0.3
26.6+1.6
16.4+0.9
20.0+1.1
TCC-PCFG 48.613.7
24.8
39.4
27.2+1.1
30.2:11.6
40.4+2.0
32.0+1.2
35.3
PERM
23.3
1.. 1.9
23.6
13.6.0..6
23.5
28.612.6
17.7
21.3
Parsing performance on WSJ. indicates the models that are trained and evaluated on WSJ (Zhao and Titov, 2021). The prefix 1 indicates that the models are trained on WSJ sentences shorter than 11 tokens but are tested on the full WSJ test set. indicates the best models (w/ pre-trained GloVe word embecdingg) trained MSCOCO but evaluated on WSJ.
Table 3:
8 domains, which cover various genres such as lore, biography, fiction, and humor (Marcus al., 1999). We divide the sentences in each domain into three parts: around 70% of the sentences for training, 15% for development, and 15% for test. We further merge the training, development, and test subsets across domains and create mixed- domain Brown (see Table 7 in Appendix). The average length of Brown test sentences is much longer than MSCOCO training captions, i.e., 17.1 10.5 tokens. Since all these subdomains differ in terms of genre from image captions, we use them for remote-domain transfer.
TVvc-PCFG. To measure model performance, resort to unlabeled corpus-level Fl (C-F1) and sentence-level FI (S-F1), which are equivalent recall in unsupervised grammar induction.
Domain-specific vocabulary. For each corpus, keep the top 10,000 frequent words in the corre- sponding training as the vocabulary. In training and test, tokens outside of the given vocabulary are treated as a special "<unk>" token (short for "un- known"). We share the vocabulary of each mixed domain among its subdomains, e.g., the subdo- mains of Brown shares the vocabulary of the mixed domain Brown, similarly for Enweb.
Enweb is short for English Web Treebank and consists of sentences from domains: we- blogs, newsgroups, email, reviews, and question- answers (Bies al., 2012). Each of these domains contains sentences that have been manually anno- tated with syntactic structures. We divide sentences in each domain in a similar way we divide Brown sentences. We also create mixed-domain Enweb (see Table 8 in Appendix). Enweb test sentences slightly longer than MSCOCO training cap- tions, i.e., 13.9 vs 10.5 tokens average. Since they belong to genres different from image cap- tions, we use them for remote-domain transfer.
test time, we use domain-speciff vocabulary rather than the training- time vocabulary (i.e., the MSCOCO vocabulary). The reasons for doing so include: (I) domain- specific vocabulary is likely to cover more target- domain words than the training-time vocabulary, and (2) this allows for fair comparison because the baseline c-PCFG also uses domain-specific vocabulary.
Test-time vocabulary.
3.3 Main Results
Lexical information is crucial for grammar in- duction despite the high unknown-word ratio. Even with domain-specific vocabulary, we observe high proportions of unknown tokens in Brown and Enweb (i.e., above 30% of total words). Sur- prisingly, Tvc-PCFG still delivers decent transfer learning performance (i.e., above 40% S-F1). This leads us to hypothesize that the parser might not rely on lexical information at all. Instead, it may
3.2 Model Configurations
Transfer learning models. We use the same im- plementations of the text-only parser C-PCFG and the visually-grounded version vc-PCFG Zhao and Titov (2020) but replace their word embed- dings with pre-trained GloVe embeddings (models are dubbed rCVF- We follow the setups in Zhao and Titov (2020) to learn and evaluate

---

Model
NP
VP
PP
SBAR
ADJP
ADVP
C-F1
S-F1
Left Branching
7.9
0.7
3.9
7.0
3.1
15.2
5.2
8,3
Right Branching
24.9
65.0
38.7
58.6
31.6
20.4
37.1
45.3
Random Trees
24.7
15.0+0.2
21.3+00.6
11.7-1.3
22.1
28.9
16.5
21.2.10.2
c-PCFG 75.
31.9216.2
67.2
54.6=3.9
27.7
29.4
47.8
61.3
LI0c-PCFG
63.3+1.8
25.5
53.7
36.2_7.9
28.2
40.243.11
38.3+6.2
42.8+8.a
Te-PCFG
34.7+8.3
28.945.9
38.8+10.5
34.2_3.8
26.3
33.342.6
27.5+1.7
33.741.8
PERM
22.9
22.6433.4
24.1
17.3
21.5
24.7+
18.7
25.4
TVc-PCFG
48.5
29.7
44.8
34.4
32.9
38.1
35.9
41,411.8
PERM
26.620.8
22.1
26.620.5
17.7-0.4
23.221..
28.7
20.2
26.5
Per-domain Performance TVC-PCFG
CF 53.4
25.4
41.4
32.1_4.3
32.646.3
31.043.0
34.0+1.3
37.541.6
CP 64.1
31.9+3.8
49.0
41.1-1.9
34.5+3.2
39.7+7.3
37.7+1.8
44.1
63.5
33.8
49.2+6.7
39.0_2.8
34.3
45.8
38.9
42.941.3
CM
61.4
36.44229
49.645.5
39.5
44.8
50.0+10.2
40.0+1.0
46.3+1.3
CG
53.644.8
25.8+3.1
39.1
31.0-1.8
26.3+3..
28.6+2.7
32,6+1.8
37.1+2.
CR 52.0+3.3
24.042.9
39.026.7
25.9_3.2
26.823.0
34.547.4
31.8+0.8
35.0+1.4
CK 61.0+4.8
29.7
46.646.3
33.1-2.2
34.0+3.3
35.5
35.9
42,641.
CL
62.243.9
32.621.6
46.426.3
39.7
35.3
40.5
36.721.4
41.3+2.0
Table 4: Parsing performance Brown. indicates the results obtained by running c-PCFG Brown; indicates the best models (w/ pre-trained GloVe word embeddings) that are trained on MSCOCO but evaluated on Brown. The subdomains of Brown are (1) CF: popular lore, (2) CG: belles lettres, biography, memoires, etc., (3) CK: general fiction. (4) CL: mystery and detective fiction, (5) CM: science fiction, (6) CN: adventure and western fiction, (7) CP: romance and love story, and (8) CR: humor.
lexical information is important for grammar induc- tion models. GloVe has been pre-trained diverse To test this hypothesis, we construct a permuta- tion baseline, PERM. For sentences of the same length, we randomly exchange the trees of the sen- tences, inferred by the parser. After the exchange. tree may not correspond to the associated sentence, i.e., the tree can be seen being inferred without using the correct lexical information. the parser does not exploit the lexical information, the perfor- mance of such bascline should be equal to that of the parser; accordingly, our hypothesis would be proven to be true.
simply learn a heuristic composition strategy (e.g., variation the right branching baseline).
text and may not best reflect lexical information relevant to the domain of MSCOCO captions (e.g.- wrong senses and parts speech), so TC-PCFG underperforms c-PCFG.
But visual groundings are specific to a domain and could regularize parser to capture domain- specific lexical information (Zhao and Titov, 2020), so Tvc-PCFG is less prone to the same issue in TC-PCFG; instead, it might be making the best of both visual groundings and pre-trained GloVe, so outperforms vc-PCFG.
In Table 3-5, we see PERM performs far worse than TVvCc-PCFG on all the test domains, dis- proving our hypothesis and demonstratigg that TVC-PCFG indeed relies on the token information.
TVC-PCFG succeeds in proximate-domain transfer. We train TVc-PCFG MSCOCO and evaluate on Flickr without further training (see Table 2). Our transfer learning model achieves the best corpus- and sentence-level FI scores on MSCOCO. When evaluated on Flickr, outper- forms c-PCFG (+20.8% S-F1; see Figure 1), so the improvements brought by visual groundings can transfer to similar text domains.
vC-PCFG benefits from pre-trained GloVe. We run experiments on MSCOCO with pre-trained GloVe word embeddings (see Table 1). When trained on both images and text. TVC-PCFG im- proves over vC-PCFG (+6.9% S-F1). But when trained only on text, TC-PCFG lags far behind c-PCFG, i.e., using pre-trained GloVe leads to reduction in performance.
TVC-PCFG fails in remote-domain transfer. We further evaluate pre-trained Tvc-PCFG on remote-domain text, including WSJ, Brown, and Enweb (see Table 3-5). On the whole, the trans-
We speculate that this is because domain-specifee

---

Model
NP
VP
PP
SBAR
ADJP
ADVP
C-F1
S-F1
Left Branching
9.9
0.9
3.4
10.1
3.9
11.2
5.8
10.9
Right Branching
27.1
66.3
41.6
59.3
30,9
29.8
38.3
45,9
Random Trees
25.1
14.7
21.6
13.0
22.1
32.9
16.8_a.2
23.1
c-PCFG 2..8
25.5
53.5
42.9
32.6
48.
39.7=4.5
43.5
LI0c-PCFG
56.412.2
24.6
33.0
24.144.3
24.2
29.2
31.5_3:2
37.5
TC-PCFG
44.9
28.0+7.0
41.1410.2
34.2+4.2
27.4
38.3
27.6_2.0
34.322.2
PERM
23.9
22.3
25.7
19.2+3.3
24.0+2.1
29.1
19.4=1.8
26.942.3
TVC-PCFG
55.0
28.6+3.6
45.4
34.9
35.1
41.4
34.6_1.5
40.4
PERM
27.640.8
21.9
28.1
20.0
25.5
33.5
20.8_0.3
28.2
Per-domain Performance of Tvc-PCFG
Weblog
4..2
26.8
40.4
31.0
30.8
42.0
32.7-a.8
38.0
Answers
58.943.4
29.3+2.9
50.7
35.1
37.7
43.4269
34.8_1.4
39.0.2.
Email
42.8
26.2+3.2
42.5
32.613.7
32.216.9
35.948.3
33.0
40.1+1.9
Newsgroup
40.9
27.3
41.2+7..
33.741.4
29.245.6
36.9
36,9424 33.7-1.8
Reviews
61.7
31.5
52.646.9
42.5
38.7.6..
42.640.6
37.9=1.3
44.9.1.6
Table 5: Parsing performance Enweb. indicates the results obtained by rumning c-ICCFG Enweb: indicates the best models (w/ pre-trained GloVe word embeddings) that are trained on MSCOCO but evaluated on Enweb.
70
Figure 1: S-F1 numbers on dif- ferent target domains. c-PCFG is trained only on the text data each domain's s training set. Tvc-PCFG is transfer learning model and Te-PCFG is the transfer learning model that trained without using vi- sual groundings. The squares in- dicate the average length of the test sentences of each domain.
C-RCrG
TVC-PCTG
TC-PCrG
60
40
12.4
13.9
10.4
10
0
53.7 66.3
38.8
45
06.6
40.5
55.7
35.3
28.0
11.3
41.4
33.7
43.5
40.4
34,3
MSCOCO
Flickr
WSJ
Brown
Enweb
fer learning model TVC-PCFG underperforms c-PCFG, which is trained individually on the train- ing set of each target domain (see Figure 1). We observe the largest S-F1 gap between TVc-PCFG and c-PCFG on WSJ (-20.4%) and the smallest S-F1 gap Enweb (-3.1%) This may be because differences language register. Both WSJ and Enweb are different from MSCOCO at a lexical level, but Enweb, consisting of web text, contains informal language which is likely to be structurally similar to that of captions.
training captions, to allow for fair comparison, for each target domain, we further train c-PCFG in- dividually on the training sentences of the length below 10.5, the average length of MSCOCO train- ing captions. We dub this model Oc-PCFG. Surprisingly, on WSJ and Brown, though the sentences used for training 1.10c-PCFF are shorter than 10.5 tokens, L LcOcPCFGG surpasses Tvc-PCFG by 12.9% and 1.4% S-F1, respectively (see Figure 2). On Enweb, while Tvc-PCFG beats L. Oc-PCFG (+2.9% S-F1), it does not always out- perform L PCFFCG on every run, despite that the average length of the Enweb sentences used for training PC-PCPFCGFFF is only 5 (cf. 10.5 tokens). Across the remote-domain test sets, we also ob- serve that the longer the sentences are used for train- ing c-PCFG, the better the performance is. For example, c-PCFG always surpasses l Oc-PCFG. Interestingly, without considering the "domain"
Regarding model performance on subdomains, we observe similar trends as we see on mixed domains. Specifically, on the subdomains of both Brown and Enweb, c-PCFG performs best, and TVC-PCFG outperforms TC-PCFG (sce Fig- 10a in Appendix).
Remote-domain training helpful. Since the average lengths of WSJ, Brown, and Enweb train- ing sentences are higher than that of MSCOCO
AAlternatively, we can choose length cutoff that results in a subset that has similar average length to MSCOCO.

---

60
training set and the Flickr test set for all three fac- tors in terms of both type-level and instance-level overlap rates (i.e., above 88% on the type level and above 99% on the instance level), so unsurprisingly, TVC-PCFG transfers to Flickr.
C-PCTG TVC-PCTG
L10C-PCTG
S-FI
30-
19.9
2014
WSJ is least similar to MSCOCO. For the WSJ test set, the type-level overlap rates for grammar rules and tokens are below 50%, though the over- lapped factor types cover the majority of the test data (i.e., 86.7% of rule occurrences and 79.7% of token occurrences). This suggests that WSJ is more diverse than MSCOCO in terms of syntactic struc- tures and lexicons, and accounts for the mediocre transfer learning performance of Tvc-PCFG on WSJ. While Brown and Enweb have slightly larger overlap rates than WSJ (i.e., around 55%). their overlaps with MSCOCO are far smaller than Flickr (55% % versus 88%).
20
17.1
12.5
13.9
10.5
10.5
105
7.10
58.7
15.3
51.3
41.4
42.8
43.5
40.4
0
Brown
Enweb
Figure 2: c-PCFG trained on sentences shorter than 41 tokens, l Oc-PCFFG is trained sentences shorter than 11 tokens, and rvc-ICFFG is our transfer learning model. The circles represent the average length sentences for training each model; the squares indicate the average length of the test sentences each domain.
variable, the improvement of L Oc-PCC over TVc-PCFG becomes larger as the average length of the sentences used for training Oc-PCFG in- creases: -2.9% < +1.4% +12.9% S-F1 with 5.0 6.3 < 7.1 tokens for Enweb, Brown, and WSJ, respectively.
In terms of phrasal-label overlap, over 87% of each remote-domain test set is covered by the MSCOCO training set. This is unsurprising be- cause the parser used for parsing MSCOCO cap- tions is pre-trained on the WSJ of the Penn Tree- bank, and both Brown and Enweb follow the anno- tation specifications of the Penn Treebank.
With regard to model performance on subdo- mains, again, we observe similar trends as we see on mixed domains. Specifically, on the subdomains of both Brown and Enweb, c-PCFG performs best, and Tvc-PCFG underperforms O0-PCFG on the subdomains of Brown but outperforms L. L10c-PCFG on the subdomains of Enweb (see Figure 5).
The larger instance-level overlaps, the better. Our results show that I. Oc-PCFG performs sur- prisingly well (48.2% S-F1) on the WSJ test set while Tvc-PCFG does not (35.3% S-F1). To inves- tigate the reasons, we compare their training sets: MSCOCO and WSJ-L11 (i.e., WSJ training sen- tences shorter than 11 tokens) (sce Figure 4). Since WSJ-L1O consists of only around 6,000 sentences (ef. 413,915 MSCOCO captions), unsurprisingly, the type-level overlap rates for grammar rules and words between WSJ-L10 and the WSJ test set are slightly lower than those between MSCOCO and the WSJ test set (e.g., -0.5% for words), but the overlaps cover higher proportions of rule and word occurrences in the WSJ test set (e.g.. +6.3% for words), possibly explaining the better performance of I. Oc-PCFG (results Brown and Enweb can be found in Figure 11 in Appendix).
3.4 Data Analysis
To investigate the relationship between transferabil- ity and source-target domain similarity, we look into three dimensions of variation (we call them "factors") that we hypothesize to influence transfer learning performance: variation in terms of distri- butions of phrasal labels, grammar rules, and words, For each of these three factors, compute overlap rates between the MSCOCO training set and each target test set. We present two types of overlap rates: type-level and instance-level. The type-level overlap rate is the proportion of all the types of a factor in test set that is covered by the MSCOCO training set, and the instance- overlap rate is the proportion of all the instances of a factor in test set that is covered by the MSCOCO training set (see Figure 3).
The lexicon coverage is the most important fac- tor. In fact, the higher type-level overlap rate for grammar rules (+4%) between MSCOCO and the WSJ test set is largely contributed by the non- lexical rule overlap (see overlap rates broken down by rule types in Figure 4). In terms the lexi- cal rule overlap, WSJ-L10 has a slightly smaller
Flickr is most similar to MSCOCO. In Figure 3, we observe large overlap between the MSCOCO

---

MSCOCC
Flickr
WSJ
Brown
Enweb
Figure 3: Overlap rates be- tween the MSCOCO training set and five target test sets: MSCOCO, Flickr, WSJ, Brown, and Enweb. The bars (or the numbers in white) indicate the instance-level overlap rates of phrasal labels, grammar rules, and words, while the circles in- dicate type-level overlap rates.
100
1O0
95.38
$7.0
$5.0
90.1
Overlap Rate (%)
40
100
100
100
100
96.
99.7
99.2
M6.7
90.9
99.6
99.1
$7.3 10.4
Phrasal Label
Grommar Rule
Word
100
MSCOCO
Figure 4: Overlap rates of the WSJ test set with the MSCOCO training set and with the WSJ- LI0 training set. The bars (or the numbers in white) indicate the instance-level overlap rates, while the circles indicate type- level overlap rates. The hori- zontal lines indicate test perfor- mance, i.c., S-F1 (%).
80
Overlap Rate (56)
64-
40
44.7
484
47
45.2 44.9
0
20
89.5
79.7
27.01
Phrasal Label
Grammar Rule
Word
Non-lexical Rule Lexical Rule
overlap rate (-0.3%) but covers much higher por- tion of lexical rule occurrences (+6.9%). Given the better performance of Oc-PCFG, conclude that lexicon overlap the most important factor in the transfcrability of vc-PCFG on WSJ. Note that a similar finding from Zhao and Titov (2021) suggests that lexical rules, out of all the rule types, have the greatest influence on the performance of un-grounded c-PCFG. We conjecture that the find- ing also applies to vc-PCFG.
conduct error analysis. Specifically, given the in- ferred parse of sentence, count correctly recog- nized gold phrases and unrecognized gold phrases. To correlate with unknown words, we instead per- form the counting for a set of sentences that have the same length and contain the same number unknown tokens, then we compute the ratio of the number of correctly recognized gold phrases to the number of unrecognized gold phrases. But, the number of sentences is not generally evenly distributed across the sentence length. To obtain sufficiently reliable statistics, we divide sentences into buckets, where each bucket contains sentences that fall into a consecutive length range.
To quantify the correlation between lexicon over- lap and the performance of Tvc-PCFG, we further compute Spearman's 's rank correlation coefficient on mixed domains and their subdomains (see Fig- ure 5). We find that (1) S-FI and lexical rule over- lap positively correlated with coefficient of 0.59 (p-value- 0.02); and (2) S-FI and unknown token rate are negatively correlated with coeffi- cient of 0.69 (p-value = 0.004), once again con- firming our conclusion.
Error rate versus the number of unseen tokens. Overall, the ratio decreases as the sentence length increases, i.e., the chance of making mistakes in- creases (see Figure 6). When the sentence length is above 17, the chance of failure becomes higher than that of success. But, as the number of unknown tokens increases, we do not observe similar ratio changes across sentence length buckets. This might be because of insufficient statistics for specific number of unknown tokens.
3.5 Error Analysis
Our analysis so far has suggested that overlap in terms of words and lexical rules critical factors in the transferability of Tvc-PCFG. But due to the domain difference, some of the target-domain words are inevitably classified as unknown words due to their low frequencies. To investigate how this influences parsing performance, we further
To measure the correlation between the ratio and the sentence length, we compute Spearman's Rank correlation coefficient for particular unknown to- ken number. We chose unknown-token numbers 0. I, and because they are covered by all sentence

---

wenriiiee Dowrspal MICOCD Loke N. Ovareo No -UC de Average Seriience largh Unannwn Word fate de unimown Token Rotw TCPPTT BewenoTrwee--
60
40
20-
Boown
CF
CP
CN
CM
0G
CR
CX
CL
knweb
Weblog
Answers
Emah
Newsgroup
Reviows
Figure 5: Transfer learning performance the subdomains of Brown and Enweb. The lexical-rule overlap rates are computed on the instance level. The average sentence lengths are computed on the test set of each domain.
2
7-11
12-16
17-21 22:26 27-31
In the future, would like to explore alterna- tive transfer learning settings. For example, when training sentences from the target domain are avail- able, we can additionally optimize the LM objec- tive on them, apart from optimizing the full objec- tive function on source-domain image-texI pairs. Moreover, we may consider a few-shot transfer learning setting and learn vc-PCFG only few target-domain sentences.
Ratio of Success to Failure
6
Limitations
If Unknown Tokens in A Sentence
We acknowledge these limitations of our work:
Figure 6: The ratio of success to failure for each sen- tence length bucket.
English-specific transfer learning. We have far focused on transferring vc-PCFG across gen- res of text in English, primarily because, for En- glish, there are many human-annotated treebank resources allowing for reliable evaluation, but this also implies that our findings may be limited to English, though our approach is applicable to other languages. Specifically, we can replace pre-trained Glo Ve embeddings with pre-trained multilingual word embeddings (Smith et al., 2017) and achieve cross-lingual transfer.
# "<unk>"
2
p p-value
-0.9170
-0.833
-0.983
0.0005
0.005
0
Table 6: Spearman's 's Rank correlation between the ratio and the sentence length for an unknown-token number.
buckets (see Figure 6). The correlation test results confirm that observation about the negative relation between the ratio and the sentence length is statistically reliable (see Table 6).
Model variance. Both Kim et al. (2019) and Zhao and Titov (2021) have noticed the vari- ance issue of cPCCGG. In our work, we also ob- served high variances, for example, on Flickr and Brown. We conjecture that this an issue of all PCFG-based models (Petrov, 2010), not just the neural extensions used in this work. To have a bet- ter understanding of this issue, we think that more thorough studics on the stability of PCFG-based grammar- innuutiinnuutionn models needed.
4 Conclusion and Future Work
We have presented simple approach that enables Vc-PCFG to transfer to text domains beyond the training domain. Our approach relies on pre-trained word embeddings and does not require training on the target domain. We empirically find that Tvc-PCFG is able to transfer to similar text domains but struggles transfer remote-domain text. Through data and result analysis, we find that the lexicon overlap between the source domain and the target domain has a great effect on the transferability of Tvc-PCFG.
Acknowledgements
Ivan Titov is partially supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053).

---

References
Mitchell Mareus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Treebank- 3. Linguistic Data Consortium, Philadelphia.
J. K. Baker. 1979. Trainable grammars for speech recog- nition. The Journal of the Acoustical Society of Amer- icu, 650$1).513--813.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532- 1552, Doha, Qatar. Association for Computational Linguisties.
Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank. Linguistic Data Con- sortium, Philadelphia, PA.
Xinlei Chen, Hao Fang. Tsung-Yi Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server.
Slav Petrov. 2010. Products of random latent vari- able grammars. In Human Language Technologies: The 2010 Annual Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, pages 19-27, Los Angeles, California. Association for Computational Linguistics.
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as ranking task: Data, models and evaluation metries. J. Artif. Int. Res., 47(1):853-899
Bryan A. Plummer, Liwei Wang. Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrass correspondences for richer image- to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV). pages 2641- 2649.
Yining Hong. Qing Li, Song-Chun Zhu, and Siyuan Huang. 2021. Vigrammar: Grounded grammar in- duction of vision and language. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 1665-1674.
Yong Jiang, Wenjuan Han, and Kewei Tu. 2016. Unsu- pervised neural dependency parsing. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 763-771, Austin, Texas, Association for Computational Linguistics.
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019, Visually grounded neural syntax quisition. In Proceedings of the 57th Annual Meet- ing of the Association for Compulational Linguistics, pages 1842-1861, Florence, Italy. Association for Computational Linguistics.
Yoon Kim, Chris Dyer, and Alexander Rush. 2019, Compound probabilistic context- free grammars for grammar induction. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics. pages 2369-2385, Florence, Italy, Asso- ciation for Computational L Linguistics.
Samuel L. Smith, David H. P. Turban, Steven Hamblin. and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In International Conference on Learning Representations.
Nikita Kitaev and Dan Klein. 2018. Constituency pars- ing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2676-2686, Melbourne, Australia. Association for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics. 2:67-78.
Songyang Zhang, Linfeng Song. Lifeng Jin, Kun Xu. Dong Yu, and Jiebo Luo. 2021. Video-aides unsuper- vised grammar induction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1513-1524, Online. Association for Computational Linguistics.
Dan Klein and Christopher D. Manning, 2002. A gener- ative constituent-come model for improved gram- mar induction. In Proceedings of the 40th Annual Meeting of the Association for Computational Lin- guistics, pages 128-135, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Lari and S.J. Young. 1990. The estimation of stochas- tie context-free grammars using the inside-outside algorithm. Computer Speech and Language, 4(1):35 56.
Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound PCFGs. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4369-4379, Online. As- sociation for Computational Linguisties.
Tsung- Yi Lin. Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014, pages 740-755, Cham. Springer Inter- national Publishing.
Yanpeng Zhao and Ivan Titov, 2021. An empirical study of compound PCFGs. In Proceedings of the Second Workshop on Domain Adaptation for NLP,, pages 166- 171, Kyiv, Ukraine, Association for Computational Linguistics.

---

Abstract
Domain specific Vacah
Training time Vocah
this Appendix, we elaborate on (1) target- domain word embedding selection (Section A); (2) statistics of the Brown and Enweb corpora (Sec- tion B); (3) more experimental results (Section C).
GlaVe Vocab
Glote
Glo Learned
Random
A Word Embedding Selection
To evaluate transfer learning model, could directly apply the learned model the target do- main, which is equivalent to using the training-time vocabulary (i.e., the MSCOCO vocabulary), but due to domain differences, the training-time vocab- ulary may cover fewer target-domain words than the target-domain vocabulary (i.c., domain-specinee vocabulary) and lead to deteriorated performance. Thus, we further study different ways of selecting the embeddings of the words in the target domain.
Target-domain Words
Figure 7: Embedding selection with the Random strat- egy (left) and the Direct strategy (right).
Surprisingly do not observe significant S-FI differences between the four embedding selection methods (see Figure 8). To gain a thorough un- derstanding of this phenomenon, we specifically investigate and compare the Random and Direct strategies. For the Random strategy, we find that there are only tens of words are assigned random embeddings, i.e., almost all of the words that are covered by the target-domain vocabulary have Glo Ve embedding. Thus, the major difference be- tween the strategies lies in unknown words. Since the "<unk>" token appears in GloVe, the two strategies share the embedding of "<unk>", so we can further narrow down the difference to the number of unknown words.
Direct. We directly use the training-timir timiing-tmmeeee vocabu- lary, i.c., for each word the test set, is treated as an unknown word if it does not appear in the training-time time vocabulary (see Figure 7 for an illus- tration).
Random. We use the vocabulary of the target domain. Each word in the test set treated as an unknown word if it is not covered by the target- domain vocabulary; otherwise, if it does not have an embedding in GloVe, we use random initializa- tion (see Figure for illustration).
To spell out the difference, for each test domain, compute the proportions of unknown words the word (i.e., word type) level and the token level, which are respectively denoted by circles and squares in Figure We can see that Random results in larger proportions unseen words than Direct on all the domains except Flickr, but the token-level proportion differences are very small, e.g., they are less than 0.5% on MSCOCO, Flickr, and Brown. This may explain why the two selection strategies lead to similar parsing performance all the test domains.
Unknown. We use the vocabulary of the target domain. Each word in the test set is treated as an unknown word if it is not covered by the target- domain vocabulary or it does not have embed- ding in GloVe. Unknown differs from Random in that it treats the small set of words that uses random embeddings in Random "<unk>" token.
Standard. We use the vocabulary of the target do- main. For each word in the test set, it is treated as unknown word if it is not covered by the target- domain vocabulary; otherwise, if is not covered by the training-time vocabulary and the GloVe Ve vocabu- lary, we use random initialization." 4 Compared with Random, for the small set of words that use random embeddings in Random, Standard further seeks their embeddings from the learned Tvc-PCFG.
Though Direct tends to perform slightly better than Random (+0.1% mean S-F1) across the test domains, we use Random as the default strategy in this work because it shares the vocabulary with the baseline parser (i.e., c-PCFG) trained solely the target-domain text and allows for fair comparison.
'A target-domain word may be covered by the training- time vocabulary but is not covered by the GloVe vocabulary. If so, we use the learned embedding from the transfer learning model, i.c., in Stondard, we always try to find a pre-trained word emhedding first.
B Data Statistics
We present statistics of Brown and Enweb in Ta- ble 7 and 8, respectively.

---

Standlard
Uninow
Rondom
0
MSCOCO
Flickr
WSJ
Brown
Enweb
Figure S-F1 numbers with different ways of selecting the embeddings target-domain words.
70
Figure S-FI numbers with different ways of select- ing the embeddings of target- domain words. The circles and squares indicate the propor- tions of words and tokens the test that are treated "<unk>", respectively (see Ap- pendix the Random and Direct strategies and Section 3.2 for the PERM baseline).
Random
Direct
Perm
60
40
16-6
S.FI
368.9
S0 S
30
26.4
5.10
10.3
9:90
BB
IB
8:70
0.00
0.50
1.00
0
MSCOCO
Flickr
WSJ
Brown
Enweb
C Experimental Results
We present the performance of L. 10c-PCFG and TC-PCFG on subdomains of Brown and Enweb in Table 9 and 10, respectively.
Figure 10 visualizes the performance of c-PCFG, L10c-PCFG, Tvc-PCFG, and TC-PCFG on all the test domains.
Figure 11 plots overlap rates of each remote- domain test set with the corresponding remote- domain training sentences (shorter than Il tokens) and with the MSCOCO training set.
Figure 12 visualizes the ratios of success to fail- ure on WSJ, Brown, and Enweb.

---

Split
CF
CG
CK
CL
CM
CN
CP
CR
All (Brown)
train
2191
2324
2708
2745
615
3267
2801
648
17299
dev
507
461
570
518
115
599
543
164
3477
test
494
451
151
549
598
155
3467
File ID Splits
train 1-22
1-25
1-19
1-18
1-21
1-20
1-6
dev
23-27
26-31
20-23
19-21
5-5
22-25
21-25
7-7
lest
28-32
32-36
24-29
22-24
6-6
26-29
26-29
8-9
Table 7: Nine subdomains the Brown corpus of Penn Treebank (Marcus et al., 1999). CF popular lore. CG:
belles lettres. biography, memoires, etc. CK: general fiction. CL: mystery and detective fiction. CM: science fiction. CN: adventure and western fiction. CP: romance and love story. CR: humor.
Split
Answers
Email
Newsgroup
Reviews
Weblog
All (Enweb)
train
2353
3362
1648
2565
1451
11379
dev
565
767
368
622
245
2567
test
569
759
371
626
334
2659
Table 8: Five subdomains of the English Web Treebank (Bies et al., 2012)
90
OACRO TOOPCRR TOR
30
MSC0C0
Plick
WSI
Baown
Eqwebb
Weblog
Answess
Enoil
Nuw Restews
CP
CN
CM
(a) Comparison between c-PCFG, TVC-PCFG, TC-PCFG all target domains.
GRRCF PORAC
30-
30
MSCOCO
Flisk
WSI
Baokb
Eawcb
Webluz
Aunwnn
Eunal
Menogauu Revizwa
CF
C2
CN
CM
C0
C3
Ck
(b) Comparison between cCCCG Tvc-PCFG, and 10c-PCFG on all target domains.
Figure 10: c-PCFG trained sentences shorter than 41 tokens, L L10c-PCFGis is trained sentences shorter than 11 tokens, Tvc-PCFG is our transfer learning model, and TC-PCFG is the transfer learning model that is trained without using visual groundings. squares indicate average length of the test sentences of each domain.

---

Model
NP
VP
PP
SBAR
ADJP
ADVP
C-F1
S-F1
c-PCFG
75.0
31.
67 68.2
54.643.9
39.7
59,42..6
47.844.4
51.326.
L10c-PCFG
63.3
23.5
53.7
36.2+7.9
28.2+8.9
40.2+3.1
38.3
42.8ga.9
Per-domain Performance of 100-PCFG
CF
58.4
24.0+23.5
49.
31.549.2
31.240.0
31.745.6
37.345.6
39.9
CP 67.4 +7.4
28.2+26.6
58.6
43.948.0
+3.9
43.8+3.7
40.2+7.5
45.9
CN 68.0+2.4
26.3424.0
58.325.1
35.647.2
30.2+10.5
45.3
39.7.6.5
43.5=8.5
CM
71.3
30.142669
66.0219.1
43.2415.1
32.3
28.146.2
44.5+8.1
48.4=10.3
CG
59.1
23.3421.0
47.6+6.3
32.348.1
24.8+8.6
36,644.99
35.945.5
39.7+7.9
CR
58.3+1.9
22.6418.2
48.3
29.245.1
22.3+3.1
42.215.9
35.843.3
39.0=4.1
CK
65.7
25.1
55.416.6
36.216.9
25.2+8.2
44.42..0
38.346.3
43.9-10:2
CL
68.3
26,9425.8
59.4
44.1+8.9
29.8+9.2
40.1+3.5
39.348.3
42.3
TC- PCFG
34.728.3
28.945.9
38.8
34.243.8
26.322,3
33.342.6
27.5+1.
33.7
PERM
22.920.2
22.643.4
24.1
17.3
21.523.0
24.740.8
18.741.
25.4g1.8
Per-domain Performance TC-PCFG
CF 32.6
26.8
38.3
33.3
21.0
29.245.2
26.641.
30.0
CP 38.5
31.946.7
41.1212.2
40.743.8
28.624.3
42.1
29.742.0
36.8
CN 38.3
32.446.3
40.7=11.7
36.345.0
26.9+1.9
44.349.8
29.5+1.
35.0
CM 36.9210.8
34.218.7
44.6213.9
39.516.2
32.3211.8
40.616.2
31.921.7
36.9
CG 32.3
25.1+6.0
35.1
31.044.2
19.1
27.2+3.2
25.0+1.9
30.1
CR 29.3
23.545.0
35.6119.3
29.4
26.8
23.3444.
23.642.
26.9
CK 35.5
28.545.2
39.6211.9
32.244.0
30.7+2.5
28.747.0
27.541.6
35.3
CL 39.4
31.745.9
42.1
38.4+7.7
26.0+7.3
33.7+7.0
29.942.
35.1
Table 9; Parsing performance on Brown. indicates the results obtained by running c-PCFG on Brown; the best models (w/ pre-trained GloVe word embeddings) trained on MSCOCO but evaluated on Brown.
indicates
Model
NP
VP
PP
SBAR
ADJP
ADVP
C-FI
S-F1 43.5_4.9
c-PCFG
62.8
25.5
53.5 33.0
52.942.4
32.645.8
48.5
39.7
.10c-PCFG
56.422.
24.6
24.1
24.2+2.3
29.2
31.513.2
37.5=2.
Per-domain Performance of 10c-PCFG
Weblog 52.6=1.4
20.0=10.1
29.6=1.8
18.6+7.1
13.545.9
31.2=5.1
29.4227
33.5-2.1
Answers
60.0=1.9
27.249.3
39.1
24.2+4.1
29.1
26.2+411
32.1
37.6
Email
54.8
24.449.5
32.0=s.1
26.2.6.3
20.1
17.2
31.0+2.9
38.4=4.4
Newsgroup
53.2+1.9
20.749.7
29.6
20.345.9
15.7
38.1
30.4124
32.9-199
Reviews
62.1=2.4
27.5
35.8=0.2
28.54448
30.1
37.8+1.
34.023.8
41.0=2.9
TC-PCFG
34.9
28.047.0
41.1
34.244.2
27.441.7
38.3
27.61:200
34.3
PERM
23.9
22.3
25.7
19.2+3.3
24.0+2.1
29.1
19.4
26.9
Per-domain Performance TC-PCFG
Weblog 30.626.2
27.4
37.3
34.7
21.245.9
43.8
26.0
31.4=22
Answers
38.5
28.346.6
44.5
34.2+3.2
30.3
39.7
28.5
33.3-2.7
Email
34.215.6
24.547.
39.0=s.5
30.7
26.5
28.9
26.0
33.5-3.6
Newsgroup 31.316.7
25.847.0
39.8
34.04s.1
21.642.8
32.1
29.7-2.6 26.112.0
Reviews
40.7
31.847.7
46.1
39.6
31.441.
43.6
31.042.4
38.4
Table 10: Parsing performance on Enweb.
indicates the results obtained by running c-PCFG Enweb; indicates models (w/ pre-trained GloVe word embeddings) trained on MSCOCO but evaluated on Enweb.

---

100
MSCOCO
WSJ-L10
98.4 .
98.4
80
Overlap Rate (%)
60
64.4
48.7
48 4
47.9
40
44.7
43.5
45.2
44.9
05,, 5-81
20
100
100
86.7
89.5
79.7
86.0
97.4
95.7
77.0
83.9
Phrasal Label
Grammar Rule
Word
Non-lexical Rule Lexical Rule
(a) WSJ Test Set.
100
MSCOCO
Brown-L10
93.8
80
Overlap Rate (%)
60
55.5
52.3
$1.5
40
41A.. 5-R1
38.7
34.9
35.9
33.9
20
100
100
90.9
88.0
87.3
82.7
96.9
95.4
85.1
80.9
Phrasal Label
Grammar Rule
Word
Non-lexical Rule Lexical Rule
(b) Brown Test Set.
100
MSCOCO
Enweb-L10
97.1
Overlap Rate (%)
60
57.8
57.0
$4.3
53.7
40
43-0
36.5
37.8
35.0
20
98.8
100
88.2
86.4
85.4
80.2
94.8
95.4
81.8
77.7
Phrasal Label
Grammar Rule
Word
Non-lexical Rule Lexical Rule
(c) Enweb Test Set.
Figure 11: Overlap rates of the WSJ/Brown//nwee test set with the MSCOCO training and with the WSJ-
LIO//rown-L. 10t.nweb-////ee----11 training set.

---

6 7-11
12-16
17-21
22-26
27-31
12
13
17
19
Ratio Success to Failure
Ratio Success Failure
12
10
3
6
2-
4
# Unknown Tokens in A Sentence
# Unknown Tokens in A Sentence
(a) WSJ Test Set.
(b) WSJ Test Set.
10
14
18
2-6
7-11
12-16
17-21
22-26 +
27-31
5
11 10
13
15
17
19
21
4.0
Ratio of Success Faduure
3.5
Ratio of Success to Failure
3.0
2.5
2.0
4
1.5
2
1.0
3
2 # Unknown Tokens A Sentence
Unknown Tokens in A Sentence
(c) Brown Test Set.
(d) Brown Test Set.
7-11
12-16
17-21
27-31
2
12
16
18
20
3
5
13
15
17
19
21
14
Ralio Succoss Failure
4
Ratio Success Failure
12
10
3
8
2
# Unknown Tokens A Sentence
Unknown Tokens in Sentence
(c) Enweh Test Set.
(f) Enweb Test Set.
Figure 12: ratio success to failure each sentence length bucket (left) and for individual sentence lengths (right). The Y axis zero indicates that Tvc-PCFG makes mistakes.


Reviews:
[{"review_id": "lXT7eCltdn", "reviewer": "Reviewer_ZVfX", "summary": "", "questions": "- Table 3, 4, 5: why isn't vc-PCFG listed as a baseline in the remote domains?\n- Line 237: given that WSJ, Brown, and Enweb covers more topics than MSCOCO and Flickr, they might also need larger vocabularies. Is limiting the vocabulary size to 10k fair for these datasets?\n- Line 279: it’s interesting to see tvc-PCFG outperforms vc-PCFG while tc-PCFG underperforms c-PCFG. Any explanation?\n- Table 4: it would be good to have an explanation for each domain abbreviation in Brown dataset.\n- Line 336: it’s claimed that longer average length of sentences used in training improves performance. However, there are two varying factors in the comparison: average length and dataset. It’s not accurate to attribute the improvements totally to the longer sentence lengths.\n- Line 466: is it possible to measure how statistically reliable the observation is?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing a detailed review and insightful comments. We would like to clarify the contribution and novelty:\n\n- **A novel research question:** To the best of our knowledge, we are the first to explore transferability in grammar induction. Our aim is to discern whether a parser truly induces a grammar of the language (i.e., English), or if it merely learns rules or shortcuts specific to a particular domain. This exploration is especially important within the context of grounded learning. Given that groundings can only be obtained for specific domains and certain language expressions, the transferability of the acquired syntactic expertise becomes crucial.\n\n- **A novel model:** Apart from conducting a variety of evaluations and analyses, we present a simple yet effective transfer learning model. Our model also provides an inspiration for developing multilingual transfer learning models for grammar induction as discussed in line 499.\n\n- **Comparison with Zhao & Titov 2021:** Zhao & Titov 2021 only perform testing in-domain. While they do observe new/unseen words present a challenge, they do not show that it is the main challenge preventing generalization to other domains. We systematically study different types of divergencies and quantify their contribution to the drop in the performance.\n\nQ: *Table 3, 4, 5: why isn't vc~PCFG listed as a baseline in the remote domains?*\n\nWSJ, Brown, and Enweb have no gold visual groundings, so we could not train vc~PCFG on them.\n\nQ: *Line 237: ... WSJ, Brown, and Enweb … Is limiting the vocabulary size to 10k fair for these datasets?*\n\nThe reason for keeping the same vocabulary cutoff of 10k is to make fair comparisons across data sets, i.e., eliminating the vocabulary size variable. But, if the goal is to optimize model performance on each data set, the vocabulary size can be treated as a hyperparameter tuned for each data set individually.\n\nQ: *Line 279: … tvc-PCFG outperforms vc-PCFG while tc-PCFG underperforms c-PCFG.*\n\nWe speculate that domain-specific lexical information is important for grammar induction models. GloVe has been pre-trained on diverse text and may not best reflect lexical information relevant to the domain of MSCOCO captions (e.g., wrong senses and parts of speech), so tc\\~PCFG underperforms c~PCFG. \n\nBut visual groundings are specific to a domain and could regularize a parser to capture domain-specific lexical information [1], so tvc\\~PCFG is less prone to the same issue as in tc\\~PCFG; instead, it might be making the best of both visual groundings and pre-trained GloVe, so it outperforms vc~PCFG.\n\nQ: *Table 4: … to have an explanation for each domain abbreviation in Brown dataset.*\n\nWe described them in Table 6 but will definitely elaborate on them in the data set section in the final version.\n\nQ: *Line 336: … not accurate to attribute the improvements totally to the longer sentence lengths.*\n\nThank you for pointing it out. We agree and will rephrase the statement.\n\nQ: *Line 466: is it possible to measure how statistically reliable the observation is?*\n\nWe observe that the ratio decreases as the sentence length increases, i.e., the chance of making mistakes increases. \n\nTo measure correlation between the ratio and the sentence length, we compute the Spearman's Rank correlation coefficient for a particular unknown token number (see Figure 6). We chose unknown token numbers 0, 1, and 2 because they are covered by all sentence buckets. Here are the results:\n\n| Number of unknown tokens    | Coefficient | P-value |\n| -------- | ------- | ------- | \n| 0 | -0.917    | 0.0005    | \n| 1 | -0.833      | 0.005    | \n| 2 |-0.983     | 0    | \n\n\nThus, the results confirm that our observation is statistically reliable.\n\n[1] Yanpeng Zhao and Ivan Titov. Visually grounded compound PCFGs. In EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The evaluations conducted in the paper have good coverage and a lot of details. One proximate-domain dataset and three remote-domain datasets are included.", "reasons_to_reject": "The main conclusion from this paper, transferability on similar domains is better than remote domains, has been widely observed in other transfer learning tasks. The further analysis suggests the lexicon overlap of the new domain is the most important factor in the transferability, which is consistent with the findings in c-PCFG in the previous work of Zhao & Titov 2021. The findings in the paper are definitely valuable, but, to some extent, are also a bit incremental to me.\n", "questions_to_authors": "- Table 3, 4, 5: why isn't vc-PCFG listed as a baseline in the remote domains?\n- Line 237: given that WSJ, Brown, and Enweb covers more topics than MSCOCO and Flickr, they might also need larger vocabularies. Is limiting the vocabulary size to 10k fair for these datasets?\n- Line 279: it’s interesting to see tvc-PCFG outperforms vc-PCFG while tc-PCFG underperforms c-PCFG. Any explanation?\n- Table 4: it would be good to have an explanation for each domain abbreviation in Brown dataset.\n- Line 336: it’s claimed that longer average length of sentences used in training improves performance. However, there are two varying factors in the comparison: average length and dataset. It’s not accurate to attribute the improvements totally to the longer sentence lengths.\n- Line 466: is it possible to measure how statistically reliable the observation is?", "paper_topic_and_main_contributions": "This paper conducts a thorough empirical evaluation of the transferability of visually grounded PCFGs. To address the mismatching vocabulary issue in the transferred domain, pretrained word embeddings are used in the model’s input layer. Evaluations are performed on both proximate-domain (Flickr30k) dataset, and remote-domain datasets (WSJ, Brown, and Enweb), which suggest the benefits from using visual grounding transfer to similar domains but fail to transfer to remote domains. Further analyses reveal that, comparing to other factors like distribution of phrasal labels and grammar rules, the distribution of words/lexicons has more impact in the transferability.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tEgLCpWfbj", "reviewer": "Reviewer_jC7f", "summary": "", "questions": "(A) Why do vocabulary cutting and unks (line 236) when you’re using the GloVe embeddings? Wouldn’t it be simpler to just restrict the vocabulary to all GloVe words?\n\n(B) Line 280 you say that using pretrained GloVe leads to a reduction in performance — isn’t it also the case that tc~PCFG is transfer learning while c~PCFG is tested in-domain? \n\n(D) Line 511 is it possible to use gradient accumulation to make the batch size artificially bigger (sorry, I’m not familiar with the mechanics of PCFG training and this might be a bad question!)", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful reading and insightful suggestions.\n\nQ: *Why do vocabulary cutting and unks  (line 236) …?*\n\nWe cut vocabulary for fair comparison (explained in line 251). Note that c\\~PCFG uses a vocabulary cutoff, i.e., the top 10,000 frequent words in the training set. Such a strategy is used mainly for better training/test efficiency. To make a fair comparison with c\\~PCFG, we use the same vocabulary for t(v)c\\~PCFG at test time rather than using the MSCOCO vocabulary, but our embeddings come from GloVe.\n\nQ: *Line 280 … isn’t it also the case that tc\\~PCFG is transfer learning while c\\~PCFG is tested in-domain?*\n\nYes. tc\\~PCFG is a transfer learning model and c\\~PCFG is the original model. The difference is that tc~PCFG uses GloVe. \n\nLine 280 discusses model performance on MSCOCO. Both tc\\~PCFG and c\\~PCFG are trained on MSCOCO without using visual groundings and are tested on the MSCOCO test set. Thus, this is actually not an evaluation of transferability; instead, we wanted to see if GloVe improves c~PCFG.\n\nRegarding why GloVe hurts the performance of c\\~PCFG. We speculate that domain-specific lexical information is important for grammar induction models. GloVe has been pre-trained on diverse text and may not best reflect lexical information relevant to the domain of MSCOCO captions (e.g., wrong senses and parts of speech), so tc\\~PCFG underperforms c~PCFG.\n\nQ: *Line 511 … use gradient accumulation to make the batch size artificially bigger…*\n\nWe actually tried gradient accumulation. While a larger batch size makes training more stable, it does not bring significant performance improvements and instead makes learning inefficient, e.g., we will have to always keep a copy of gradients during training. Thus, we kept using a batch size of 5.\n\nThank you again for your suggestions on presentation. We will incorporate them in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has a solid set of experiments with good baselines that cover many datasets, and further experiments in Data Analysis and Error Analysis that dive into trying to understand some of the initial results. I also think that this paper introduced a neat idea to PCFG parsing: the idea of using pretrained embeddings to not have to worry about vocabulary transfer and be able to check the syntactic viability of transfer.", "reasons_to_reject": "No clear reasons to reject. ", "questions_to_authors": "(A) Why do vocabulary cutting and unks (line 236) when you’re using the GloVe embeddings? Wouldn’t it be simpler to just restrict the vocabulary to all GloVe words?\n\n(B) Line 280 you say that using pretrained GloVe leads to a reduction in performance — isn’t it also the case that tc~PCFG is transfer learning while c~PCFG is tested in-domain? \n\n(D) Line 511 is it possible to use gradient accumulation to make the batch size artificially bigger (sorry, I’m not familiar with the mechanics of PCFG training and this might be a bad question!)", "paper_topic_and_main_contributions": "The paper suggests a way of doing domain transfer of visually-grounded PCFGs by using pretrained GloVe vocabulary embeddings to get the terminal rules. This way, the trained model is not specific to a limited vocabulary, and can use the embedding space to get semantic components of unseen words in new domains. The results show that visual grounding does help in the transfer scenario (compared to no visual grounding) but that transfer is not that good for remote domains. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 3: Is the 3rd and 4th row (the c~PCFG) from training on WSJ? I would perhaps make this clearer \n\nLine 259: The PERM baseline and the discussion around lexicalization might merit its own subection or paragraph, it’s a bit odd under Test-time vocabulary. I would make it a separate paragraph heading starting at line 254\n\nIt might be better to introduce what S-F1 and C-F1 mean a bit earlier, and they could maybe also be the first columns of the table rather than the last. \n\nFigure 5: there’s a lot of information on this figure, it might be nice to make it a bit sparser. Do all the points/bars use the same y-axis? \n\nSome minor suggestions and typos:\n\n Line 52: It’s unclear where the T comes from and why it stands for pretrained word embeddings. It might be clearer to make that explicit.\n\nLine 87: It might be nice to have a bit of the explanation before the function. As it stands the reader sees a function where every variable is unknown and then has to read the following paragraph to understand\n\nLine 168: double parentheses, could use \\citet instead"}}, {"review_id": "MQUeIO1b0E", "reviewer": "Reviewer_qPWB", "summary": "", "questions": "a) I don’t quite understand the PERM baseline, why would its performance indicate whether the model exploits lexical information or not?\n\nb) In Table 1, do you have a hypothesis on why using pre-trained Glove leads to much worse performance when the model is only trained on text data of MSCOCO? This seems counter-intuitive.\n\nc) In Table 2, why adding visual grounding to the in-domain Flick30k model hurts performance, while adding images from the MSCOCO dataset greatly improves performance in the transfer-learning setting?\n\nd) In the L10 variants, why don’t we set the length cutoff to a large number so that the average length of the subsets is closer to that of the training set? It might make it easier to compare the results.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing a detailed review and insightful comments. We would like to clarify the contribution and novelty:\n\n- **A novel research question:** To the best of our knowledge, we are the first to explore transferability in grammar induction. Our aim is to discern whether a parser truly induces a grammar of the language (i.e., English), or if it merely learns rules or shortcuts specific to a particular domain. This exploration is especially important within the context of grounded learning. Given that groundings can only be obtained for specific domains and certain language expressions, the transferability of the acquired syntactic expertise becomes crucial.\n\n- **A novel model:** Apart from conducting a variety of evaluations and analyses, we present a simple yet effective transfer learning model. Our model also provides an inspiration for developing multilingual transfer learning models for grammar induction as discussed in line 499.\n\nQ: *relying on vc-PCFG*\n\nWe agree that our work relies on technical details of vc\\~PCFG and will expand on vc\\~PCFG in Section 2.1 to make our paper self-contained.\n\nQ: *the PERM baseline*\n\nIn PERM baseline, we randomly exchange the inferred trees for sentences of the same length. After the exchange, a tree may not correspond to the associated sentence, i.e., the tree can be seen as being inferred without using the correct lexical information. Our hypothesis would be proven to be true if the PERM baseline has a similar performance to tvc~PCFG.\n\nWe will revise the paper to make it clear.\n\nQ: *tc\\~PCFG underperforms c~PCFG on MSCOCO*\n\nWe speculate that domain-specific lexical information is important for grammar induction models. GloVe has been pre-trained on diverse text and may not best reflect lexical information relevant to the domain of MSCOCO captions (e.g., wrong senses and parts of speech), so tc\\~PCFG underperforms c~PCFG. \n\nBut visual groundings are specific to a domain and could regularize a parser to capture domain-specific lexical information [1], so tvc\\~PCFG is less prone to the same issue as in tc\\~PCFG; instead, it might be making the best of both visual groundings and pre-trained GloVe, so it outperforms vc~PCFG.\n\nQ: *why adding visual groundings hurts model performance on Flickr30k but is helpful on MSCOCO*\n\nWe do not have a good explanation for this but also the variances across runs with (v)c\\~PCFG are high.\n\nQ: *better L10 variants*\n\nWe agree that a better length cutoff is the one that makes the average length similar to that of the MSCOCO training set, but this method has its own issues: it may incorporate sentence lengths that are not covered by MSCOCO captions, given that MSCOCO captions are generally short; accordingly, further comparison will not be completely fair, either.\n\nMoreover, we use the L10 variant to show that training on the target domain is helpful. Though our L10 variant has a smaller average sentence length, it outperforms tvc\\~PCFG, so it is adequate for supporting our hypothesis. But again, we agree with you that using the better length cutoff would make our experiments more comprehensive. \n\n[1] Yanpeng Zhao and Ivan Titov. Visually grounded compound PCFGs. In EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) This paper studies an interesting problem of cross-domain transfer of visually grounded PCFG models.\n\nb) Very comprehensive experiments on various potential factors for transferability and careful analysis.\n", "reasons_to_reject": "a) This paper focuses on one single aspect of one specific model, transferability of visually grounded PCFGs, which might not be of general interest to the field.\n\nb) This paper is a bit hard to follow, especially for readers not familiar with the original vc-PCFG work. I’ll list my questions and comments below.\n", "questions_to_authors": "a) I don’t quite understand the PERM baseline, why would its performance indicate whether the model exploits lexical information or not?\n\nb) In Table 1, do you have a hypothesis on why using pre-trained Glove leads to much worse performance when the model is only trained on text data of MSCOCO? This seems counter-intuitive.\n\nc) In Table 2, why adding visual grounding to the in-domain Flick30k model hurts performance, while adding images from the MSCOCO dataset greatly improves performance in the transfer-learning setting?\n\nd) In the L10 variants, why don’t we set the length cutoff to a large number so that the average length of the subsets is closer to that of the training set? It might make it easier to compare the results.\n", "paper_topic_and_main_contributions": "This paper investigates whether a visually grounded PCFG (vc-PCFG) model trained on a source domain of image captioning data can be transferred to other text domains. The original vc-PCFG model does not use pretrained word embeddings like Glove, making it hard to transfer across domains due to vocabulary mismatch. Therefore, this paper proposed tvc-PCFG by replacing the word embeddings in the original model with Glove embeddings that are kept frozen during training, and using domain-specific vocabulary at test time.\n\nExperiments on several domains including both image captioning data and text-only data show that the proposed method transfers well to similar text domains but struggles on domains that are different from the training domain. The paper also conducts extensive ablation studies and analysis to show that domain similarity is indeed correlated with transfer performance, and that lexicon coverage is the most important factor in determining transferability.\n\nMain contributions:\n\na) Empirical results showing that vc-PCFG models can achieve good cross-domain performance when using pretrained embeddings.\n\nb) Detailed error analysis showing that lexical discrepancy is the main challenge in transferability of vc-PCFG models.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "a) L252, the term “c-PCFG” is used without definition (not until page 6).\n\nb) L288, the performance improvement should be 20.8% instead of 10.8%?\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts a thorough empirical evaluation of the transferability of visually grounded PCFGs. The experiments are very solid, covering several datasets with detailed analysis. The introduced idea is also intersting. The major drawback is that as an emperical study, the findings in this paper are plain, which makes the contributions of this paper incremental.
Decision: Accept-Findings