Question: What hypotheses do the experimental designs address?
Evidence: Our experimental designs address the following hypotheses:
Hypothesis 1 (H1) : Training complex abstractive models with limited in-domain or large quantities of out-of-domain data won't be enough to outperform extractive baselines.
Hypothesis 2 (H2) : Domain transfer helps abstractive models even if in-domain and out-of-domain data are very different and the amount of in-domain data is very small.
Hypothesis 3 (H3) : Enriching abstractive training data with synthetic data helps overcome in-domain data scarcity.
Hypothesis 4 (H4) : The proposed template-based synthesis model outperforms a simple word replacement model.
Hypothesis 5 (H5) : Combining domain transfer with data synthesis outperforms using each approach on its own.
Hypothesis 6 (H6) : The synthesis model can be extended to perform reflection summarization directly.
Extractive Baselines (for testing H1). While BIBREF0 used Lead-3 as an extractive baseline, in our data sentence order doesn't matter as reflections are independent. We thus use a similar in concept baseline: randomly select N reflections. Since the baseline is random we report the average result of 100 runs. Following BIBREF5, we compare results to MEAD BIBREF20 and to BIBREF5's extractive phrase-based model. Since these models extracted 5 phrases as extractive summary, we use N=5 for our three extractive baselines. Additionally we compare to running only the extractive part of Fast-RL.
Domain Transfer (for testing H2, H5).
To observe the impact of using out-of-domain (news) data for pretraining to compensate for low resource in-domain (reflection) data, we train 3 variants of PG-net: model training on CNN/DM; model training on reflections; and model training on CNN/DM then tuning using reflections. Table TABREF11 shows example summaries generated by the three variants of PG-net for a CS document. For all experiments where reflections are used for training/tuning, we train using a leave one course out approach (i.e, in each fold, three courses are used for training and the remaining course for testing). If the experiment involves tuning a combined dictionary of CNN/DM and reflections is used to avoid domain mismatch. To tune model parameters, the best number of steps for training, the learning rate, etc., a randomly selected 50% of the training data is used for validation. We choose the parameters that maximize ROUGE scores over this validation set.
To implement PG-net we use OpenNMT BIBREF21 with the original set of parameters. The out-of-domain model is trained for 100k steps using the CNN/DM dataset. Following base model training, we tune the model by training it using student reflections. The tuning is done by lowering the LR from 0.15 to 0.1 and training the model for additional 500 steps. The in-domain model is trained only using reflections. We use the same model architecture as above and train the model for 20k steps using adagrad and LR of 0.15.
Synthesis Baseline (for testing H3, H4). Following BIBREF12, we developed a data synthesis baseline using word replacement via WordNet. The baseline iterates over all words in a summary. If word $X$ has $N$ synonyms in WordNet, the model creates $N$ new versions of the summary and corresponding reflections by replacing the word $X$ with each of the $N$ synonyms.
Template Synthesis Model (for testing H4, H5). To synthesize summaries, we use the same leave one course out approach. For each course, we use the data from the other three courses to train the rewriting module and tune the scoring language model. We can also use the summaries from CNN/DM data as additional samples to further train the rewriting module. We then start synthesizing data using that training data as input. First templates are constructed. The templates are then clustered into 8 clusters. We decided to use 8 to avoid clustering templates from POI with MP, as the templates from both prompts would contain very different supporting words. We also wanted to avoid a high level of dissimilarity within each cluster, and allow some diversity. Following the clustering, the rewriting model produces candidate summaries for each human summary. The rewriting model is another PG-net with the same exact parameters. After producing the candidate summaries, a language model is used to score them. The language model is a single layer LSTM language model trained on 36K sentences from Wikipedia and fine tuned using student reflections. In this work we decided to pick only the highest 3 scored candidate summaries as synthetic data, to avoid adding ill-formed summaries to the training data. Since we are adding $N$ synthetic summaries for each set of reflections, that means we are essentially duplicating the size of our original reflection training data by $N$, which is 3 in our case. Table TABREF11 shows a human summary, the keywords extracted, then the output of injecting keywords in a different template using rewriting.
Template-based Summarization (for testing H6). While the proposed template-based model was intended for data synthesis, with minor modification it can be adapted for summarization itself. Because the modifications introduce few parameters, the model is suitable for small datasets. Recall that for data synthesis, the input to the template method is a summary. Since for summarization the input instead is a set of reflections, we perform keyword extraction over the set of reflections. We then add an extra logistic regression classifier that uses the set of reflections as input and predicts a cluster of templates constructed from other courses. Using the keywords and the predicted cluster of templates, we use the same rewriting model to produce candidate summaries. The last step in the pipeline is scoring. In data synthesis, a reference summary is used for scoring; however, in summarization we don't have such a reference. To score the candidate summaries, the model only uses the language model and produces the candidate with the highest score.
Our experimental designs address the following hypotheses: Hypothesis 1 (H1) : Training complex abstractive models with limited in-domain or large quantities of out-of-domain data won’t be enough to outperform extractive baselines. Hypothesis 2 (H2) : Domain transfer helps abstractive models even if in-domain and out-of-domain data are very different and the amount of in-domain data is very small. Hypothesis 3 (H3) : Enriching abstractive training data with synthetic data helps overcome in-domain data scarcity. Hypothesis 4 (H4) : The proposed template-based synthesis model outperforms a simple word replacement model. Hypothesis 5 (H5) : Combining domain transfer with data synthesis outperforms using each approach on its own. Hypothesis 6 (H6) : The synthesis model can be extended to perform reflection summarization directly.
Answer: Hypothesis 1 (H1) : Training complex abstractive models with limited in-domain or large quantities of out-of-domain data won’t be enough to outperform extractive baselines. Hypothesis 2 (H2) : Domain transfer helps abstractive models even if in-domain and out-of-domain data are very different and the amount of in-domain data is very small. Hypothesis 3 (H3) : Enriching abstractive training data with synthetic data helps overcome in-domain data scarcity. Hypothesis 4 (H4) : The proposed template-based synthesis model outperforms a simple word replacement model. Hypothesis 5 (H5) : Combining domain transfer with data synthesis outperforms using each approach on its own. Hypothesis 6 (H6) : The synthesis model can be extended to perform reflection summarization directly.