Question: Do they use any human evaluation metrics in the Chinese QG task?
Evidence: We evaluate our model on the zero-shot cross-lingual answer-aware question generation task. The goal of question generation (QG) is to generate a question that asks towards the answer with the given passage and the expected answer. In the following experiments, we extend the QG task to the cross-lingual setting. By only using English QG training data, our goal is to generate questions in English or Chinese with the given passage-answer pair in English or Chinese.
We use SQuAD 1.1 BIBREF30 as the English QG dataset. It is a popular English question answering dataset containing over 100,000 questions and their corresponding annotated passages. Following BIBREF31, we regard the original development set as the test set, and sample 5000 examples from the training data of two datasets as the development sets. For Chinese QG, we follow the default data splits of WebQA BIBREF32. We regard the provided annotated evidence sentences as the input passages instead of entire documents. To construct the input sequence, we view the whole input passage as a single sentence, and concatenate the passage and the answer into one sequence with a special token S between them. During decoding Chinese, we utilize a subset of vocabulary, which is obtained from the passage sentences of the WebQA dataset.
We first conduct experiments on the supervised English-English QG setting. We compare our model to the following baselines:
CorefNqg BIBREF33 A sequence-to-sequence model with attention mechanism and a feature-rich encoder.
Mp-Gsn BIBREF31 A sequence-to-sequence model with gated self-attention and maxout pointer mechanism.
Xlm BIBREF5 The current state-of-the-art cross-lingual pre-training model. We initialize the Transformer-based sequence-to-sequence model with pre-trained XLM.
We evaluate models with BLEU-4 (BL-4), ROUGE (RG) and METEOR (MTR) metrics. As shown in Table TABREF16, our model outperforms the baselines, which demonstrates that our pre-trained model provides a good initialization for NLG.
We conduct experiments on the zero-shot Chinese-Chinese QG task to evaluate the cross-lingual transfer ability. In this task, models are trained with English QG data but evaluated with Chinese QG examples. We include the following models as our baselines:
Xlm Fine-tuning XLM with the English QG data.
Pipeline (Xlm) The pipeline of translating input Chinese sentences into English first, then performing En-En-QG with the XLM model, and finally translating back to the Chinese. We use the Transformer as the translator, which is also trained on the MultiUN dataset.
Pipeline (Xlm) with Google Translator Same to Pipeline (Xlm) but using Google Translator to translate the texts.
We evaluate models by both automatic evaluation metrics and human experts. The automatic metrics scores are computed by regarding each Chinese character as a token. For human evaluation, we consider three metrics for the generated questions: relatedness, fluency, and correctness, which are represented as integers ranged from 1 to 3. We randomly select 100 passage-answer pairs from the English QG test set, and use the models to generate questions. Then we present these examples to three experts to ask for the above scores. In Table TABREF17 and Table TABREF18, we present the results for the zero-shot Zh-Zh-QG. The results of monolingual supervised models are also reported in Table TABREF16 as reference. In the automatic evaluation, our model consistently performs better than baselines in both zero-shot and monolingual supervised setting. In the human evaluation, our model also obtains significant improvements in terms of relatedness and correctness.
In the zero-shot English-Chinese question generation experiments, we use Xlm and Pipeline (Xlm) as our baselines. Pipeline (Xlm) is a pipeline method that uses En-En-QG with Xlm to generate questions, and then translates the results to Chinese. Because there is no annotations for En-Zh-QG, we perform human evaluation studies for this setting. Table TABREF19 shows the human evaluation results, where our model surpasses all the baselines especially in terms of relatedness and correctness.
We also conduct experiments for zero-shot Chinese-English question generation, and adopt the same evaluation procedure to En-Zh-QG. Pipeline (Xlm) first translates Chinese input to English, and then conduct En-En-QG with Xlm. As shown in Table TABREF20, human evaluation results indicate that Xnlg achieves significant improvements on the three metrics.
We evaluate models by both automatic evaluation metrics and human experts. The automatic metrics scores are computed by regarding each Chinese character as a token. For human evaluation, we consider three metrics: relatedness, fluency, and correctness, which are represented as integers ranged from 1 to 3.
Answer: Yes. Relatedness, fluency, and correctness.