Package: quinductor
QuinductorA multilingual data-driven method for generating reading comprehension questions. The official repository for the Quinductor article:https://arxiv.org/abs/2103.10121DataWe useTyDi QA dataset, which you can easily get by runningget_tydiqa_data.shHow to work with the induced templates?Quinductor is now available as a Python package that can be installed viapip install quinductor. After that you can download the induce templates for your language by running the following in the Python shell (the example is for English).>>>importquinductorasqi>>>qi.download('en')The avaible languages with a wide set of templates are:Arabic (ar)English (en)Finnish (fi)Indonesian (id)Japanese (ja)Russian (ru)Templates are also available for the other languages listed below, but Quinductor did not manage to induce many templates on the TyDiQA.Korean (ko)Telugu (te)After having downloaded the templates for your language, you can get access to them by running>>>tools=qi.use('en')Starting from v0.2.0, you can also use thetoolsdictionary to quickly induce QA-pairs using the following piece of code.importquinductorasqiimportudon2tools=qi.use("en")trees=udon2.ConllReader.read_file("example.conll")res=qi.generate_questions(trees,tools)print("\n".join([str(x)forxinres]))Each element in thereslist above will be an instance ofGeneratedQAPairclass, which has the following properties:q-- generated question as a stringa-- generated answer as a stringscore-- the Quinductor score for this QA-pair (the list is sorted in the descending order of the scores)template-- a list of templates that resulted in the induced QA-pairHow to induce templates yourself?Generate auxiliary models:IDFs by runningcalculate_idf.shranking models by runningget_qword_stat.shInduce templates and guards by runninginduce_templates.shIf you want to induce templates only for a specific language, please choose the correpsonding lines from the shell scripts.Using your own templatesQuinductor templates constitute a plain text file with a number of induced templates. However, in order for them to be used, Quinductor requires a number of extra files in addition to the templates file:guards file -- a plain text file with guards for all templates, i.e. conditions on the dependency trees that must be satisfied for applying each templateexamples file -- a file containing the sentences from the training corpus that gave rise to each templatequestion word model -- a dill binary file containing the question word model (see the associated article for explanations), can be induced by usingqword_stat.pyscriptanswer statistics file -- a dill binary file containng the statistics about pos-morph expressions for the root tokens of the answers in the training set, used for filtering (can be induced usingqword_stat.pyscript also)pos-morph n-gram model folder -- a folder containing a number of plain text files with n-gram models of pos-morph expressions (see the associated article for more details andewt_dev_freq.txtfor the example of the file format)Quinductor templates along with all aforementioned extra files constitute a Quinductor model. Each such model must be organized as a folder with the following structure:|- language code
  |- pos_ngrams -- a folder with pos-morph n-gram model
  |- dataset name -- a name of the dataset used for inducing templates
    |- a unique name for templates -- a timestamp if templates induced by the script from this repo
      |- guards.txt -- guards file
      |- templates.txt -- templates file
      |- sentences.txt -- examples file
    |- atmpl.dill -- answer statistics file
    |- qwstats.dill -- question word model fileIf you want to use a custom Quinductor model, you should organize your folder according to the structure above and give the path to the folder withtemplates.txtfile as an extra argument calledtemplates_folderto theqi.usemethod, as shown below.importquinductorasqitools=qi.use('sv',templates_folder='my_templates/sv/1613213402519069')If you want only parts of a Quinductor model to differ from one of the default models, you can specify more fine-grained self-explanatory arguments to theqi.usemethod:guards_files,templates_files,pos_ng_folder,example_files,qw_stat_file,a_stat_file.How to evaluate?We usenlg-eval packageto calculate automatic evaluation metrics.
This package requires to have hypothesis and ground truth files, where each line correspond to a question generated based on the same sentence.
To generate these files, please runevaluate.sh(if you want to induce templates only for a specific language, please choose the correpsonding lines from the shell scripts.).Then automatic evaluation metrics can be calculated by running a command similar to the following (example is given for Arabic):nlg-eval --hypothesis templates/ar/1614104416496133/eval/hypothesis_ar.txt --references templates/ar/1614104416496133/eval/ground_truth_ar_0.txt --references templates/ar/1614104416496133/eval/ground_truth_ar_1.txt --references templates/ar/1614104416496133/eval/ground_truth_ar_2.txt --no-glove --no-skipthoughtsCite us@misc{kalpakchi2021quinductor,
      title={Quinductor: a multilingual data-driven method for generating reading-comprehension questions using Universal Dependencies}, 
      author={Dmytro Kalpakchi and Johan Boye},
      year={2021},
      eprint={2103.10121},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}