Package: qs-kpa
🏅Quantitative Summarization – Key Point Analysis🏅Keypoint AnalysisThis library is based on the Transformers library by HuggingFace.Keypoint Analysisquickly embeds the statements with the provided supported topic and the stances toward that topic. It is a part of our approach in theQuantitative Summarization – Key Point Analysisshared task byIBM. We use the ArgKP dataset (Bar-Haim et al., ACL-2020), which contains ~24K argument/key-point pairs, for 28 controversial topics in our training and evaluation.What's NewJuly 1, 2021First release ofqs-kpapython packageJuly 4, 2021Our method achieved the 4th position in track I of the shared task[paper]InstallationInstall with pip (stable version)pipinstallqs-kpaInstall from sources (latest version)gitclonehttps://github.com/VietHoang1512/KPA
pipinstall-e.Quick exampleCurrently, a pretrained KPA encoder withRoBERTabackbone is available, which can be automatically downloaded from Google Drive when initializing theKeyPointAnalysisinstance. We used the 4 last hidden state representations of the [CLS] token as the whole sentence embedding and trained it withTupletMarginLossandIntraPairVarianceLosson the ArgKP dataset. For the code, seemain.py.# Import needed librariesfromqs_kpaimportKeyPointAnalysis# Create a KeyPointAnalysis model# Set from_pretrained=True in order to download the pretrained modelencoder=KeyPointAnalysis(from_pretrained=True)# Model configurationprint(encoder)# Preparing data (a tuple of (topic, statement, stance) or a list of tuples)inputs=[("Assisted suicide should be a criminal offence","a cure or treatment may be discovered shortly after having ended someone's life unnecessarily.",1,),("Assisted suicide should be a criminal offence","Assisted suicide should not be allowed because many times people can still get better",1,),("Assisted suicide should be a criminal offence","Assisted suicide is akin to killing someone",1),]# Go and embedd everythingoutput=encoder.encode(inputs,convert_to_numpy=True)In acomparisonwith the baseline model-which directly uses sentence embedding from RoBERTa model, in a subset of ArgKP dataset (for avoiding target leakage), our model strongly outperforms and exhibits rich representation learning capacity. Evaluation metrics (relaxed and strict mean Average Precision) are retained from theKPA_2021_shared_task.Model using roBERTa directly: mAP strict = 0.4633403767342183 ; mAP relaxed = 0.5991767005443296
Our pretrained model: mAP strict = 0.9170783671441644 ; mAP relaxed = 0.9722347939653511Detailed trainingGiven a pair of key point and argument (along with their supported topic & stance) and the matching score. Similar pairs with label 1 are pulled together, or pushed away otherwise.ModelModelBERT/ConvBERTBORTLUKEDistilBERTALBERTXLNetRoBERTaELECTRABARTMPNetSiamese Baseline✔️✔️✔️✔️✔️✔️✔️✔️✔️✔️Siamese Question Answering-like✔️✔️✔️✔️✔️✔️✔️✔️✔️✔️Custom loss Baseline✔️✔️✔️✔️✔️✔️✔️✔️✔️✔️LossConstrastiveOnline ConstrastiveTripletOnline Triplet (Hard negative/positive mining)DistanceEuclideanCosineManhattanUtilsK-foldsFull-flowPseudo-labelGroup the arguments by their key point and consider the order of that key point within the topic as their labels (seepseudo_label). We can now utilize available Pytorch metrics learning distance, losses, miners or reducers from this greatopen-sourcein the main training workflow. For training, we use a key point and some of its positive/negative arguments as a batch. The goal is to minimize the distance between the keypoint and positive arguments as well as the arguments themselves. This is also our best approach (single-model) so far.Training exampleAn example script for training RoBERTa on the ArgKP dataset. It runs in about 15 minutes each fold on a single Google Colab Tesla P100.OUTPUT_DIR=outputs/pseudo_label/roberta-base/echo"OUTPUT DIRECTORY$OUTPUT_DIR"mkdir-p$OUTPUT_DIRcpqs_kpa/pseudo_label/models.py$OUTPUT_DIRforfold_idin1234567doecho"TRAINING ON FOLD$fold_id"pythonscripts/main.py\--experiment"pseudolabel"\--output_dir"$OUTPUT_DIR/fold_$fold_id"\--model_name_or_pathroberta-base\--tokenizerroberta-base\--distance"cosine"\--directory"kpm_k_folds/fold_$fold_id"\--test_directory"kpm_k_folds/test/"\--logging_dir"$OUTPUT_DIR/fold_$fold_id"\--logging_steps20\--max_pos30\--max_neg90\--max_unknown15\--overwrite_output_dir\--num_train_epochs5\--early_stop10\--train_batch_size1\--val_batch_size128\--do_train\--evaluate_during_training\--warmup_steps0\--gradient_accumulation_steps1\--learning_rate0.00003\--margin0.3\--drop_rate0.2\--n_hiddens4\--max_len30\--statement_max_len50\--stance_dim32\--text_dim256\--num_workers4\--seed0doneTraining with the previously defined hyper-parameters yields above mentioned mAP score. Other approaches could be found inbin.ContributorsPhan Viet HoangNguyen Duc LongBibTeX@misc{hoang2021qskpa,author={Phan, V.H. & Nguyen, D.L.},title={Keypoint Analysis},year={2021},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/VietHoang1512/KPA}}}