Package: mdmls
Generating Extended and Multilingual Summaries with Pre-trained TransformersCode for the paper Generating Extended and Multilingual Summaries with Pre-trained Transformers accepted at LREC 2022.Getting startedCreate the environnement, activate and install requirements.condacreate-nmdmlspython=3.7
condaactivatemdmls
pipinstall-rrequirements.txtWikinewsSum datasetPlease refer tohttps://github.com/airklizz/wikinewssumto download the dataset.Place thetrain.json,validation.json, andtest.jsonfiles in thewikinewssum/folder.PreprocessingPrepare the dataset to fine-tune an abstractive model using an extractive pre-abstractive step.pythonmdmls/main.pypreprocessextractive-bert\wikinewssum/train.json\wikinewssum/train_pre_abstractive.json\--model-checkpointdistilbert-base-multilingual-cased\--pre-abstractive\--abstractive-model-checkpointgoogle/mt5-smallTokenize the dataset.pythonmdmls/main.pypreprocesstokenize\wikinewssum/train_pre_abstractive.json\wikinewssum/train_pre_abstractive_tokenized.json\--sourcedistilbert-base-multilingual-cased_extractive_summary\--model-checkpointgoogle/mt5-smallThe same steps need to be performed for the validation set.Fine-tuningUse the command line interface to fine-tune a new model of the WikinewsSum dataset.pythonmdmls/main.pytrainrun\--train-data-fileswikinewssum/train_pre_abstractive_tokenized.json\--validation-data-fileswikinewssum/validation_pre_abstractive_tokenized.json\--training-scenario"new-fine-tuning"\--model-checkpointgoogle/mt5-baseTo see all the parameters.>pythonmdmls/main.pytrainrun--help
Usage:main.pytrainrun[OPTIONS]Options:--train-data-filesTEXT--validation-data-filesTEXT--training-scenarioTEXT--model-checkpointTEXT[default:google/mt5-small]--batch-sizeINTEGER[default:8]--gradient-accumulation-stepsINTEGER[default:1]--num-train-epochsINTEGER[default:8]--learning-rateFLOAT[default:5.6e-05]--weight-decayFLOAT[default:0.01]--save-total-limitINTEGER[default:3]--push-to-hub/--no-push-to-hub[default:push-to-hub]--languageTEXT--max-number-training-sampleINTEGER--helpShowthismessageandexit.optiondescription--languageif specified, only the samples of the specified language are kept. For example:--language ento train on the English samples only--max-number-training-sampleif specified, limit the number of training sample to the valueEvaluationROUGE scoresMethodsMetricsEnglishGermanFrenchSpanishPortuguesePolishItalianAll LanguagesExtractive SummarisationDistilmBERTR-1 F41.3729.3729.8029.7029.6224.8335.1833.51R-2 F14.358.4212.5712.5214.3310.4812.5912.34R-L F19.6613.6517.1017.0718.7515.0318.4317.30mBERTR-1 F41.3729.7429.7435.5029.6624.8234.9333.60R-2 F14.488.7012.6213.3114.5110.5512.6812.51R-L F19.6313.8317.1318.1018.8615.0718.8617.36XLM-RoBERTaR-1 F40.9229.0029.7035.4029.3924.7435.6833.27R-2 F14.228.3312.5213.0314.1310.4912.5412.26R-L F19.6613.5417.0718.0518.4315.0319.5417.26OracleR-1 F49.5037.2134.4142.2435.3229.8941.8540.29R-2 F25.7215.7717.3120.8921.4015.7219.9420.35R-L F22.6715.9317.3820.5419.1915.3318.6119.16Abstractive Summarisation after Oracle Pre-Abstractive Extractive StepmT5 Cross-lingual zero-shot transferR-1 F44.269.139.6311.2310.776.939.7119.99R-2 F21.732.852.523.713.261.762.488.53R-L F24.256.316.327.817.515.056.5311.92mT5 In-language multi-taskR-1 F43.1933.1436.9237.6934.5427.9537.0037.05R-2 F21.3313.4717.4017.4618.0513.6513.8717.51R-L F23.7017.0021.4421.3321.4416.9819.0120.78mT5 In-languageR-1 F44.2635.0639.4143.8141.0032.264.2740.04R-2 F21.7313.6317.7619.2920.2214.340.5818.23R-L F24.2517.5322.0323.7624.4418.673.0621.93Abstractive Summarisation after mBERT Pre-Abstractive Extractive StepmT5 Cross-lingual zero-shot transferR-1 F37.247.199.1410.029.566.3012.4017.08R-2 F13.001.681.872.482.271.302.825.25R-L F19.685.085.976.896.744.587.3710.00mT5 In-language multi-taskR-1 F35.5627.0532.5932.9430.0123.5332.9031.30R-2 F12.287.8413.0611.6513.149.3710.2411.24R-L F18.7013.7118.9318.1618.8214.2216.9317.25mT5 In-languageR-1 F37.2429.6536.0239.7937.2128.474.3235.03R-2 F13.008.3214.0813.8615.4610.660.1012.37R-L F19.6814.7620.0821.1713.2016.652.8018.04ROUGE F-measure results of the three evaluations presented in the paper on WikinewsSum. We compare the extractive models, and mT5 in the three training scenarios and with two different pre-abstractive extractive steps (Oracle and mBERT) for each language of the WikinewsSum dataset in addiction to the all dataset.BERTScore scoresMethodsMetricsEnglishGermanFrenchSpanishPortuguesePolishItalianAll LanguagesExtractive SummarisationDistilmBERTB-S P0.69200.66690.63570.68070.66800.64550.67060.6697B-S R0.71960.68900.68460.71040.70840.68340.70680.7021B-S F0.70520.67740.65850.69490.68690.66330.68790.6850mBERTB-S P0.69080.66790.63540.68100.66730.64560.66180.6695B-S R0.72150.69310.68550.71240.70840.68480.70330.7041B-S F0.70550.67990.65870.69600.68650.66400.68160.6859XLM-RoBERTaB-S P0.69000.66580.63510.67940.66600.64510.67520.6684B-S R0.71730.68780.68340.70870.70610.68310.70990.7005B-S F0.70310.67620.65760.69340.68480.66290.69170.6836OracleB-S P0.72380.69470.65280.70580.69300.66380.69190.6955B-S R0.74360.71440.69670.72280.72660.70240.71900.7217B-S F0.73320.70390.67310.71380.70870.68180.70470.7077Abstractive Summarisation after Oracle Pre-Abstractive Extractive StepmT5 Cross-lingual zero-shot transferB-S P0.75260.68140.66870.70140.68640.64680.68200.7009B-S R0.71990.64310.65790.66500.66410.62180.64800.6717B-S F0.73540.66140.66270.68240.67460.63370.66440.6855mT5 In-language multi-taskB-S P0.74940.72190.71300.73060.72740.68870.72030.7274B-S R0.71900.69370.71740.70300.71400.68470.69420.7074B-S F0.73340.70700.71380.71610.71970.68570.70660.7165mT5 In-languageB-S P0.75260.72640.71640.73740.73810.69740.46030.7321B-S R0.71990.69390.71790.70730.71940.69080.52610.7092B-S F0.73540.70930.71530.72160.72770.69310.49050.7196Abstractive Summarisation after mBERT Pre-Abstractive Extractive StepmT5 Cross-lingual zero-shot transferB-S P0.72020.66800.65710.68580.67570.64120.66930.6828B-S R0.70040.63630.65170.65760.65860.61800.64590.6615B-S F0.70980.65150.65380.67120.66660.62900.65720.6716mT5 In-language multi-taskB-S P0.71570.69580.69530.70690.70940.67000.70450.7022B-S R0.69810.67740.70330.68910.70110.67020.68690.6910B-S F0.70650.68610.69820.69760.70460.66930.69520.6960mT5 In-languageB-S P0.72020.70430.70200.71510.71860.68360.44950.7091B-S R0.70040.68070.70690.69480.70640.68030.52130.6949B-S F0.70980.69190.70260.70440.71160.68110.48220.7012BERTScore (Zhang et al., 2020b) precision (B-S P), recall (B-S R), and F1 (B-S F) results of the three evaluations presented in the paper on WikinewsSum. We compare the extractive models, and mT5 in the three training scenarios and with two different pre-abstractive extractive steps (Oracle and mBERT) for each language of the WikinewsSum dataset in addiction to the all dataset. Hash code for the BERTScore metric: bert-base-multilingual-cased_L9_no-idf_version=0.3.11(hug_trans=4.13.0)_fast-tokenizerUsageThemdmlspip package allows to run the combination of an extractive method combined with an abstractive one.pipinstallmdmlsIt can be used as follows in Python.frommdmlsimportSummarizersum=Summarizer()summary=sum(LONG_TEXT_TO_SUMMARIZE)Or directly using the CLI.mdmls"LONG_TEXT_TO_SUMMARIZE"ModelsAll the fine-tuned abstractive models are available on the HuggingFace Hub:https://huggingface.co/models?sort=downloads&search=airklizz+mt5+wikinewssum