Package: textflint
Unified Multilingual Robustness Evaluation Toolkit 
  for Natural Language ProcessingTextFlint is a multilingual robustness evaluation platform for natural language processing, which unifies texttransformation,sub-population,adversarial attack,and their combinations to provide a comprehensive robustness analysis. So far, TextFlint supports 13 NLP tasks.If you're looking for robustness evaluation results of SOTA models, you might want theTextFlint IOpage.FeaturesFull coverage of transformation types, including 20 general transformations, 8 subpopulations and 60 task-specific transformations, as well as thousands of their combinations.Subpopulation, which is to identify the specific part of dataset on which the target model performs poorly.Adversarial attackaims to find a perturbation of an input text that is able to fool the given model.Complete analytical reportto accurately explain where your model's shortcomings are, such as the problems in lexical rules or syntactic rules.Online DemoYou can test most of transformations directly on ouronline demo.Table of ContentsSetupUsageArchitectureLearn MoreContributingCitationSetupRequirepython version >= 3.7, recommend install withpip.pipinstalltextflintOnce TextFlint is installed, you can run it via command-line (textflint ...) or integrate it inside another NLP project.UsageWorkflowThe general workflow of TextFlint is displayed above. Evaluation of target models could be divided into three steps:For input preparation, the original dataset for testing, which is to be loaded byDataset, should be firstly formatted as a series ofJSONobjects. You can use the built-inDatasetfollowing thisinstruction. TextFlint configuration is specified byConfig. Target model is also loaded asFlintModel.In adversarial sample generation, multi-perspective transformations (i.e.,80+Transformation,SubpopulationandAttackRecipe), are performed onDatasetto generate transformed samples. Besides, to ensure semantic and grammatical correctness of transformed samples,Validatorcalculates confidence of each sample to filter out unacceptable samples.Lastly,Analyzercollects evaluation results andReportGeneratorautomatically generates a comprehensive report of model robustness.For example, on the Sentiment Analysis (SA) task, this is a statistical chart of the performance ofXLNETwith different types ofTransformation/Subpopulation/AttackRecipeon theIMDBdataset.We release tutorials of performing the whole pipeline of TextFlint on various tasks, including:Machine Reading ComprehensionPart-of-speech TaggingNamed Entity RecognitionChinese Word SegmentationQuick StartUsing TextFlint to verify the robustness of a specific model is as simple as running the following command:$textflint--datasetinput_file--configconfig.jsonwhereinput_fileis the input file of csv or json format,config.jsonis a configuration file with generation and target model options.  Transformed datasets would save to your out dir according to yourconfig.json.Based on the design of decoupling sample generation and model verification,TextFlintcan be used inside another NLP project with just a few lines of code.fromtextflintimportEnginedata_path='input.json'config='config.json'engine=Engine()engine.run(data_path,config)For more input and output instructions of TextFlint, please refer to theIO format  document.ArchitectureInput layer:receives textual datasets and models as input, represented asDatasetandFlintModelseparately.DataSet: a container, provides efficient and handy operation interfaces forSample.Datasetsupports loading, verification, and saving data in Json or CSV format for various NLP tasks.FlintModel: a target model used in an adversarial attack.Generation layer:there are mainly four parts in generation layer:Subpopulation: generates a subset of aDataSet.Transformation: transforms each sample ofDatasetif it can be transformed.AttackRecipe: attacks theFlintModeland generates aDataSetof adversarial examples.Validator: verifies the quality of samples generated byTransformationandAttackRecipe.textflint provides an interface to integrate the easy-to-use adversarial attack recipes implemented based ontextattack. Users can refer totextattackfor more information about the supportedAttackRecipe.Report layer:analyzes model testing results and provides robustness report for users.Learn MoreSectionDescriptionDocumentationFull API documentation and tutorialsTutorialThe tutorial of textflint components and pipelineWebsiteProvides evaluation results of SOTA models and transformed data downloadOnline DemoInteractive demo to try single text transformationsPaperOur system paper which was received by ACL2021ContributingWe welcome community contributions to TextFlint in the form of bugfixes 🛠️ and new features💡!   If you want to contribute, please first readour contribution guideline.CitationIf you are using TextFlint for your work, please kindly cite ourACL2021 TextFlint demo paper:@inproceedings{wang-etal-2021-textflint,
    title ={TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing},
    author ={Wang, Xiao  and Liu, Qin  and Gui, Tao  and Zhang, Qi and others},
    booktitle ={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations},
    month ={aug},
    year ={2021},
    address ={Online},
    publisher ={Association for Computational Linguistics},
    url ={https://aclanthology.org/2021.acl-demo.41},
    doi ={10.18653/v1/2021.acl-demo.41},
    pages ={347--355}}