Package: tmpproject
🛠 ALToolboxALToolbox is a framework for practical active learning in NLP.Installation|Quick Start|Overview|Docs|CitationALToolbox is a framework foractive learningannotation in natural language processing. Currently, the framework supports text classification and sequence tagging tasks. ALToolbox provides state-of-the-art query strategies, serverless annotation tool for Jupyter IDE, and a set of tools that help to reduce computational overhead / duration of AL iterations and increase annotated data reusability.⚙️ InstallationpipinstalltmpprojectTo annotate instances for active learning in Jupyter Notebook or Jupyter Lab one have to install additional widget after framework installation. In case of Jupyter Notebook usage run:jupyternbextensioninstall--py--symlink--sys-prefixtext_selector
jupyternbextensionenable--py--sys-prefixtext_selectorIn case of Jupyter Lab usage run:jupyterlabextensioninstalljs
jupyterlabextensioninstalltext_selector💫 Quick StartFor quick start, please see the examples of launching an active learning annotation or benchmarking a novel query stategy / unlabeled pool subsampling strategy for sequence tagging and text classification tasks:#Notebook1Launching Active Learning for Token Classification2Launching Active Learning for Text Classification3Benchmarking a novel AL query strategy / unlabeled pool subsampling strategy🔭 Overview1. Query Strategies#StrategyCitation1ALPSCitation2BADGECitation3BAITCitation4BALDCitation5BatchBALDCitation6Breaking Ties (BT) (also Maximum Margin)Citation7Contrastive Active Learning (CAL)Citation8Cluster MarginCitation9CoresetCitation10Expected Gradient Length (EGL)Citation11Embeddings KMCitation12EntropyCitation13Least Confidence (LC)Citation14Mahalanobis DistanceCitation15Maximum Normalized Log-Probability (MNLP)Citation16Random (No AL)-3. Unlabeled Pool Subsampling Strategies#StrategyCitation1UPSCitation2NaïveCitation3Random-4. Pipelines for postprocessing of annotated data and preparation of acquisition modelsPLASM postprocessing pipeline for annotated data reusability.Acquisition model distillation.Domain adaptation of acquisition models.5. GUI Annotator tool in Jupyter IDEOur framework provides a serverless GUI annotation tool integrated into the Jupyter IDE:6. Extensible benchmark for query strategiesTODO:📕 DocumentationUsageTheconfigsfolder contains config files with general settings. Theexperimentsfolder contains config files with experimental design. To run an experiment with a chosen configuration, specify config file name inHYDRA_CONFIG_NAMEvariable and runtrain.shscript (see./examples/alfor details).For example to launch PLASM on AG-News with ELECTRA as a successor model:cd PATH_TO_THIS_REPO
HYDRA_CONFIG_PATH=../experiments/ag_news HYDRA_EXP_CONFIG_NAME=ag_plasm python active_learning/run_tasks_on_multiple_gpus.pyConfig structure explanationcuda_devices: list of CUDA devices to use: one experiment on one CUDA device.cuda_devices=[0,1]means using zero-th and first devices.config_name: name of config fromconfigsfolder with general settings: dataset, experiment setting (e.g. LC/ASM/PLASM), model checkpoints, hyperparameters etc.config_path: path to config with general settings.command:.pyfile to run. For AL experiments, userun_active_learning.py.args: arguments to modify from a general config in the current experiment.acquisition_model.name=xlnet-base-casedmeans thatxlnet-base-casedwill be used as an acquisition model.seeds: random seeds to use.seeds=[4837, 23419]means that two separate experiments with the same settings (except forseed) will be run: one withseed == 4837, one withseed == 23419.Output ExplanationBy default, the results will be present in the folderRUN_DIRECTORY/workdir_run_active_learning/DATE_OF_RUN/${TIME_OF_RUN}_${SEED}_${MODEL_CHECKPOINT}. For instance, when launching from the repository folder:al_nlp_feasible/workdir/run_active_learning/2022-06-11/15-59-31_23419_distilbert_base_uncased_bert_base_uncased.When running a classic AL experiment (acquisition and successor models coincide, regardless of using UPS), the file with the model metrics isacquisition_metrics.json.When running an acquisition-successor mismatch experiment, the file with the model metrics issuccessor_metrics.json.When running a PLASM experiment, the file with the model metrics istarget_tracin_quantile_-1.0_metrics.json(-1.0stands for the filtering value, meaning adaptive filtering rate; when using a deterministic filtering rate (e.g.0.1), the file will be namedtarget_tracin_quantile_0.1_metrics.json). The file with the metrics of the modelwithout filteringistarget_metrics.json.Post-processingOur framework provides tools for effective data post-processing for its re-usability and a possibility to build powerful models on it.
PLASM, which aims to alleviate the acquisition-successor mismatch problem and allow to build a model of an
arbitrary type using the labeled data without performance degradation, is implemented inpost_processing/pipeline_plasm.
It uses the configcls_plasm/ner_plasm(from `jupyterlab_demo/configs). A brief explanation of the config structure:pseudo-labeling model parameters are contained in the keylabeling_model;successor model parameters are contained in the keysuccessor_model;post-processing options are contained in the keypost_processing:label_smoothing: str / float / None, a parameter for label smoothing (LS) for pseudo-labeled instances. Accepts several options:"adaptive": LS value equals the quality of the labeling model on the validation data.float, 0 < value < 1: absolute value of label smoothingNone (default): no label smoothing is usedlabeled_weight: int / float, weight for the labeled-by-human data. 1 < value < +infuse_subsample_for_pl: int / float / None, the size of the subsample used for pseudo-labeling
(float means taking the share of the unlabeled data). None means that no subsampling is used.uncertainty_threshold: float / None, the value of the threshold for filtering by uncertainty. If None,
no filtering by uncertainty is used.filter_by_quantile: bool, only used for classification, ignored ifuncertainty_thresholdis None. If True,uncertainty_thresholdmost uncertain instances are filtered. Otherwise, all instances whose (1 - max_prob) <uncertainty_thresholdare filtered.tracin:use: bool, whether to use TracIn for filteringmax_num_processes: int, value > 0, maximum number of processes per one GPUquantile: str / float (0 < value < 1), share of unlabeled data instances to filter using the TracIn score.num_model_checkpoints: int, value > 0, how many model checkpoints to save and use for TracIn.nu: float / int, value for TracIn algorithm.🆕️ New strategies additionAn AL query strategy should be designed as a function that:Receives 3 positional arguments and additional strategy kwargs:
-modelof inherited classTransformersBaseWrapperorPytorchEncoderWrapperorFlairModelWrapper: model wrapper;
-X_poolof classDatasetorTransformersDataset: dataset with the unlabeled instances;
-n_instancesof classint: number of instances to query;
-kwargs: additional strategy-specific arguments.Outputs 3 objects in the following order:query_idxof classarray-like: array with the indices of the queried instances;queryof classDatasetorTransformersDataset: dataset with the queried instances;uncertainty_estimatesof classnp.ndarray: uncertainty estimates of the instances fromX_pool. The higher the value - the more uncertain the model is in the instance.The function with the strategy should be named the same as the file where it is placed (e.g. functiondef my_strategyinside a filepath_to_strategy/my_strategy.py).
Use your strategy, settingal.strategy=PATH_TO_FILE_YOUR_STRATEGYin the experiment config.The example is presented inexamples/benchmark_custom_strategy.ipynb🆕️ New pool subsampling strategies additionThe addition of a new pool subsampling query strategy is similar to the addition of an AL query strategy. A subsampling strategy should be designed as a function that:It must receive 2 positional arguments and additional subsampling strategy kwargs:
-uncertainty_estimatesof classnp.ndarray: uncertainty estimates of the instances in the order they are stored in the unlabeled data;
-gamma_or_k_confident_to_saveof classfloatorint: either a share / number of instances to save (as in random / naive subsampling) or an internal parameter (as in UPS);
-kwargs: additional subsampling strategy specific arguments.It must output the indices of the instances to use (sampled indices) of classnp.ndarray.The function with the strategy should be named the same as the file where it is placed (e.g. functiondef my_subsampling_strategyinside a filepath_to_strategy/my_subsampling_strategy.py).
Use your subsampling strategy, settingal.sampling_type=PATH_TO_FILE_YOUR_SUBSAMPLING_STRATEGYin the experiment config.The example is presented inexamples/benchmark_custom_strategy.ipynbDatasetsThe research has employed 2 Token Classification datasets (CoNLL-2003, OntoNotes-2012) and 2 Text Classification datasets (AG-News, IMDB). If one wants to launch an experiment on a custom dataset, they need to use one of the following ways to add it:Upload toHugging Face datasetsand set:config.data.path=datasets, config.data.dataset_name=DATASET_NAME, config.data.text_name=COLUMN_WITH_TEXT_OR_TOKENS_NAME, config.data.label_name=COLUMN_WITH_LABELS_OR_NER_TAGS_NAMEUpload todata/DATASET_NAMEfolder, createtrain.csv/train.jsonfile with the dataset, and set:config.data.path=PATH_TO_THIS_REPO/data, config.data.dataset_name=DATASET_NAME, config.data.text_name=COLUMN_WITH_TEXT_OR_TOKENS_NAME, config.data.label_name=COLUMN_WITH_LABELS_OR_NER_TAGS_NAME* Upload todata/DATASET_NAMEtrain.txt,dev.txt, andtest.txtfiles and set the arguments as in the previous point.** Upload todata/DATASET_NAMEwith each folder for each class, where each file in the folder contains a text with the label of the folder. For details, please see thebbc_newsdataset in./data. The arguments must be set as in the previous two points.* - only for Token Classification datasets** - only for Text Classification datasetsModelsThe current version of the repository supports all models fromHuggingFace Transformers, which can be used withAutoModelForSequenceClassification/AutoModelForTokenClassificationclasses (for Text / Token classification). For CNN-based / BiLSTM-CRF models, please see theal_cls_cnn.yaml/al_ner_bilstm_crf_flair.yamlconfigs from./configsfolder for details.TestingBy default, the tests will be run on thecuda:0device if CUDA is available or on CPU, otherwise. If one wants to manually specify the device for running the tests:On CPU:CUDA_VISIBLE_DEVICES="" python -m pytest PATH_TO_REPO/tests;On CUDA:CUDA_VISIBLE_DEVICES="DEVICE_OR_DEVICES_NUMBER" python -m pytest PATH_TO_REPO/tests.We recommend to use CPU for the robustness of the results. The tests for CUDA are written underTesla V100-SXM3 32GB, CUDA V.10.1.243.👯 AlternativesFAMIE,Small-Text,modAL,ALiPy,libact💬 Citation📄 License© 2022 Autonomous Non-Profit Organization "Artificial Intelligence Research Institute" (AIRI). All rights reserved.Licensed under theMIT License.