Package: distilbert-punctuator
Distilbert-punctuatorIntroductionDistilbert-punctuator is a python package provides a bert-based punctuator (fine-tuned model ofpretrained huggingface DistilBertForTokenClassification) with following three components:data process: funcs for processing user's data to prepare for training. If user perfer to fine-tune the model with his/her own data.training: training pipeline and doing validation. User can fine-tune his/her own punctuator with the pipelineinference: easy-to-use interface for user to use trained punctuator.If user doesn't want to train a punctuator himself/herself, two pre-fined-tuned model from huggingface model hubQishuai/distilbert_punctuator_en📎Model detailsQishuai/distilbert_punctuator_zh📎Model detailsmodel examples in huggingface web page.English modelSimplified Chinese modelInstallationInstalling the package from pypi:pip install distilbert-punctuatorfor directly usage of punctuator.Installing the package with option to do data processingpip install distilbert-punctuator[data_process].Installing the package with option to train and validate your own modelpip install distilbert-punctuator[training]For development and contributionclone the repomake installData ProcessComponent for pre-processing the training data. To use this component, please install aspip install distilbert-punctuator[data_process]The package is providing a simple pipeline for you to generateNERformat training data.Exampleexamples/data_sample.pyTrainComponent for providing a training pipeline for fine-tuning a pretrainedDistilBertForTokenClassificationmodel fromhuggingface.
The latest version has the implementation ofR-Dropenhanced training.R-Drop github repoPaper of R-DropExampleexamples/english_train_sample.pyTraining_arguments:Arguments required for the training pipeline.basic argumentstraining_corpus(List[List[str]]): list of sequences for training, longest sequence should be no longer than pretrained LM # noqa: E501validation_corpus(List[List[str]]): list of sequences for validation, longest sequence should be no longer than pretrained LM # noqa: E501training_tags(List[List[int]]): tags(int) for trainingvalidation_tags(List[List[int]]): tags(int) for validationmodel_name_or_path(str): name or path of pre-trained modeltokenizer_name(str): name of pretrained tokenizertraining argumentsepoch(int): number of epochbatch_size(int): batch sizemodel_storage_dir(str): fine-tuned model storage pathlabel2id(Dict): the tags label and id mappingearly_stop_count(int): after how many epochs to early stop training if valid loss not become smaller. default 3 # noqa: E501gpu_device(int): specific gpu card index, default is the CUDA_VISIBLE_DEVICES from environwarm_up_steps(int): warm up steps.r_drop(bool): whether to train with r-dropr_alpha(int): alpha value for kl divengence in the loss, default is 0plot_steps(int): record training status to tensorboard among how many stepstensorboard_log_dir(Optional[str]): the tensorboard logs output directory, default is "runs"model argumentsaddtional_model_config(Optional[Dict]): additional configuration for modelYou can also train your own NER models with the trainer provided in this repo.
The example can be found innotebooks/R-drop NER.ipynbEvaluationValidation of fine-tuned modelExampleexamples/train_sample.pyValidation_arguments:evaluation_corpus(List[List[str]]): list of sequences for evaluation, longest sequence should be no longer than pretrained LM's max_position_embedding(512)evaluation_tags(List[List[int]]): tags(int) for evaluation (the GT)model_name_or_path(str): name or path of fine-tuned modeltokenizer_name(str): name of tokenizerbatch_size(int): batch sizelabel2id(Optional[Dict]): label2id. Default one is from model config. Pass in this argument if your model doesn't have a label2id inside configgpu_device(int): specific gpu card index, default is the CUDA_VISIBLE_DEVICES from environInferenceComponent for providing an inference interface for user to use punctuator.Architecture+----------------------+              (child process)
 |   user application   |             +-------------------+
 +                      + <---------->| punctuator server |
 |   +inference object  |             +-------------------+
 +----------------------+The punctuator will be deployed in a child process which communicates with main process through pipe connection.
Therefore user can initialize an inference object and call itspunctuationfunction when needed. The punctuator will never block the main process unless doing punctuation.
There is agraceful shutdownmethodology for the punctuator, hence user dosen't need to worry about the shutting-down.Exampleexamples/inference_sample.pyInference_argumentsArguments required for the inference pipeline.model_name_or_path(str): name or path of pre-trained modeltokenizer_name(str): name of pretrained tokenizertag2punctuator(Dict[str, tuple]): tag to punctuator mapping.
dbpunctuator.utils provides two default mappings for English and ChineseNORMAL_TOKEN_TAG="O"DEFAULT_ENGLISH_TAG_PUNCTUATOR_MAP={NORMAL_TOKEN_TAG:("",False),"COMMA":(",",False),"PERIOD":(".",True),"QUESTIONMARK":("?",True),"EXLAMATIONMARK":("!",True),}DEFAULT_CHINESE_TAG_PUNCTUATOR_MAP={NORMAL_TOKEN_TAG:("",False),"C_COMMA":("，",False),"C_PERIOD":("。",True),"C_QUESTIONMARK":("? ",True),"C_EXLAMATIONMARK":("! ",True),"C_DUNHAO":("、",False),}for own fine-tuned model with different tags, pass in your own mappingtag2id_storage_path(Optional[str]): tag2id storage path. Default one is from model config. Pass in this argument if your model doesn't have a tag2id inside config