Package: bunruija
BunruijaBunruija is a text classification toolkit.
Bunruija aims at enabling pre-processing, training and evaluation of text classification models withminimum coding effort.
Bunruija is mainly focusing on Japanese though it is also applicable to other languages.Seeexamplefor understanding how bunruija is easy to use.FeaturesMinimum requirements of coding: bunruija enables users to train and evaluate their models through command lines. Because all experimental settings are stored in a yaml file, users do not have to write codes.Easy to compare neural-based model with non-neural-based model: because bunruija supports models based on scikit-learn and PyTorch in the same framework, users can easily compare classification accuracies and prediction times of neural- and non-neural-based models.Easy to reproduce the training of a model: because all hyperparameters of a model are stored in a yaml file, it is easy to reproduce the model.Installpip install bunruijaExample configsExample ofsklearn.svm.SVCdata:label_column:categorytext_column:titleargs:path:data/jsonloutput_dir:models/svm-modelpipeline:-type:sklearn.feature_extraction.text.TfidfVectorizerargs:tokenizer:type:bunruija.tokenizers.mecab_tokenizer.MeCabTokenizerargs:lemmatize:trueexclude_pos:-助詞-助動詞max_features:10000min_df:3ngram_range:-1-3-type:sklearn.svm.SVCargs:verbose:falseC:10.Example of BERTdata:label_column:categorytext_column:titleargs:path:data/jsonloutput_dir:models/transformer-modelpipeline:-type:bunruija.feature_extraction.sequence.SequenceVectorizerargs:tokenizer:type:transformers.AutoTokenizerargs:pretrained_model_name_or_path:cl-tohoku/bert-base-japanese-type:bunruija.classifiers.transformer.TransformerClassifierargs:device:cpupretrained_model_name_or_path:cl-tohoku/bert-base-japaneseoptimizer:type:torch.optim.AdamWargs:lr:3e-5weight_decay:0.01betas:-0.9-0.999max_epochs:3CLI# Training a classifierbunruija-train-yconfig.yaml# Evaluating the trained classifierbunruija-evaluate-yconfig.yamlConfigdataYou can set data-related settings indata.data:label_column:categorytext_column:titleargs:# Use local data in `data/jsonl`. In this path is assumed to contain data files such as train.jsonl, validation.jsonl and test.jsonlpath:data/jsonl# If you want to use data on Hugging Face Hub, use the following args instead.# Data is from https://huggingface.co/datasets/shunk031/livedoor-news-corpus# path: shunk031/livedoor-news-corpus# random_state: 0# shuffle: truedata is loaded viadatasets.load_dataset.
So, you can load local data as well as data onHugging Face Hub.
When loading data,argsare passed toload_dataset.label_columnandtext_columnare field names of label and text.Format ofcsv:category,sentence
sports,I like sports!
…Format ofjson:[{"category","sports","text":"I like sports!"}]Format ofjsonl:{"category","sports","text":"I like suports!"}pipelineYou can set pipeline of your model inpipelinesection.
It is a list of components that are used in your model.For each component,typeis a module path andargsis arguments for the module.
For instance, when you set the first component as follows,TfidfVectorizeris instanciated with given arguments, and then applied to data at first in your model.-type:sklearn.feature_extraction.text.TfidfVectorizerargs:tokenizer:type:bunruija.tokenizers.mecab_tokenizer.MeCabTokenizerargs:lemmatize:trueexclude_pos:-助詞-助動詞max_features:10000min_df:3ngram_range:-1-3Prediction using the trained classifier in Python codeAfter you trained a classification model, you can use that model for prediction as follows:frombunruijaimportPredictorpredictor=Predictor.from_pretrained("output_dir")whileTrue:text=input("Input:")label:list[str]=predictor([text],return_label_type="str")print(label[0])output_diris a directory that is specified inoutput_dirin config.