Package: nexula
Nexula (Nexus Lab)Open Source Benchmark Toolkit (Still in development). Easy, Extendable and Reproducible Toolkit for benchmarking NLP problems.
Currently still has minimum features.Expect a lot of bugs in the source code :).How to installpip install nexulaThe installation above will not install deep learning packages.
If you want to use Deep Learning, installpytorchandtorchtextmanually.OverviewThis library want to overcome the needs on searching the code of several
Machine Learning model on separate site on benchmark or testing several models.Have you ever benchmarked several machine learning models and need to
go to many websites to collect the code. After that, you need to run
and configure each of them one by one to benchmark the result. For us, this is really a pain in the neck.We want this library make us easier to benchmark and find all famous models that is ready to be benchmarked.
We also want this library EXTENDABLE (can be customized by user) and easier to REPRODUCE. We want to make sure
the library is easy to use.For now, this library is far from that dream, but we will achieve it.QuickstartSeeexamplesfolder. There will be a README.md that should guide you.CLI Command-h,--helpshowthishelpmessageandexit-rRUN_YAML,--run-yamlRUN_YAMLYamlfileasacommandofthenexula-v,--verboseAddverbosity(loggingfrom`info`to`debug`)-cCUSTOM_MODULE,--custom-moduleCUSTOM_MODULEAddcustommoduledirectory(yourcustomcodeinacode)ExampleYour working directory:sample_run.yaml
custom_nexula/custom_preprocessing.pyRun yaml and include your custom code.python -r sample_run.yaml -c custom_nexulaRun as Module/APITo be denouncedFeaturesNexula uses features mostly from:scikit-learnpytorch-lightningNexula only have these choices on how to setup the data:dataset input should be separated into train, dev, testPipelineWe separate the pipeline process into 2 stepsCreate dataloader for the input of the modelTraining and predict the modelWe separate the model type into two kindsBoomer (Shallow Learning) by using scikit-learnMillenial (Deep Learning) by using pytorch (wrapped by pytorch-lightning)Data PreprocessingLowercase (nexus_basic_preprocesser) : Lowercase the input.Data Feature Representer BoomerTF-IDF (nexus_tf_idf_representer) : Use TF-IDF vectorizer on training datasetData Feature Representer TorchTextTorchText (nexus_millenial_representer) : Use TorchText on generating sequence of text in index.Boomer ModelAll of them are imported fromscikit-learnpackages.nexus_boomer_logistic_regressionnexus_boomer_linear_svcnexus_boomer_gaussian_processnexus_boomer_random_forestnexus_boomer_ada_boostnexus_boomer_multinomial_nbnexus_boomer_quadratic_discriminantMillenial ModelAll of them are coded in this repository.nexus_millenial_ccn1d_classificationnexus_millenial_lstm_classificationRun CLIRun yaml as the process controller. Below is the yaml example. See Command Explanation.md in examples
folder on how to read the yaml.nexula_data:data_choice_type:'manual_split'data_reader_type:'read_csv'data_reader_args:train:file:'tests/dummy_data/train.csv'dev:file:'tests/dummy_data/dev.csv'test:file:'tests/dummy_data/test.csv'data_pipeline:boomer:data_representer_func_list_and_args:-process:'nexus_tf_idf_representer'nexula_train:models:-model:'nexus_boomer_logistic_regression'callbacks:-callback:'model_saver_callback'params:output_dir:'output/integration_test/'-callback:'benchmark_reporter_callback'params:output_dir:'output/integration_test/'Customizable and ExtendableFor every step in the pipeline, you can specify your own process.
You must extend the abstract class innexula.nexula_inventory.inventory_base.fromnexula.nexula_inventory.inventory_baseimportNexusBaseDataInventoryimportnumpyasnpclassAddNewData(NexusBaseDataInventory):name='add_new_data2'def__init__(self,new_data_x='this is a new data',new_data_y=1,**kwargs):super().__init__(**kwargs)self.new_data_x=new_data_xself.new_data_y=new_data_yself.model=Nonedefget_model(self):returnself.modeldef__call__(self,x,y,fit_to_data=True,*args,**kwargs):"""Lowercase the textParameters----------xyfit_to_dataargskwargsReturns-------"""x=np.concatenate(x,[self.new_data_x])y=np.concatenate(y,[self.new_data_y])returnx,yYour preprocessing can be included into yaml (innexula_datapart)nexula_data:data_choice_type:'manual_split'data_reader_type:'read_csv'data_reader_args:train:file:'tests/dummy_data/train.csv'dev:file:'tests/dummy_data/dev.csv'test:file:'tests/dummy_data/test.csv'data_pipeline:boomer:data_preprocesser_func_list_and_args:-process:'add_new_data2'params:init:new_data_x:'testing'new_data_y:0data_representer_func_list_and_args:-process:'nexus_tf_idf_representer'CallbacksModel Saver (model_saver_callback) : Save the model after fitting into the training datasetBenchmark Reporter Callback (benchmark_reporter_callback) : Output the benchmark result. The benchmark result contains:Metrics choice (currently only supports F1 Score and Accuracy Score)Inference runtimeTraining runtimeThey are also extendable!End