Package: chariot
chariotDeliver the ready-to-train data to your NLP model.Prepare DatasetYou can prepare typical NLP datasets through thechazutsu.Build & Run PreprocessYou can build the preprocess pipeline likescikit-learn Pipeline.Preprocesses for each dataset column are executed in parallel byJoblib.Multi-language text tokenization is supported byspaCy.Format BatchSampling a batch from preprocessed dataset and format it to train the model (padding etc).You can use pre-trained word vectors through thechakin.chariotenables you to concentrate on training your model!Installpip install chariotPrepare datasetYou can download various dataset by usingchazutsu.importchazutsufromchariot.storageimportStoragestorage=Storage("your/data/root")r=chazutsu.datasets.MovieReview.polarity().download(storage.path("raw"))df=storage.chazutsu(r.root).data()df.head(5)Thenpolarity	review
0	0	synopsis : an aging master art thief , his sup...
1	0	plot : a separated , glamorous , hollywood cou...
2	0	a friend invites you to a movie . this film wo...Storageclass manage the directory structure that followscookie-cutter datascience.Project root
  └── data
       ├── external     <- Data from third party sources (ex. word vectors).
       ├── interim      <- Intermediate data that has been transformed.
       ├── processed    <- The final, canonical datasets for modeling.
       └── raw          <- The original, immutable data dump.Build & Run PreprocessBuild a preprocess pipelineAll preprocessors are defined atchariot.transformer.Transformers are implemented by extendingscikit-learnTransformer.Because of this, the API of Transformer is familiar to you. And you can mixscikit-learn's preprocessors.importchariot.transformerasctfromchariot.preprocessorimportPreprocessorpreprocessor=Preprocessor()preprocessor\.stack(ct.text.UnicodeNormalizer())\.stack(ct.Tokenizer("en"))\.stack(ct.token.StopwordFilter("en"))\.stack(ct.Vocabulary(min_df=5,max_df=0.5))\.fit(train_data)preprocessor.save("my_preprocessor.pkl")loaded=Preprocessor.load("my_preprocessor.pkl")There is 6 type of transformers are prepared in chariot.TextPreprocessorPreprocess the text before tokenization.TextNormalizer: Normalize text (replace some character etc).TextFilter: Filter the text (delete some span in text stc).TokenizerTokenize the texts.It powered byspaCyand you can chooseMeCaborJanomefor Japanese.TokenPreprocessorNormalize/Filter the tokens after tokenization.TokenNormalizer: Normalize tokens (to lower, to original form etc).TokenFilter: Filter tokens (extract only noun etc).VocabularyMake vocabulary and convert tokens to indices.FormatterFormat (preprocessed) data for training your model.GeneratorGenrate target data to train your (language) model.Build a preprocess for datasetWhen you want to make preprocess to each of your dataset column, you can useDatasetPreprocessor.fromchariot.dataset_preprocessorimportDatasetPreprocessorfromchariot.transformer.formatterimportPaddingdp=DatasetPreprocessor()dp.process("review")\.by(ct.text.UnicodeNormalizer())\.by(ct.Tokenizer("en"))\.by(ct.token.StopwordFilter("en"))\.by(ct.Vocabulary(min_df=5,max_df=0.5))\.by(Padding(length=pad_length))\.fit(train_data["review"])dp.process("polarity")\.by(ct.formatter.CategoricalLabel(num_class=3))preprocessed=dp.preprocess(data)# DatasetPreprocessor has multiple preprocessor.# Because of this, save file format is `tar.gz`.dp.save("my_dataset_preprocessor.tar.gz")loaded=DatasetPreprocessor.load("my_dataset_preprocessor.tar.gz")Train your model with chariotchariothas feature to traing your model.formatted=dp(train_data).preprocess().format().processedmodel.fit(formatted["review"],formatted["polarity"],batch_size=32,validation_split=0.2,epochs=15,verbose=2)forbatchindp(train_data.preprocess().iterate(batch_size=32,epoch=10):model.train_on_batch(batch["review"],batch["polarity"])You can use pre-trained word vectors bychakin.fromchariot.storageimportStoragefromchariot.transformer.vocabularyimportVocabulary# Download word vectorstorage=Storage("your/data/root")storage.chakin(name="GloVe.6B.50d")# Make embedding matrixvocab=Vocabulary()vocab.set(["you","loaded","word","vector","now"])embed=vocab.make_embedding(storage.path("external/glove.6B.50d.txt"))print(embed.shape)# (len(vocab.count), 50)