Package: nlpkit-ml
NLPkit - Transformers for text classificationA library of scikit compatible text transformers, that are ready to be integrated in an NLP pipeline for various classification tasks.Project structure.
├── nlpkit
│   ├── __init__.py
│   └── nlp_feature_extraction
│       ├── __init__.py
│       ├── liwc
│       ├── word_embeddings_features.py
│       ├── syntax_features.py
│       ├── ner_features.py
│       ├── pos_features.py
│       ├── liwc_features.py
│       ├── text_statistics_features.py
│       └── tests
│           ├── __init__.py
│           ├── test_data
│           ├── test_liwc_feature_extraction.py
│           ├── test_ner_feature_extraction.py
│           ├── test_pos_feature_extraction.py
│           ├── test_syntax_feature_extraction.py
│           └── test_word_embeddings_feature_extraction.py
├── examples
├── README.md
├── LICENCE
├── requirements.txt
└── setup.pyGetting StartedThese instructions will get you a copy of the project up and running on your local machine.PrerequisitesPython 3.6Stanford CoreNLP Server (for some transformers)Pre-trained word vectors for the w2v transformerStanford CoreNLP Server with DockerStanford CoreNLP is required for constituency parsing, POS and NER tagging.The easiest way is to have a CoreNLP Server running is to use Docker. You can find a Dockerfile and instructions to have the server running atStanford CoreNLP Server - Docker.Word vectorsIf you don't want to train your own word embeddings, you can download pre-trained word vectors from theStanford GloVe project.
For example, the Wikipedia model has a vocabulary of 400K words, represented using 300 dimensional vectors.
The word vectors come in the GloVe format and need to be converted into the word2vec format.
While the formats are almost identical, you can use gensim to do the conversion.python -m gensim.scripts.glove2word2vec -i glove.txt -o word2vec.txtList of transformersPOSTagPreprocessor: Pre-processes text documents by tagging each word in the form of word_TAG_ e.g. what_WP. Can be used to generate POS tagged n-gramsNERPreprocessor: Pre-processes text documents by replacing named entities with generic tags e.g. PERSON, LOCATIONWordEmbedsDocVectorizer: Converts text documents to word2vec based document vector representations. It maps
the words of a document to word2vec vectors, and averages them across dimensions to produce a document vector
representationPOSExtractor: Extracts Parts of Speech (POS) counts for a collection of text documentsCFGExtractor: Extracts the Context Free Grammar (CFG) production rules found in a collection of text documentsNamedEntitiesCounter: Extracts Named Entity counts per entity type (e.g. PERSON) for a collection of text documentsLIWCExtractor: Extracts proportions of words that fall in the various LIWC categories for a collection of text documentsTextStatsExtractor: Calculates various text statistics and readability scores for a collection of text documentsUsageAll the custom transformers extend the BaseEstimator and TransformerMixin and implement the fit and transform methods.# POSExtractorsf_parser=CoreNLPParser(url='http://localhost:9000/',tagtype='pos')pos_extractor=POSExtractor(sf_parser)X=pos_extractor.fit_transform(corpus)They can also be used in pipelines e.g.pipeline=Pipeline([('pre',TextPreprocessor(stemming=False)),('w2v',WordEmbedsDocVectorizer(self._word2vec,tfidf_weights=True)),('clf',SVC(kernel='linear',C=1,probability=True))])For more, you can run the examples included in the examples folder.TestsThe Pytest framework is used for unit testing. All of the custom text transformers produced in this project come with an extensive set of unit tests.
To run the tests use:pytest srcProject repositoryhttps://github.com/evanll/nlpkit-mlAuthorWritten by Evan Lalopoulosevan.lalopoulos.2017@my.bristol.ac.ukas part of his thesis inFake News detection using NLP.Evan Lalopoulos-evanll