angeloskath/php-nlp-tools — pull request #18 by yooper (2013-10-01 11:38:54+00:00)
Title: TokenTransformationInterface implementation

I did run into an issue that we need to resolve with the implementation. When the transformer changes the token to null, for example when the token is a stop word, we will need to address the null tokens in the document. This will require creating a clean up method in the Document classes to remove null tokens. I wanted to discuss implementation ideas prior to writing the code.

We can create a separate interface for clean up, or within the applyTransformation method perform the clean up of the null tokens.

Any thoughts?