fossology/fossology — pull request #2 by wuan (2014-07-04 17:52:15+00:00)
Title: monk - full text license scanner

This pull request adds another license scanner called monk.

Monk is scanning files and compares their content to the full text of all licenses known to fossology or other text sources.

Monk is based on the following workflow:
* First both texts are tokenized word by word.
* Then both texts are compared token by token.
Monk differentiates between full matches and matches with additions/removal and provides a full diff like view of its result

Currently the performance of monks algorithm is good enough to check every source file agains every known license. Differing from the first proposal in #7065 (see sourceforge) monk can be used as a standalone license scanner and its operation does not depend on e. g. nomos results.

In addition to the monk agent we started to generalize the license browser and viewer, which was extended to show licenses found by different license scanning agents (currently nomos and monk) while browsing the file tree. We would like to gather more feedback from the community as this general license browser might replace agent specific license browsers later.