1) Optimized parameters for heavy crawling. 2) Changed the crawler so that it also crawls pages which do not respect the pattern, in order to reach more pages that do respect the pattern. 3) Added two fields to every project that will allow us to get rid of the junk data around the targeted integer value for the trend both in the DB and in the web UI and in the crawler. Tested, working on a simple example (XX Comments -> XX)
Merge branch 'master' of github.com:tdubourg/universal-trending-tool