use agent to update mongodb

used agent as a worker queue.
make regexp of URLs as TRIE