Refactor the indexing service to be able to better support realtime tasks

This pull request introduces many changes on the indexing service to support real time tasks.