added method i was using for indexing
refacting to make it easier to override in corpora