break up into parts and provide a means to skip a full re-index
update doc to include --since