bunch of changes to add intermediate steps
updated to work with elasticsearch