delete parse_index, introduce generic parse_document which is used for index parsing and in parse_documents for document parsing, check if document contains --- seperator