The main challenges of document search fall into two areas – data preparation and ingestion, and search relevance.
In document search use cases, the body of documents (corpus) originates from user-generated or other uncurated content. This content usually contains typos or other errors, repetitions, and nonsense data. Before loading this data into a search engine, you need to curate, cleanse, and normalize the data. After the data is prepared, you need to load that data into the engine (by calling the ingestion APIs). Finally, you need a process to update the documents as they change.
The core value of document search is to retrieve documents that are relevant to the user’s query – search relevance. During retrieval, the search engine scores and sorts all matching documents via a statistical measure (BM25). BM25 uses the search term’s uniqueness crossed with their count in the matching documents. The more times the query matches more unique terms, the higher its score. You must adjust the scoring function for your particular data set; machine learning (ML) techniques help you improve your ranking. The search is only as good as the relevance of the documents it retrieves, and you want the best.