ARTICLE

Toward Document Understanding for
Information Retrieval
Mostafa Dehghani
University of Amsterdam
dehghani@uva.nl
Abstract
Document understanding for the purpose of assessing the relevance of a document or
passage to a query based only on the document content appears to be a familiar goal for
information retrieval community, however, this problem has remained largely intractable,
despite repeated attacks over many years. This is while people are able to assess the relevance quite well, though unfamiliar topics and complex documents can defeat them. This
assessment may require the ability to understand language, images, document structure,videos, audio, and functional elements. In turn, understanding of these elements is built on
background information about the world, such as human behavior patterns, and even more
fundamental truths such as the existence of time, space, and people. All this comes naturally
to people, but not to computers!
Recently, large-scale machine learning has altered the landscape. Deep learning has
greatly advanced machine understanding of images and language. Since document and query
understanding incorporate these elements, deep learning can hold great promise. But it
comes with a drawback: general purpose representations (like CNNs for images) have proved
somewhat elusive for text. In particular, embeddings act as a distributed representation
not just of semantic information but also application-specific learning, which are hard totransfer. In short, conditions seem right for a renewed attempt on the fundamental document
understanding problem.

1

What is document understanding?

In order to think about the way we can approach this problem, I think we should first answer
some questions: Can we understand documents? What is “understanding”? Getting at thetrue meaning of a document? Okay, but then what is “meaning“? How do we even approachsuch an ill-defined goal?
As an initial disclaimer, the goal here is not to understand documents in some abstract
sense, but rather to understand documents well enough to perform a particular task: determining whether the document is relevant to a query. This assumption simplifies theproblem. In other words, we dont need to understand a calculus tutorial well enough tocompute integrals; rather, only well enough to determine whether it is relevant for queries
such as “integrate rational functions“. Of course, we can hope that pursuing this narrow
goal will produce insights that will someday allow us to understand documents in a broader

ACM SIGIR Forum

27

Vol. 51 No. 3 December 2017

sense, but even for this narrower goal, specific problem instances seem to be extraordinarily
difficult, with a lot of challenges. As an example of such a challenge, how can we map along natural-language text into a format suitable for large-scale machine learning without
destroying useful information? For example, if we supply the text to a deep neural network
as a bag of tokens, we discard both low-level information (the characters that make up thetokens) and higher-level information (word order) before deep learning enters the picture.

2 How can a search engine successfully fake document understanding?
I am not aware of what search engines do, but I can imagine how they can successfully fakedocument understanding. Consider a search engine as a person who works in a library fullof Persian books, but he does not read or speak this language. A visitor asks for a bookabout, say, “ “. The search engine has no idea what this means. But he searches the shelves
for books with those two words, particularly in prominent places such as titles or chapter
names. He places let’s say 10 candidates on the counter. The visitor frowns at a few, butthen smiles at another and takes that one to read. Later, when someone else asks for abook about “ ”, he shows the smile-inducing book and holds back the frown-inducing ones.
After employing this strategy for some years, he becomes a pretty good recommender of
Persian books, despite having no clue what any of them actually say. Of course, he mightmishandle complex or new topics. The key idea is that he substitutes an ability to interpretand remember human reactions for the ability to read the text himself1.
This can be actually the case as search engines just want to provide users with documents
relevant to their queries, even though they cannot understand the documents themselves. Toachieve this goal, a search engine can employ very simple techniques, e.g. using keywordoriented heuristics like ensuring that all important queries terms (or synonyms) appear in
the document and prefer documents where these terms appear often, prominently, close to
one another, etc. plus favoring documents to which people react positively, with respect tothe evidence like clicks.

3 What is wrong with fake document understanding?
In order to fake understanding of documents, search engines recall human reactions to thosedocuments. The form of reaction they mostly rely upon in web ranking is clicks on search
results. This user feedback can be extremely helpful and central to the strategy I explained
before, but has problems like:
People sometimes click on engaging or higher-ranked results, but with irrelevant content.
Clicks are based on limited information. Users click on documents before reading them souser clicks are perhaps driven more strongly by the presentation of results on the search
page, than by the content of the underlying documents. Fresh content lacks clicks. Soalthough click-through information is probably the best form of user feedback that can beused for document understanding, but has serious drawbacks. In particular, understanding
documents to explain user clicks is different from understanding documents for the purpose

ACM SIGIR Forum

28

Vol. 51 No. 3 December 2017

of assessing relevance. In machine learning terms, the training data does not precisely matchthe task.

4 How can we benefit from real document understanding?
Although faking document understanding might look sufficient, but there are some particularpoints that we can benefit from real document understanding including but not limited to:
Getting benefit in ranking around fresh, recently-changed, and rarely-seen documents.
Improving performance on verbose natural language queries, which may be more common
in communications with an assistant. Classic relevance assessment mechanisms perform
relatively poor in these areas. Seeing documents from a human perspective which can leadto new features and even new applications. For instance, we can presumably save people timeby presenting results in a manner that makes clear the case for relevance. From Memorization
to Generalization I think the first step for moving from fake document understanding to thereal one would be moving from pure memorization toward generalization. A search engine
might become a good Persian librarian purely by memorizing which books people like anddislike on each topic. But he can do even better by noting patterns in the behaviors heobserves. For example, he might notice that books in a particular series always elicit afrown, no matter what the library visitor specifically requested, so he should probably stopsuggesting those. Following this, a search engine can memorize past user behavior in thehope of predicting future behavior. Going beyond this, a search engine can generalize; thatis, find patterns in past user behavior that help to anticipate behavior in new situations.
In other words, a search engine can learn not only highly-specific rules in form of “showdocument D for query Q”, but also rules that are more widely applicable.
As I mentioned, to start we can translate “understanding” as “memorizing with theability of generalization” which is explaining user behavior in terms of stimuli to whichpeople actually respond. In this light, we are currently well short of understanding, but thegoal is not hopelessly out of reach.

5

Useful document representations

When document understanding for information retrieval comes to action, in the first step,
we can think of it as “Learning Useful Document Representations for Search”. Generating
representation for documents is still one the key challenges in information retrieval. So wewant to process the document in order to produce a representation of it that preserves ourability to judge relevance while stripping away nonessential data.
Most of the time, due to efficiency reasons, this is done as a preprocessing stage where
we generate query-independent document representation and it has one straightforward motivation: processing documents online would be terribly expensive. Note that this is not tosuggest that query-dependent document representations are not useful, for instance, we cantrain a neural network which predicts the relevance of a query and a document based onwords near hits which could be both efficient and effective.
Since deep learning turned to be a tool using which we can renew our attempts forrepresenting documents, it is good to think about this question again that ”What does make

ACM SIGIR Forum29

Vol. 51 No. 3 December 2017for useful document representations?”. Of course representations that lead to better results
in search are appreciated, but it is not enough. Here, I’m going to shortly talk about somecharacteristics that I think a useful query-independent document representation for the taskof search should possess.
Good representations of documents should ideally satisfy the following properties for
maximal utility in information retrieval:
1. Semantic: The representation should be the same or similar even if the text is rewritten
in a different manner as long as it has the same meaning. In other words, it should beresilient to paraphrases.
2. Similarity/Relevance measure: A similarity measure between two document representations should be computable. This would in a sense be the inverse of a distancemeasure1. Alternatively, the similarity between a query representation and a document
representation can be considered, which is a measure of whether the intent of query iscovered by the document.
3. Composability: Composability allows us to combine representations in various ways,
for instance, compose representations at the sentence level to get a representation for aparagraph. We can further compose those to get a representation of a document. Thisway we consider documents as volumes rather than points. As another application ofcomposability, we can think of generating query representations based on its (pseudo)relevant documents, by performing pseudo-relevance feedback either online or offline.
It is particularly nice to come up with representations where compositions can be doneusing the linear combination, as it is simple to understand and play with.
4. Similarity is congruent with relevance to queries: In other words, if we haverelevance judgments for results of a query, the level of similarity between relevant
results is higher than the level of similarity between irrelevant results and irrelevant
results are dissimilar to relevant results.
There are also some additional optional properties that are nice to have:
5. Common representation for queries and documents: If queries are representablein the same space as documents, then a similarity measure between the query representation and the document representation could be indicative of relevance, although
this is hard to satisfy in practice if a point representation is used. The reason thisproperty is not necessary is that in practice, we could derive a query representationfrom properties (3) and (4): For example, we could represent each query by a suitablelinear combination of the representations of the search results from a first pass, or wecan compute offline representations of past search queries using a similar trick and try
to predict the representation for a new user query by learning from representations ofsimilar past search queries. Note that conversely, we could start with query representations and derive a document representation as well. However, a document is muchricher in terms of directly available content signals than a query, so it seems reasonable
to expect more useful representations for both documents and queries by starting fromdocument representations.
6. Transparency: The representation should be transparent and debuggable. I knowthat effective embedding representations learned using neural-based models might not
be as understandable, but having insight and a handle to prob the representations

ACM SIGIR Forum30

Vol. 51 No. 3 December 2017

would help a lot to improve them. Hierarchical: Semantics for larger and larger blocksof text might be best represented hierarchically. When writing formally, the title serves
as a high-level summary. Each paragraph should be self-coherent and support theroot concept. Lastly, each sentence should have a single ”thought.” When comparing
queries to documents, different granularity might serve different needs. For example, afactoid queries need to find a short trustworthy passage (maybe even a single sentence).
The overall topic of the page is related, but less useful than determining if the query
is satisfied (has the answer). On the other side of the spectrum, broad information
seeking (like exploratory search) probably cares more about the overall topic of thepage, than each support sentence.
7. Language agnostic: If multiple documents have the exact same semantics but are
in different languages, they should probably have a similar representation. Represents
more than just natural language: In web search, where documents are more than text,
i.e. they have structure, they include images, etc. we should be able to cover themwith a similar representation.

6Conclusion

Fully understanding documents is probably far out of reach. For example, coping withhumor and complex inferences would probably require true artificial intelligence. So, tosome extent, we must continue to “fake” document understanding through memorization
but try to increase the generalization for a long time to come. But we also need to keep
in mind that we need to move toward a direction in order to reduce our reliance on fakedocument understanding!

ACM SIGIR Forum31

Vol. 51 No. 3 December 2017