Hypencoder: Hypernetworks for Information Retrieval
Authors: name: Julian Killingback, name: Hansi Zeng, name: Hamed Zamani

The vast majority of retrieval models depend on vector inner products to
produce a relevance score between a query and a document. This naturally limits
the expressiveness of the relevance score that can be employed. We propose a
new paradigm, instead of producing a vector to represent the query we produce a
small neural network which acts as a learned relevance function. This small
neural network takes in a representation of the document, in this paper we use
a single vector, and produces a scalar relevance score. To produce the little
neural network we use a hypernetwork, a network that produce the weights of
other networks, as our query encoder or as we call it a Hypencoder. Experiments
on in-domain search tasks show that Hypencoder is able to significantly
outperform strong dense retrieval models and has higher metrics then reranking
models and models an order of magnitude larger. Hypencoder is also shown to
generalize well to out-of-domain search tasks. To assess the extent of
Hypencoder's capabilities, we evaluate on a set of hard retrieval tasks
including tip-of-the-tongue retrieval and instruction-following retrieval tasks
and find that the performance gap widens substantially compared to standard
retrieval tasks. Furthermore, to demonstrate the practicality of our method we
implement an approximate search algorithm and show that our model is able to
search 8.8M documents in under 60ms.