Title: DISSECTING AN ADVERSARIAL FRAMEWORK FOR IN-FORMATION RETRIEVAL

Abstract: Recent advances in Generative Adversarial Networks facilitated by improvements to the framework and successful application to various problems has resulted in extensions to multiple domains. IRGAN attempts to leverage the framework for Information-Retrieval (IR), a task that can be described as modeling the correct conditional probability distribution p(d|q) over the documents (d), given the query (q). The work that proposes IRGAN claims that optimizing their minimax loss function will result in a generator which can learn the distribution, but their setup and baseline term steer the model away from an exact adversarial formulation, and this work attempts to point out certain inaccuracies in their formulation. Analyzing their loss curves gives insight into possible mistakes in the loss functions and better performance can be obtained by using the co-training like setup we propose, where two models are trained in a co-operative rather than an adversarial fashion.

1. INTRODUCTION:

Information-Retrieval (IR) involves providing a list of ranked documents {d 1 , d 2 , . . . , d k } in answer to a query q. This general formulation can be extended to various tasks like web-search, where the documents are web pages and information needs are queries, content-recommendation, where the documents are items/content to suggest and queries are users, and Question-Answering, where the documents are answers and queries are questions. The retrieved list can also be viewed as a probability distribution over candidates, one example being Rank q (d i ) ≡ p(d i |q) ∝ ( 1 Rankq(di) ) l , where l is a hyperparameter. Even if the probability distribution is not explicit, it is desirable to retrieve a higher ranked document more often than a lower ranked document.

GANs were proposed as alternatives to generative models and have been shown to be capable of modeling the true data well. High dimensional settings like images and word sequences have seen some success. Given that the generator in GANs tries to model the training data's distribution, adversarial setups seem like a natural fit for IR. The learned distribution can then be used to retrieve relevant documents for incoming queries. IRGAN is a framework proposed by Wang et al. (2017) , with the hope of giving Information-Retrieval, access to the large literature of GANs.

IRGAN consists of a discriminator and a generator. Like in a typical setup, the discriminator learns to distinguish between documents produces by the real probability distribution or the real ranking and the generator's probability distribution. It increases the likelihood of the former and decreases it for the latter. The generator tries to bring its probability distribution closer to the real one so that it increases the likelihood of confusing the discriminator into believing that it is the true distribution. Ideally, equilibrium is achieved when the generator manages to rank the documents according to the true distribution.

However, the formulation and implementation of the loss function in the work seems to have a few issues. Specifically, the use of the baseline term recommended in the work results in pitting the loss functions of the discriminator and the generator directly against each other and this leads to issues that are conspicuous in the loss curves. The training starts off with a pre-trained discriminator and generator, and the performance of the generator decreases as the training proceeds, while you would actually expect the opposite. When pre-training is not used, the generator does not learn at all. This forces IRGAN to choose the generator or discriminator based on whichever has better performance, while it expected that the generator is chosen at equilibrium. Given the traction this paper has received since its inception (53 citations as of 27 th September 2018), it is important to critically analyze the work and attribute the claimed performance improvements correctly. To this end, we propose two models which outperform IRGAN on two of the three tasks and give a comparable performance on the third. They also serve as an ablation study by experimentally showing that the generator might not be playing a vital role during train or test time.

The following contributions are made in this work (Goodfellow et al. (2014) ) were proposed as an alternative to generative models (Salakhutdinov & Larochelle (2010) ) which used Markov Chains or other approximations to compute intractable probability distributions. In essence, the generator tries to model the real data distribution and the discriminator learns to differentiate between real data points and generated data points. GANs are notoriously unstable to train and works like DCGANs (Radford et al. (2015) ) and Wasserstein GAN (Arjovsky et al. (2017) ) have successfully attempted to alleviate a few issues. Nonetheless, GANs have been widely applied to various problems like image generation, text generation, cross-modal retrieval and more niche ones like Interactive Image Generation (Zhu et al. (2016) ), Text to Image (Zhang et al. (2017) ), Image to Image style transfer (Isola et al. (2017) ) and robotics (Bousmalis et al. (2017) ).

While GANs allow generation based on a random variable z, Conditional GANs (Mirza & Osindero (2014) ) partition the sample variable into two parts (z and y). y is used to denote which part of the probability distribution the generator has to generate from, and z plays the same role played in Vanilla GANs (Goodfellow et al. (2014) ). Conditional GANs dovetail with IR because y can be used to represent the query or its embedding, and in theory, the model should be able to generate the required document.

y ∼ query G(z|y) ∼ p θ (d|z, q)

We feel that an eventual adversarial formulation for IR will be similar to this in flavor.

2.2. RETRIEVAL OF IMAGE RESPONSES:

Creswell & Bharath (2016) employed Sketch-GANs for the interesting task of retrieving similar merchant seals (images) based on an input image. DCGANs (Radford et al. (2015) ) are used to generate an image, and post training, the last layer of the discriminator is popped off and the rest of it is used as an encoder. This model, however, is specifically for retrieving image responses.

3. BACKGROUND:

In the subsequent sections, D denotes the discriminator, G the generator, p true the real probability distribution over documents, φ the parameters of the discriminator, θ the parameters of the generator, d the document, q the query and r the rank of a document with respect to a query.

The equations used to train the discriminator and generator in Goodfellow et al. (2014) are the following respectively.

∇ θ d 1 m m i=1 log D x (i) + log 1 -D G z (i) ∇ θg 1 m m i=1 log 1 -D G z (i)

The discriminator minimizes the likelihood of a "generated" data point and maximizes it for a "real" data point, while the generator tries to generate data points which the discriminator thinks is "real". The two models are trained alternatively and the procedure culminates in a generator which is able to produce data which looks like the real data.

4. IRGAN FORMULATION:

This section elucidates the IRGAN formulation (Wang et al. (2017) ). Comments by the authors are in italics (in this section alone), while normal typeface is a paraphrased version of IRGAN. IRGAN is motivated by the combination of two schools of thoughts, the generative retrieval model and the discriminative retrieval model.

4.1. DISCRIMINATOR AND GENERATOR:

The generative retrieval model p θ (d|q, r) tries to sample relevant documents from a candidate pool with the aim of cloning the true probability distribution p true . The discriminative retrieval model f φ (q, d), which is a binary classifier, tries to discriminate between real and generated pairs (q, d).

Two different loss functions IRGAN-Pointwise and IRGAN-Pairwise are proposed.

4.2. IRGAN-POINTWISE:

This is called so because each data point is independently used to train, unlike in IRGAN-Pairwise where pairs of points are used. The dataset is expected to have some cue with respect to how often a document is correctly retrieved for a query, if at all.

J G * ,D * = min θ max φ N n=1 (E d∼ptrue(d|qn,r) [(log D(d|q n ))] + E d∼p θ (d|qn,r) [(log 1 -D(d|q n ))])

Note that the generator G can alternately be written as p θ (d|q n , r), which denotes the modeled probability distribution, and D(d|q) = σ(f φ (d, q)) represents the discriminator's score.

4.3. IRGAN-PAIRWISE:

In some IR problems the training data may not be a set of relevant documents for each query, but rather a set of ordered document pairs

R n = [< d i , d j > |d i d j ]

, where d i d j means that the first document is more relevant for query q n than the second document. o represents a real pair < d u , d v > and o represents a generated pair < d u , d v >. The discriminator's goal in this setting is to discriminate between o and o , with

D(o|q) = σ(f φ (d u , q) -f φ (d v , q)) J G * ,D * = min θ max φ N n=1 (E o∼ptrue(o|qn) [(log D(o|q n ))] + E o ∼p θ (o |qn) [(log 1 -D(o |q n ))])

Note the similarity between this and the previous formula. The problem with this formula is that D(o|q) is actually supposed to denote the probability that the pair o is from the real data distribution and not the probability that the pair is correctly ranked, as mentioned in the paper.

4.4. OPTIMIZING THE GENERATOR:

The generator samples documents from the candidate pool based on its belief (relevance score). This sampling has the downside that the gradients cannot be backpropagated, and policy gradients (Sutton et al. (2000) ) have to be used. As an intuition, the documents can be considered as the arms of a contextual multi-arm bandit (Auer et al. (2002) , Lu et al. (2010) ), and picking an arm can be viewed as analogous to choosing the document as relevant. The policy discovered gives us the relevance of each document and -log(1 -D(d|q)) is the reward for picking that action/document (d). Let J G represent the objective function of the generator that it has to maximize. The policy gradient (REINFORCE) can be written as the following.

∇ θ J G (q n ) = 1 K K k=1 ∇ θ log p θ (d k |q n , r) log(1 + exp(f φ (d k , q n )))

4.5. BASELINE TERM:

To reduce the variance in REINFORCE, a standard trick is to use the advantage function instead of just the reward. This does not change the optimal parameters.

log(1 + exp(f φ (d k , q n ))) -E p θ (d k |qn) [log(1 + exp(f φ (d k , q n )))]

Another baseline term that is suggested for each query is f φ (d + , q), where d + represents the positive document. This is legal because the term does not depend on the document (action). This is motivated by the belief of a larger generator score if f φ (d + , q) is large and lower if f φ (d + , q) is low. This baseline term is used in two of their three tasks and causes the violation of adversarial formulation, as we show in the following section.

5. INSIGHTS INTO IRGAN MINIMAX LOSS FUNCTION:

Having shown that the generator can be optimized using REINFORCE, we focus on the loss function and show how the baseline term exacerbates training. We consider Stochastic Gradient Descent updates for ease of illustration. Consider a triple (q, d r , d g ), where d r denotes the correct document according to the true distribution and d g denotes the generated document. The discriminator's updates are in the direction of ∇J D , with the following definition.

J D = log D(d r |q) + log(1 -D(d g |q))

With the baseline term included, the generator's updates are in the direction of ∇J G , with the following definition.

J G = log(1 -D(d r |q)) + log D(d g |q)

Since maximizing log(1 -z) with respect to z is the same as maximizing -log z, we can write the following equivalent loss functions

J D = log D(d r |q) -log D(d g |q) J G = -log D(d r |q) + log D(d g |q)

Note that this substitution is similar in principle to the substitution in Goodfellow et al. (2014) , where the motivation is to allow easier flow of gradients. It is apparent that the discriminator and the generator are optimizing directly opposite loss functions and this detrimental to the performance of the models. We provide experimental proof later that the performance improvements shown in IRGAN are mainly because of the discriminator maximizing the likelihood of the real data and not because of the generator.

6. PROPOSED MODELS:

We propose two models to compare and critically analyze performance gains facilitated by IRGAN and illustrate them in Figure 1 .

The first model increases the likelihood of the training data and decreases the likelihood of documents which are not relevant to the query but have a high score according to its own parameters. It maximizes the following, where the sampling for the second term is from a candidate pool with only negative answers (denoted by p -). Not following this will lead to undesirable updates because sampling positive documents for the second term will result in decreasing the likelihood of real data. ψ denotes the parameters of the only model. 

EQUATION

J M odel1 = max ψ1 N n=1 (E d∼ptrue(d|qn,r) [(log D(d|q n ))] + E d∼p - ψ 2 (d|qn,r) [(log 1 -D(d|q n ))]) J M odel2 = max ψ2 N n=1 (E d∼ptrue(d|qn,r) [(log D(d|q n ))] + E d∼p - ψ 1 (d|qn,r) [(log 1 -D(d|q n ))])

7. EXPERIMENTAL SETUP:

This section describes the datasets, the task and hyperparameters.

7.1. DATASETS:

We conduct experiments on three tasks, Web Search, Item Recommendation and Question Answering, using the same datasets mentioned in IRGAN. InsuranceQA by Feng et al. (2015) 7.2 TASK In Web Search, the task is to retrieve the document which is most relevant to the query. Each query on average has around 5 positive documents. In Content Recommendation, users give ratings for movies and given a user, the task is to retrieve a movie that they would probably rate high. In IRGAN, any movie retrieved for which the user rating is greater than or equal to 4 (out of a scale of 5) is considered correct. Based on the dataset statistics, around 55% of the user-movie ratings are ≥ 5. This makes the problem easy to solve. In Question Answering, every query has just one relevant document in most cases. This is thus the hardest task.

7.3. HYPERPARAMETERS:

The hyperparameters for the proposed model are the same except for absence of G Epochs, for obvious reasons. Information about hyperparameter tuning is mentioned in the Appendix. 

8. EXPERIMENTS AND DISCUSSION:

We report only the P@5 and NDCG@5 values because all other metrics follow the same trend.

8.1. WEB SEARCH:

Table 3 reports the performance of various models. As can be seen, both the Single Discriminator and the Co-training models outperform IRGAN models. The fact that each query is associated approximately with 5 positive documents provides evidence that the proposed models can perform well in sparse reward settings. This task, in contrast to the other two, has multiple relevant documents that can be retrieved for each query, making it slightly easier. Each user (query) rates a movie (document), and 55% of the entries in the train set and 56% in the test set are relevant pairs. It can be seen in Table 4 that the single discriminator model achieves only a slightly lower score, and given the small size of the dataset (943 users), it makes just 7 more mistakes when compared to IRGAN. This is not a statistically significant number, especially because the IRGAN generator is pre-initialized to a model which scores 0.34 but our model learns from scratch.

8.3. QUESTION ANSWERING:

After close correspondence with the authors of IRGAN, we obtained all the hyperparameters required for the models. Multiple random seeds were used in vain, the results in the paper for Question-Answering tasks could not be replicated. We instead mention the best results out of all random seeds. We believe that if there is some random seed which gives better performance for IR-GAN, it should do so for our model as well. The co-training model outperforms IRGAN-Pairwise. In the minimax setting, since the generator is expected to capture the real data distribution, its performance is supposed to improve and this can indirectly be seen in GANs and DCGANs where the samples generated look more and more like real-world data points. Further, a deteriorating generator implies that the discriminator's improvement in performance is only because of the first term of J D , which hints that our proposed models might be able to do better than IRGAN. The reason offered in the paper is that "A worse generator could be the result of the sparsity of document distribution, i.e., each question usually has only one correct answer". But this reason does not seem plausible, given that DCGANs have been able to model very high dimensional data, where the probability distribution is only a tiny part of the real space.

Further, the increase in performance of the discriminator in all cases is coupled with a deteriorating generator. This substantiates our claim that the discriminator and the generator are optimizing directly opposite loss functions.

Item-recommendation task is a little different from the other two tasks at hand because of a large number of positive answers. When the loss curves are plotted, though the generator's performance improves, the discriminator's loss remains high and almost constant throughout the procedure, as shown in Figure 3 . This is another indication that the performance of IRGAN is not actually because of the adversarial setup, but because of the maximization of the likelihood of the real data.

9. CONNECTIONS TO PREVIOUS WORK:

We have already shown in Section 2 that Conditional GANs are connected directly to Information Retrieval. The problem can also be viewed as a contextual multi-armed bandit problem (Li et al. In Pfau & Vinyals (2016) , a parallel is drawn between Actor-Critic algorithms (Konda & Tsitsiklis ( 2000)) and GANs. This is directly related to our work because REINFORCE (Sutton et al. ( 2000)) with a baseline can be connected to Actor-Critic algorithms when bootstrapping is used (Sutton & Barto ( 2018)). The work shows a restricted scenario which involves a stateless MDP, each action setting all the pixels of the image and cross-entropy loss instead of mean-squared Bellmann residual in which GANs are equivalent to Actor-Critic algorithms. But this equivalence holds only when the baseline term is not used so the formulation in IRGAN is not exactly equivalent to a GAN framework. Another study (Finn et al. (2016) ) draws a parallel between Inverse Reinforcement Learning (Ng et al. (2000) ) and GANs because both the methods try to "learn" the cost function to optimize for.

10. CONCLUSION AND FUTURE WORK:

The experiments performed show that IRGAN is by no means state-of-the-art on those datasets. where the likelihood of the real data is increased. We showed that the discriminator and generator are optimizing directly opposite loss functions and this is the cause of deleterious training.

The poor performance of IRGAN on Web-Search and Question Answering and only a satisfactory performance on Content-Recommendation (which has dense rewards) lead us to speculate that it does not work well in sparse reward scenarios. This is similar to a well-known problem called the Sparse Reward Reinforcement Learning. We think that a correct formulation along with established techniques from the former, like reward shaping (Ng et al. (1999) ) may lead to better performance. Newer methods like Hindsight Experience Replay (Andrychowicz et al. (2017) ) which allow models to learn both from mistakes and rewards may further ameliorate learning.

We would also like to explore in the direction of learning correct adversarial frameworks for more complex tasks like Image Retrieval and Question Answering which will involve learning end-toend trainable models. With advances in modeling sequences, this could also involve generation of documents rather than sampling them.

:



Figures and Tables:

Figure 1: Models

Figure 2: Performance curves

Figure 3: Discriminator Loss for Content-Recommendation

Further, the performance does not justify the large training time of 4 hours per generator epoch and 1 hour of discriminator epoch as opposed to 2 hours per epoch of the co-training model (11 GB GPU and Question Answering task). The shaky mathematical formulation renders the generator useless after training, and any gains in performance can be attributed directly to the first term of J D ,

We propose a model motivated by Co-training which outperforms IRGANs • We point out inaccuracies in the minimax loss function used in IRGANs • We substantiate the same by drawing conclusions from the loss curves

Table content: <table><tr><td>2 RELATED WORK</td></tr><tr><td>2.1 GENERATIVE ADVERSARIAL NETWORKS</td></tr><tr><td>Generative Adversarial Networks (GANs)</td></tr></table>

Datasets

Table content: <table><tr><td>Task</td><td>Dataset</td></tr><tr><td>Web Search</td><td>LETOR by Liu et al. (2007)</td></tr><tr><td>Item-Recommendation</td><td>Movielens</td></tr><tr><td>Question Answering</td><td/></tr></table>

Hyperparameters for IRGAN

Table content: <table><tr><td>Hyperparameter</td><td>Description</td></tr><tr><td>Learning Rate</td><td>For both generator and discriminator</td></tr><tr><td>Batch Size</td><td>Batch size for training</td></tr><tr><td>Embed Dim</td><td>Embedding dimension of query or document</td></tr><tr><td>Epochs</td><td>Number of epochs of training</td></tr><tr><td>D Epochs</td><td>Number of epochs the discriminator is trained per epoch</td></tr><tr><td>G Epochs</td><td>Number of epochs the generator is trained per epoch</td></tr><tr><td>Temperature</td><td>Temperature parameter for softmax sampling of documents</td></tr></table>

Results on LETOR dataset used in IRGAN

Table content: <table><tr><td>Model</td><td>P@5</td><td>NDCG@5</td></tr><tr><td>RankNet (Burges et al. (2005))</td><td>0.1219</td><td>0.1709</td></tr><tr><td colspan="2">LambdaRank (Burges et al. (2007)) 0.1352</td><td>0.1920</td></tr><tr><td>IRGAN-pointwise</td><td>0.1657</td><td>0.2225</td></tr><tr><td>IRGAN-pairwise</td><td>0.1676</td><td>0.2154</td></tr><tr><td>Single Discriminator</td><td>0.1676</td><td>0.2190</td></tr><tr><td>Co-training</td><td>0.1733</td><td>0.2252</td></tr><tr><td>8.2 ITEM-RECOMMENDATION</td><td/><td/></tr></table>

Results on Movielens Dataset

Table content: <table><tr><td>Model</td><td>P@5</td><td>NDCG@5</td></tr><tr><td>BPR (Rendle et al. (2009))</td><td>0.3044</td><td>0.3245</td></tr><tr><td colspan="2">LambdaFM (Yuan et al. (2016)) 0.3474</td><td>0.3749</td></tr><tr><td>IRGAN-pointwise</td><td>0.3750</td><td>0.4099</td></tr><tr><td>Single Discriminator</td><td>0.3675</td><td>0.3925</td></tr><tr><td>Co-training</td><td>0.345</td><td>0.373</td></tr><tr><td colspan="2">Table 5: P@1 on InsuranceQA</td><td/></tr><tr><td>Model</td><td>P@1</td><td/></tr><tr><td>IRGAN-Pairwise</td><td>0.616</td><td/></tr><tr><td colspan="2">Single Discriminator 0.614</td><td/></tr><tr><td>Co-training</td><td>0.623</td><td/></tr><tr><td>8.4 LOSS CURVES</td><td/><td/></tr><tr><td>(a) Web Search</td><td colspan="2">(b) Question Answering</td></tr></table>

Peer review (iclr2019):
 This paper trains an information retrieval (IR) model by contrasting the joint query-document distributions, p(q, d) with negative samples drawn from a resampling of the product of marginals, p(q) x p(d). They use a second discriminator to provide the re-weighting (I believe picking to top negative sample from the other model) and train this other model in a way that mirrors the first. They also attempt to point out some theoretical problems with a competing model, IRGAN, which uses a generator that is trying to model the joint. While I like the proposal idea, I think the paper has too many problems to warrant publication. First, the story is very disappointing. The authors phrase most of the paper as a critique of IRGAN, but this critique falls short. Really this is more of a paper about where to get negative samples when training a model of the joint (or the log-ratio in this case). Using negative samples from real data with noise contrastive estimation [1] is found in numerous works in NLP [2][3], and has gained some recent attention in the context of representation learning [4][5]. The first algorithm proposed is essentially doing a sort of ranking loss on negative samples, which mirrors similar works [6]. In fact, the generator in IRGAN could be viewed as just a parametric / adaptive negative sampling distribution in the context of NCE for the ultimate purpose of learning an estimate of the log-ratio. The most interesting thing I think of this work here is the co-training, i.e., using another model to help re-sample, and I think this idea should be explored in more detail. Second, the paper spends far too much time revisiting prior work than addressing their own model, doing more analysis, providing more insight. Third, the paper is just poorly written. The notation is confusing, some of the equations are unclear (I have no idea how "r" is used in any of this), and the arguments of the baseline in IRGAN don*t really doesn*t make any sense. Notes: P1 I don*t really follow why IRGAN is so central to this work. Good ideas aren*t difficult to motivate, especially if empirically everything works out. P2 I*m having trouble with claims, especially more recently, about GAN instability, particularly since numerous approaches [7][8] seem to have more or less solved the problem. The use of "|" in G is awfully confusing. P3 Almost 2 pages of unnecessary background P4 Why are we using "|" in functions? What*s wrong with ","? theta = .theta I don*t understand the point of the quote (in italics). What happened to "r" in all of this? The last two equations and their relationship could be more clear. You use italics, so is this supposed to be a quote? But then you have a section which attempts to show this. P5 I have no idea what*s supposed to be going on in 5). The samples from the real joint don*t factor in the generator gradient, or at least it*s absolutely not clear that this pops out of the baseline? Then you switch from log (1 - x) to - log x and there*s some claim about this violating the adversarial objective? It took me more than a few reads to figure out what the equation at the bottom of P5 is doing: is this resampling? It*s fairly unclear. [1] Gutmann, Michael U., and Aapo Hyvärinen. "Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics." [2] Mnih, Andriy, and Koray Kavukcuoglu. "Learning word embeddings efficiently with noise-contrastive estimation." [3] Mikolov, Tomas, et al. "Distributed representations of words and phrases and their compositionality." [4] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." [5] Hjelm, R. Devon, et al. "Learning deep representations by mutual information estimation and maximization." [6] Faghri, Fartash, et al. "VSE++: Improving Visual-Semantic Embeddings with Hard Negatives." [7] Miyato, Takeru, et al. "Spectral normalization for generative adversarial networks." [8] Mescheder, Lars, Andreas Geiger, and Sebastian Nowozin. "Which Training Methods for GANs do actually Converge?."