Learning to Give Checkable Answers with Prover-Verifier Games

1 INTRODUCTION . The astonishing performance of today ’ s dominant learning paradigm – optimizing powerful differentiable function approximators to minimize suitable loss functions – often comes at the cost of poor robustness and reliability . It is common for powerful deep learning systems to be vulnerable to adversarial attacks ( Goodfellow et al. , 2015 ) , to display erratic behaviour on out-of-distribution data and be very confident of wrong predictions ( Che et al. , 2021 ) . How can we train our learning algorithms to produce outputs that can be checked by humans or by learning algorithms that are cheaper or better understood ? Over the past decades , the field of computational complexity has greatly expanded our notion of proof to include formalisms such as interactive , zero-knowledge , and probabilistically checkable proofs ( Goldreich , 2008 ) . Most of these notions can be thought of in terms of a game between a powerful but untrusted prover and a computationally limited but trusted verifier . Taking inspiration from this , we propose the Prover-Verifier Game ( PVG ) , where two learning agents play the role of prover and verifier , and are allowed to converse . The verifier aims to determine the correct answer to a decision problem , and the prover aims to convince the verifier of a particular answer ( regardless of its correctness ) . Since the prover is untrustworthy , the verifier will only find its messages useful to the extent that it can independently verify the information . If all goes well , then the game dynamics will lead to a proof protocol which , if not mathematically sound , is at least sufficient for the whole system to achieve more reliable predictions than the verifier can achieve unaided . We analyze the desirability of the game equilibria implied by several variants of the Prover-Verifier Game concept and narrow the space down to a subset of games that theoretically have the desired equilibria . Picking the right game equilibrium concept turns out to be essential : we prove that formulating the PVG as a sequential game in which the prover agent plays first leads to dysfunctional solutions . On the other hand , the simultaneous ( connected to Nash equilibria ) and verifier-first sequential formulations ( connected to verifier-leading Stackelberg equilibria ) have desirable equilibria . We formally show , on a illustrative problem , that gradient based differentiable game optimizers can find desirable proof-verification protocols . Do reliable justification protocols emerge in practice if we let artificial agents play the Prover-Verifier Game ? To complement our novel game formulation , we develop a rigorous evaluation methodology whereby the verifier is frozen and the prover ( or the message ) is optimized to convince the verifier . We then run simulations on two algorithmic tasks using a practical instantiation of the PVG concept . As predicted by our theory , PVG-trained verifiers learn to receive useful and reliable information from untrusted provers by following sensible verification protocols , whereas those trained alongside fully collaborative provers are easily deceived . 2 BACKGROUND . 2.1 INTERACTIVE PROOF SYSTEMS ( IPS ) . Interactive proof systems generalize the notion of a mathematical proof to a dialogue between two agents - a prover and a verifier - to solve a decision problem ( Arora & Barak , 2009 ; Thaler , 2019 ) . The verifier agent , who is trustworthy but computationally constrained , is tasked with producing a correct answer to the decision problem . It can exchange messages with a computationally unbounded , yet potentially adversarial prover agent . The communication protocol between the prover and verifier constitutes an interactive proof system if and only if it is sound and complete : Definition 1 ( Completeness and Soundness ) . The verifier of an interactive proof system is complete iff there exists a prover that can always convince the verifier that the answer is “ yes '' , if the correct answer is “ yes '' . It is sound iff there doesn ’ t exist a prover that can trick the verifier into answering “ yes '' if the correct answer is “ no '' . 2.2 DIFFERENTIABLE GAME OPTIMIZATION . A two-player differentiable game consists of two agents whose strategies are parametrized by w = ( w1 , w2 ) 2 Rd that take turns to minimize their differentiable loss functions ( L1 , L2 ) : Rd ! R. An equilibrium concept determines which strategies will be adopted by the players . A Nash equilibrium is achieved when no player can unilaterally improve its objective function . Definition 2 ( Nash Equilibrium ( Von Neumann & Morgenstern , 2007 ) ) . The strategies parametrized by ( w⇤1 , w ⇤ 2 ) constitute a Nash equilibrium 1 of the two-player sequential differentiable game with loss functions ( L1 , L2 ) : Rd ! R if they minimize their loss functions keeping the other player ’ s parameters fixed : w⇤1 = argmin w1 L1 ( w1 , w⇤2 ) , w⇤2 = argmin w2 L2 ( w⇤1 , w2 ) ( 1 ) The notion of equilibrium considered by Generative Adversarial Networks ( Goodfellow et al. , 2014 ) is an example of a Nash Equilibrium . A Stackelberg equilibrium differs from the Nash equilibrium in that one of the players is deemed the “ leader '' and the other the “ follower '' ( Wang * et al. , 2020 ) . It is assumed that the follower always picks the optimal strategy for a given leader strategy . In response , the leader modifies its strategy by factoring in how the follower agent will respond to the modification . Definition 3 ( Stackelberg Equilibrium ( Fiez et al. , 2020 ) ) . Let w1 parametrize the strategy of the “ leader agent '' and w2 parametrize that of the “ follower agent '' . The loss functions of the agents are L1 and L2 respectively . The strategies parametrized by ( w1 , w2 ) constitute a Stackelberg equilibrium1 if and only if ( 1 ) the follower ’ s strategy is optimal given the leader ’ s strategy , and ( 2 ) the leader ’ s strategy is optimal taking into consideration how the follower will respond to modifications . w⇤1 = argmin w1 L1 ( w1 , w⇤2 ( w1 ) ) , w⇤2 ( w1 ) = argmin w2 L2 ( w⇤1 , w2 ) ( 2 ) 3 PROVER-VERIFIER GAME . The Prover-Verifier Game aims to learn decision rules with a reliable internal verification step . We describe what we mean for a verification protocol to be reliable , then outline the game formulation . 3.1 DESIDERATA AND PROVER-VERIFIER INCENTIVES . Desiderata : Designing the right prover-verifier game requires precisely defining what a desirable outcome/equilibrium of the game looks like . With the “ completeness '' and “ soundness '' definitions in mind ( Section 2.1 ) , we list the properties we seek in a desirable verifier protocol : 1We assume uniqueness of equilibria for simplicity . • Possibility of Perfect Recall : There should exist a prover that can help the verifier achieve perfect recall — the ratio of true positive predictions to all positive examples . • Guarantee of Perfect Precision : There shouldn ’ t exist any prover which can trick the verifier into achieving non-perfect precision — the ratio of true positive predictions to all positive predictions . “ Possibility of perfect recall '' is connected to completeness , and implies that with the right proofs , the verifier can achieve zero false negative rate . “ Guarantee of perfect precision '' is related to soundness and implies that the verifier always has zero false positive rate regardless of which proof is used . Picking the right prover-verifier incentives : We propose to set up the prover-verifier game with the following incentive structure : the verifier is incentivized to always give the correct answer to the given decision problem , and the prover is incentivized to get the verifier to answer “ yes '' to the decision problem , regardless of the correct answer . This structure encourages collaborative dynamics when the correct answer to the decision problem is “ yes '' ( which is linked to recall ) and encourages adversarial dynamics if the correct answer is “ no '' ( which is linked to precision ) . As we will prove formally in Section 3.3 , this incentive structure , when embedded in the right game setup with the right loss functions , can lead to a proof-verification protocol that meets the desiderata . 3.2 DIFFERENT PROVER-VERIFIER GAME FORMULATIONS . Notation Let x ⇠ pX ( x ) , y ⇠ pY ( y ) and y0 ⇠ pY 0 ( y0 ) be the input , verifier label and prover label random variables where we have x 2 Rnx , y 2 { 0 , 1 , ... , K } and y0 2 { 0 , 1 , ... , K } . Let their joint distribution be pD . The prover agent Pwp , parametrized by wp 2 Rnp , outputs the conditional probability distribution pp ( z|x ) over real valued “ message '' vectors z 2 Rnz conditioned on inputs . The verifier agent Vwv , parametrized by wv 2 Rnv , represents the conditional probability distribution pv ( ŷ|x , z ) over the predicted labels ŷ 2 { 0 , 1 , ... , K } , conditioned on the inputs x and messages z . Loss Functions : We pick the following loss functions for the verifier and prover : Lv = E ( x , y , y0 ) ⇠pD , z⇠pp ( z|x ) [ log pv ( y|x , z ) ] ( 3 ) Lp = E ( x , y , y0 ) ⇠pD , z⇠pp ( z|x ) [ log pv ( y 0|x , z ) ] ( 4 ) Since we wish for the verifier agent to try and solve the problem to the best of its ability , we set pY to be the correct label distribution . Since we wish to incentivize the prover to try and defend a particular answer regardless of its correctness , we set pY 0 ( y0 ) = 1 where y0 2 { 0 , 1 , . . . , K } is the output that the prover is defending . In a decision problem , y0 will be one of 0 or 1 . One can consider variations on these loss functions by replacing the negative log with other functions that are convex and monotonically increasing . We prefer the aforementioned loss functions when the verifier agent represents a softmax policy over its outputs . This guarantees that the gradients vanish iff the agents fulfill their incentives ( see Appendix A ) . Suitable Equilibrium concepts : There are three obvious ways to set up the order in which the agents play ( i.e . pick their strategies parametrized by wp and wv ) the Prover-Verifier Game : 1 ) The prover and verifier play simultaneously , 2 ) The prover plays first , 3 ) The verifier plays first . The simultaneous setup leads to picking Nash equilibrium as the equilibrium concept . The latter two lead to picking Stackelberg equilibrium , where prover or verifier is the leader and can therefore reason about how the opponent will respond to a given strategy . Interestingly , as we show in Section 3.3 , not all of these formulations lead to equilibria that satisfy our desiderata . When Instances are Revealed : We can also arrive at different game formulation depending on whether the problem instances ( i.e. , the input and prover-verifier labels ) are revealed to the agents before or after they pick their strategies ( i.e . weights ) . This leads to eight different game formulations : two from the simultaneous setup ( before or after the simultaneous moves ) and six from the sequential setup ( before , in-between and after the moves , for each of the two sequential formulations ) . How the Prover and Verifier Interact : The design space of how the prover and verifier interact with each other is large . We only consider the single-step , no feedback formulation throughout the paper and leave the analysis of other formulations as future work . Note that the single-step , no feedback formulation is related to NP proof systems and include sophisticated proof-verification protocols . We can also derive numerous PVG formulations by modifying the communication channel between the prover and verifier . Various decisions regarding the channel include : 1 ) If the channel is stochastic or deterministic ( and what kind of noise there is ) 2 ) Whether prover ’ s messages are real valued vectors , or discrete tokens 3 ) Whether the prover and verifier use an already constructed communication protocol , such as natural language .

Summary: The paper explores the idea of learning a game-theory-inspired prover-verifier system to augment neural networks with verifiable predictions.   In this paper, the authors set up a differentiable prover-verifier game and establish conditions on the formulation of the game to ensure the learned verifier satisfies the soundness and completeness constraints. The paper has the following results: - All verifier leading Stackelberg equilibria in a verifier-leading sequential PVG formulation in which the problem instance is revealed after the verifier picks its strategy give a desirable proof-verification protocol. - For the BEC-based prover-verifier system, the paper also guarantees that alternate gradient descent ascent converges to equilibrium under suitable learning rates verifier-leading PVG formulations. - Empirical evidence from the BEC experiments and the FindThePlus experiments show the effectiveness of the learning scheme in a practical setting.