Learning to Give Checkable Answers with Prover-Verifier Games

1 INTRODUCTION . The astonishing performance of today ’ s dominant learning paradigm – optimizing powerful differentiable function approximators to minimize suitable loss functions – often comes at the cost of poor robustness and reliability . It is common for powerful deep learning systems to be vulnerable to adversarial attacks ( Goodfellow et al. , 2015 ) , to display erratic behaviour on out-of-distribution data and be very confident of wrong predictions ( Che et al. , 2021 ) . How can we train our learning algorithms to produce outputs that can be checked by humans or by learning algorithms that are cheaper or better understood ? Over the past decades , the field of computational complexity has greatly expanded our notion of proof to include formalisms such as interactive , zero-knowledge , and probabilistically checkable proofs ( Goldreich , 2008 ) . Most of these notions can be thought of in terms of a game between a powerful but untrusted prover and a computationally limited but trusted verifier . Taking inspiration from this , we propose the Prover-Verifier Game ( PVG ) , where two learning agents play the role of prover and verifier , and are allowed to converse . The verifier aims to determine the correct answer to a decision problem , and the prover aims to convince the verifier of a particular answer ( regardless of its correctness ) . Since the prover is untrustworthy , the verifier will only find its messages useful to the extent that it can independently verify the information . If all goes well , then the game dynamics will lead to a proof protocol which , if not mathematically sound , is at least sufficient for the whole system to achieve more reliable predictions than the verifier can achieve unaided . We analyze the desirability of the game equilibria implied by several variants of the Prover-Verifier Game concept and narrow the space down to a subset of games that theoretically have the desired equilibria . Picking the right game equilibrium concept turns out to be essential : we prove that formulating the PVG as a sequential game in which the prover agent plays first leads to dysfunctional solutions . On the other hand , the simultaneous ( connected to Nash equilibria ) and verifier-first sequential formulations ( connected to verifier-leading Stackelberg equilibria ) have desirable equilibria . We formally show , on a illustrative problem , that gradient based differentiable game optimizers can find desirable proof-verification protocols . Do reliable justification protocols emerge in practice if we let artificial agents play the Prover-Verifier Game ? To complement our novel game formulation , we develop a rigorous evaluation methodology whereby the verifier is frozen and the prover ( or the message ) is optimized to convince the verifier . We then run simulations on two algorithmic tasks using a practical instantiation of the PVG concept . As predicted by our theory , PVG-trained verifiers learn to receive useful and reliable information from untrusted provers by following sensible verification protocols , whereas those trained alongside fully collaborative provers are easily deceived . 2 BACKGROUND . 2.1 INTERACTIVE PROOF SYSTEMS ( IPS ) . Interactive proof systems generalize the notion of a mathematical proof to a dialogue between two agents - a prover and a verifier - to solve a decision problem ( Arora & Barak , 2009 ; Thaler , 2019 ) . The verifier agent , who is trustworthy but computationally constrained , is tasked with producing a correct answer to the decision problem . It can exchange messages with a computationally unbounded , yet potentially adversarial prover agent . The communication protocol between the prover and verifier constitutes an interactive proof system if and only if it is sound and complete : Definition 1 ( Completeness and Soundness ) . The verifier of an interactive proof system is complete iff there exists a prover that can always convince the verifier that the answer is “ yes '' , if the correct answer is “ yes '' . It is sound iff there doesn ’ t exist a prover that can trick the verifier into answering “ yes '' if the correct answer is “ no '' . 2.2 DIFFERENTIABLE GAME OPTIMIZATION . A two-player differentiable game consists of two agents whose strategies are parametrized by w = ( w1 , w2 ) 2 Rd that take turns to minimize their differentiable loss functions ( L1 , L2 ) : Rd ! R. An equilibrium concept determines which strategies will be adopted by the players . A Nash equilibrium is achieved when no player can unilaterally improve its objective function . Definition 2 ( Nash Equilibrium ( Von Neumann & Morgenstern , 2007 ) ) . The strategies parametrized by ( w⇤1 , w ⇤ 2 ) constitute a Nash equilibrium 1 of the two-player sequential differentiable game with loss functions ( L1 , L2 ) : Rd ! R if they minimize their loss functions keeping the other player ’ s parameters fixed : w⇤1 = argmin w1 L1 ( w1 , w⇤2 ) , w⇤2 = argmin w2 L2 ( w⇤1 , w2 ) ( 1 ) The notion of equilibrium considered by Generative Adversarial Networks ( Goodfellow et al. , 2014 ) is an example of a Nash Equilibrium . A Stackelberg equilibrium differs from the Nash equilibrium in that one of the players is deemed the “ leader '' and the other the “ follower '' ( Wang * et al. , 2020 ) . It is assumed that the follower always picks the optimal strategy for a given leader strategy . In response , the leader modifies its strategy by factoring in how the follower agent will respond to the modification . Definition 3 ( Stackelberg Equilibrium ( Fiez et al. , 2020 ) ) . Let w1 parametrize the strategy of the “ leader agent '' and w2 parametrize that of the “ follower agent '' . The loss functions of the agents are L1 and L2 respectively . The strategies parametrized by ( w1 , w2 ) constitute a Stackelberg equilibrium1 if and only if ( 1 ) the follower ’ s strategy is optimal given the leader ’ s strategy , and ( 2 ) the leader ’ s strategy is optimal taking into consideration how the follower will respond to modifications . w⇤1 = argmin w1 L1 ( w1 , w⇤2 ( w1 ) ) , w⇤2 ( w1 ) = argmin w2 L2 ( w⇤1 , w2 ) ( 2 ) 3 PROVER-VERIFIER GAME . The Prover-Verifier Game aims to learn decision rules with a reliable internal verification step . We describe what we mean for a verification protocol to be reliable , then outline the game formulation . 3.1 DESIDERATA AND PROVER-VERIFIER INCENTIVES . Desiderata : Designing the right prover-verifier game requires precisely defining what a desirable outcome/equilibrium of the game looks like . With the “ completeness '' and “ soundness '' definitions in mind ( Section 2.1 ) , we list the properties we seek in a desirable verifier protocol : 1We assume uniqueness of equilibria for simplicity . • Possibility of Perfect Recall : There should exist a prover that can help the verifier achieve perfect recall — the ratio of true positive predictions to all positive examples . • Guarantee of Perfect Precision : There shouldn ’ t exist any prover which can trick the verifier into achieving non-perfect precision — the ratio of true positive predictions to all positive predictions . “ Possibility of perfect recall '' is connected to completeness , and implies that with the right proofs , the verifier can achieve zero false negative rate . “ Guarantee of perfect precision '' is related to soundness and implies that the verifier always has zero false positive rate regardless of which proof is used . Picking the right prover-verifier incentives : We propose to set up the prover-verifier game with the following incentive structure : the verifier is incentivized to always give the correct answer to the given decision problem , and the prover is incentivized to get the verifier to answer “ yes '' to the decision problem , regardless of the correct answer . This structure encourages collaborative dynamics when the correct answer to the decision problem is “ yes '' ( which is linked to recall ) and encourages adversarial dynamics if the correct answer is “ no '' ( which is linked to precision ) . As we will prove formally in Section 3.3 , this incentive structure , when embedded in the right game setup with the right loss functions , can lead to a proof-verification protocol that meets the desiderata . 3.2 DIFFERENT PROVER-VERIFIER GAME FORMULATIONS . Notation Let x ⇠ pX ( x ) , y ⇠ pY ( y ) and y0 ⇠ pY 0 ( y0 ) be the input , verifier label and prover label random variables where we have x 2 Rnx , y 2 { 0 , 1 , ... , K } and y0 2 { 0 , 1 , ... , K } . Let their joint distribution be pD . The prover agent Pwp , parametrized by wp 2 Rnp , outputs the conditional probability distribution pp ( z|x ) over real valued “ message '' vectors z 2 Rnz conditioned on inputs . The verifier agent Vwv , parametrized by wv 2 Rnv , represents the conditional probability distribution pv ( ŷ|x , z ) over the predicted labels ŷ 2 { 0 , 1 , ... , K } , conditioned on the inputs x and messages z . Loss Functions : We pick the following loss functions for the verifier and prover : Lv = E ( x , y , y0 ) ⇠pD , z⇠pp ( z|x ) [ log pv ( y|x , z ) ] ( 3 ) Lp = E ( x , y , y0 ) ⇠pD , z⇠pp ( z|x ) [ log pv ( y 0|x , z ) ] ( 4 ) Since we wish for the verifier agent to try and solve the problem to the best of its ability , we set pY to be the correct label distribution . Since we wish to incentivize the prover to try and defend a particular answer regardless of its correctness , we set pY 0 ( y0 ) = 1 where y0 2 { 0 , 1 , . . . , K } is the output that the prover is defending . In a decision problem , y0 will be one of 0 or 1 . One can consider variations on these loss functions by replacing the negative log with other functions that are convex and monotonically increasing . We prefer the aforementioned loss functions when the verifier agent represents a softmax policy over its outputs . This guarantees that the gradients vanish iff the agents fulfill their incentives ( see Appendix A ) . Suitable Equilibrium concepts : There are three obvious ways to set up the order in which the agents play ( i.e . pick their strategies parametrized by wp and wv ) the Prover-Verifier Game : 1 ) The prover and verifier play simultaneously , 2 ) The prover plays first , 3 ) The verifier plays first . The simultaneous setup leads to picking Nash equilibrium as the equilibrium concept . The latter two lead to picking Stackelberg equilibrium , where prover or verifier is the leader and can therefore reason about how the opponent will respond to a given strategy . Interestingly , as we show in Section 3.3 , not all of these formulations lead to equilibria that satisfy our desiderata . When Instances are Revealed : We can also arrive at different game formulation depending on whether the problem instances ( i.e. , the input and prover-verifier labels ) are revealed to the agents before or after they pick their strategies ( i.e . weights ) . This leads to eight different game formulations : two from the simultaneous setup ( before or after the simultaneous moves ) and six from the sequential setup ( before , in-between and after the moves , for each of the two sequential formulations ) . How the Prover and Verifier Interact : The design space of how the prover and verifier interact with each other is large . We only consider the single-step , no feedback formulation throughout the paper and leave the analysis of other formulations as future work . Note that the single-step , no feedback formulation is related to NP proof systems and include sophisticated proof-verification protocols . We can also derive numerous PVG formulations by modifying the communication channel between the prover and verifier . Various decisions regarding the channel include : 1 ) If the channel is stochastic or deterministic ( and what kind of noise there is ) 2 ) Whether prover ’ s messages are real valued vectors , or discrete tokens 3 ) Whether the prover and verifier use an already constructed communication protocol , such as natural language .

Summary: This paper proposes a new learning methodology for training neural networks based on Prover-Verifier Games (PVGs), which are inspired by interactive proof systems (IPS). PVG consists of two learners, which work in both collaborative and adversarial manners, and the hope is that two learners together could achieve more reliable predictions. This paper studies eight possible game instantiations, depending on player order and when the problem instance is revealed, and the theoretical analysis shows that two instantiations are superior to others, which is also confirmed by an empirical evaluation. This paper also discovers that stress-testing of verifier's robustness is a more meaningful measurement of the learning success in contrast to prediction accuracy during training.