Abstract: Models such as Sequence-to-Sequence and Image-to-Sequence are widely used in real world applications. While the ability of these neural architectures to produce variablelength outputs makes them extremely effective for problems like Machine Translation and Image Captioning, it also leaves them vulnerable to failures of the form where the model produces outputs of undesirable length. This behaviour can have severe consequences such as usage of increased computation and induce faults in downstream modules that expect outputs of a certain length. Motivated by the need to have a better understanding of the failures of these models, this paper proposes and studies the novel output-size modulation problem and makes two key technical contributions. First, to evaluate model robustness, we develop an easy-to-compute differentiable proxy objective that can be used with gradient-based algorithms to find output-lengthening inputs. Second and more importantly, we develop a verification approach that can formally verify whether a network always produces outputs within a certain length. Experimental results on Machine Translation and Image Captioning show that our output-lengthening approach can produce outputs that are 50 times longer than the input, while our verification approach can, given a model and input domain, prove that the output length is below a certain size.

Chenglong Wang
Rudy Bunel
Edward Grefenstette
♣ Deepmind

Knowing When to Stop: Evaluation and Verification of Conformity to Output-size Specifications

Introduction Neural networks with variable output lengths have become ubiquitous in several applications. In particular, recurrent neural networks (RNNs) such as LSTMs  [17] , used work done during an internship at DeepMind | now at Facebook AI Research to form "sequence" models  [30] , have been successfully and extensively applied in in image captioning  [34, 28, 21, 6, 12, 37, 26, 24, 1] , video captioning  [33, 38, 35, 40, 41] , machine translation (MT)  [30, 9] , summarization  [10] , and in other sequence-based transduction tasks. The ability of these sequence neural models to generate variable-length outputs is key to their performance on complex prediction tasks. However, this ability also opens a powerful attack for adversaries that try to force the model to produce outputs of specific lengths that, for instance, lead to increased computation or affect the correct operation of down-stream modules. To address this issue, we introduce the output-length modulation problem where given a specification of the form that the model should produce outputs with less than a certain maximum length, we want to find adversarial examples, i.e. search for inputs that lead the model to produce outputs with a larger length and thus show that the model under consideration violates the specification. Different from existing work on targeted or untargeted attacks where the goal is to perturb the input such that the output is another class or sequence in the development dataset (thus within the dataset distribution), the outputmodulation problem requires solving a more challenging task of finding inputs such that the output sequences are outside of the training distribution, which was previously claimed difficult  [5] . The naive approach to the solution of the output-length modulation problem involves a computationally intractable search over a large discrete search space. To overcome this, we develop an easy-to-compute differentiable proxy objective that can be used with gradient-based algorithms to find output-lengthening inputs. Experimental results on Machine Translation show that our adversarial outputlengthening approach can produce outputs that are 50 times longer than the input. However, when evaluated on the image-to-text image captioning model, the method is less successful. There could have been two potential reasons for this result: the image-to-text architecture is truly robust, or the adversarial approach is not powerful enough to find adversarial examples for this model. To resolve this question, we develop a verification method for checking and formally proving whether a network is consistent with the outputsize specification for the given range of inputs. To the best of our knowledge, our verification algorithm is the first formal verification approach to check properties of recurrent models with variable output lengths. Our Contributions To summarize, the key contributions of this paper are as follows: • We propose and formulate the novel output-size modulation problem to study the behaviour of neural architectures capable of producing variable length outputs, and we study its evaluation and verification problems. • For evaluation, we design an efficiently computable differentiable proxy for the expected length of the output sequence. Experiments show that this proxy can be optimized using gradient descent to efficiently find inputs causing the model to produce long outputs. • We demonstrate that popular machine translation models can be forced to produce long outputs that are 50 times longer than the input sequence. The long output sequences help expose modes that the model can get stuck in, such as undesirable loops where they continue to emit a specific token for several steps. • We demonstrate the feasibility of formal verification of recurrent models by proposing the use of mixedinteger programming to formally verify that a certain neural image-captioning model will be consistent with the specification for the given range of inputs. Motivations and Implications Our focus on studying the output-length modulation problem is motivated by the following key considerations: • Achieving Computational Robustness: Many ML models are now offered as a service to customers via the cloud. In this context, ML services employing variable-output models could be vulnerable to denialof-service attacks that cause the ML model to perform wasteful computations by feeding it inputs that induce long outputs. This is particularly relevant for variable compute models, like Seq2Seq  [9, 30] . Given a trained instance of the model, no method is known to check for the consistency of the model with a specification on the number of computation steps. Understanding the vulnerabilities of ML models to such outputlengthening and computation-increasing attacks is important for the safe deployment of ML services. • Understanding and Debugging Models: By designing inputs that cause models to produce long outputs, it is possible to reason about the internal representations learned by the model and isolate where the model exhibits undesirable behavior. For example, we find that an English to German sequence-to-sequence model can produce outputs that end with a long string of question marks ('?'). This indicates that when the output decoder state is conditioned on a sequence of '?'s, it can end up stuck in the same state. • Uncovering security vulnerabilities through adversarial stress-testing: The adversarial approach to outputlength modulation tries to find parts of the space of inputs where the model exhibits improper behavior. Such inputs does not only reveal abnormal output size, but could also uncover other abnormalities like the privacy violations of the kind that were recently revealed by  [4]  where an LSTM was forced to output memorized data. • Canonical specification for testing generalization of variable-output models: Norm-bounded perturbations of images  [31]  have become the standard specification to test attacks and defenses on image classifiers. While the practical relevance of this particular specification can be questioned  [14] , it is still served as a useful canonical model encapsulating the essential difficulty in developing robust image classifiers. We believe stability of output-lengths can serve a similar purpose: as a canonical specification for variable outputlength models. The main difficulties in studying variable output length models in an adversarial sense (the non-differentiability of the objective with respect to inputs) are exposed in output-lengthening attack, making it a fertile testing ground for both evaluating attack methods and defenses. We hope that advances made here will facilitate the study of robustness on variable compute models and other specifications for variableoutput models such as monotonicity.
 Related Work There are several recent studies on generating adversarial perturbations on variable-output models.  [27, 20]  show that question answering and machine comprehension models are sensitive to attacks based on semantics preserving modification or the introduction of unrelated information.  [11, 39]  find that character-level classifiers are highly sensitive to small character manipulations.  [29]  shows that models predicting the correctness of image captions struggle against perturbations consisting of a single word change.  [5]  and  [8]  further study adversarial attacks for sequential-output models (machine-translation, image captioning) with specific target captions or keywords. We focus on sequence output models and analyze the output-length modulation problem, where the models should produce outputs with at least a certain number of output tokens. We study whether a model can be adversarially perturbed to change the size of the output, which is a more challenging task compared to targeted attacks (see details in Section 3). On the one hand, existing targeted attack tasks aim to perturb the input such that the output is another sequence in the validation dataset (thus within the training distribution), but attacking output size requires the model to generate out-of-distribution long sequences. On the other hand, since the desired output sequence is only loosely constrained by the length rather than directly provided by the user, the attack algorithm is required to explore the output size to make the attack possible. For models that cannot be adversarially perturbed, we develop a verification approach to show that it isn't simply a lack of power by the adversary but the sign of true robustness from the model. Similar approaches have been investigated for feedforward networks  [3, 7, 32]  but our work is the first to handle variable output length models and the corresponding decoding mechanisms.
 Modulating Output-size We study neural network models capable of producing outputs of variable length. We start with a canonical abstraction of such models, and later specialize to concrete models used in machine translation and image captioning. We denote by x the input to the network and by X the space of all inputs to the network. We consider a set of inputs of interest S, which can denote, for example, the set of "small" 1 perturbations of a nominal input. We study models that produce variable-length outputs sequentially. Let y t 2 Y denote the t-th output of the model, where Y is the output vocabulary of the model. At each timestep, the model defines a probability over the next element P (y t+1 |x, y 0:t ). There exists a special end-of-sequence element eos 2Y that signals termination of the output sequence. In practice, different models adopt different decoding strategies for generating y t+1 from the probability P (y t+1 |x, y 0:t )  [13, 19, 22] . In this paper, we focus on the commonly used deterministic greedy decoding strategy  [13] , where at each time step, the generated token is given by the argmax over the logits: y 0 = argmax {P (•|x)} (1a) y t+1 = argmax {P (•|x, y 0:t )} if y t 6 = eos (1b) Since greedy decoding is deterministic, for a given sample x with a finite length output, we can define the length of the  1  The precise definition of small is specific to the application studied. greedily decoded sequence as: `(x)=t s.t y t = eos y i 6 = eos 8i<t y i+1 = argmax {P (.|x, y 0:i )}8 i<t (2) Note that there is a unique t that satisfies the above conditions, which is precisely the first t at which y t = eos when using greedy decoding. Output length modulation specification A network is said to satisfy the output length modulation specification parameterized by S, K, if for all inputs x in S, the model terminates within K steps under greedy decoding for all x 2S, formally: 8x 2S `(x)  K (3 ) In Section 3.1, we study the problem of finding adversarial examples, i.e., searching for inputs that lead the model to produce outputs with a larger length and thus show that the model violates the specification. In Section 4, we use formal verification method to prove that a model is consistent with the specification for the given range of inputs, if such attacks are indeed impossible.
 The Output-Size Modulation Problem In order to check whether the specification, Eq. (  3 ), is valid, one can consider a falsification approach that tries to find counterexamples proving that Eq. (  3 ) is false. If an exhaustive search over S for such counterexamples fails, the specification is indeed true. However, exhaustive search is computationally intractable; hence, in this section we develop gradient based algorithms that can efficiently find counterexamples (although they may miss them even if they exist). To develop the falsification approach, we study the solution to the following optimization objective: max x2S `(x) (4) where S is the valid perturbation space. If the optimal solution x in the space S has `(x) > K, then (3) is false. The attack spaces S we consider in this paper include both continuous inputs (for image-to-text models) and discrete inputs (for Seq2Seq models).
 Continuous inputs: For continuous inputs, such as image captioning tasks, the input is an n ⇥ m image with pixel values normalized to be in the range  [ 1, 1] . x is an n ⇥ m matrix of real numbers and X =[ 1, 1] n⇥m . We define the perturbation space S(x, ) as follows: S(x, )={x 0 2X |kx 0 xk 1  } i.e., the space of perturbations of the input x in the `1 ball. Discrete inputs: For discrete inputs, e.g., machine translation tasks, inputs are discrete tokens in a language vocabulary. Formally, given the vocabulary V of the input language, the input space X is defined as all sentences composed of tokens in V , i.e., X = {(x 1 ,...,x n ) | x i 2 V, n > 0}. Given an input sequence x =( x 1 ,...,x n ), we define the -perturbation space of a sequence as all sequences of length n with at most d • ne tokens different from x (i.e., 2 [0, 1] denotes the percentage of tokens that an attacker is allowed to modify). Formally, the perturbation space S(x, ) is defined as follows: S(x, δ)={(x 0 1 ,...,x 0 n ) 2 V n | n P i=1 [xi 6 = x 0 i ] dδ • ne}
 Extending Projected Gradient Descent Attacks In the projected gradient descent (PGD) attacks  [25] , 2 given an objective function J(x), the attacker calculates the adversarial example by searching for inputs in the attack space to maximize J(x). In the basic attack algorithm, we perform the following updates at each iteration: x 0 =Π S(x,δ) (x + ↵r x J(x)) ( 5 ) where ↵>0 is the step size and Π S(x,δ) denotes the projection of the attack to the valid space S(x, ). Observe that the adversarial objective in Eq. (  4 ) cannot be directly used as J(x) to update x as the length of the sequence is not a differentiable objective function. This hinders the direct application of PGD to output-lengthening attacks. Furthermore, when the input space S is discrete, gradient descent cannot be directly be used because it is only applicable to continuous input spaces. In the following, we show our extensions of the PGD attack algorithm to handle these challenges. Greedy approach for sequence lengthening We introduce a differentiable proxy of `(x). Given an input x whose decoder output logits are (o 1 ,...,o k ) (i.e., the decoded sequence is y = (argmax(o 1 ),...,argmax(o k ))), instead of directly maximizing the output sequence length, we use a greedy algorithm to find an output sequence whose length is longer than k by minimizing the probability of the model to terminate within k steps. In other words, we minimize the log probability of the model to produce eos at any of the timesteps between 1 to k. Formally, the proxy objective J is defined as follows: J(x)= k P t=1 max ⇢ o t [eos] max z6 =eos o t [z], ✏ where ✏>0 is a hyperparameter to clip the loss. This is piecewise differentiable w.r.t. the inputs x (in the same  2  Here the adversarial objective is stated as maximization, so the algorithm is Projected Gradient Ascent, but we stick with the PGD terminology since it is standard in the literature sense that the ReLU function is differentiable) and can be efficiently optimized using PGD.
 Continuous relaxation for discrete inputs While we can apply the PGD attack with the proxy objective on the model with continuous inputs by setting the projection function Π S(x,δ) as the Euclidean projection, we cannot directly update discrete inputs. To enable a PGDtype attack in the discrete input space, we use the Gumbel trick  [18]  to reparameterize the input space to perform continuous relaxation of the inputs. Given an input sequence x =( x 1 ,...,x n ), for each x i , we construct a distribution ⇡ i 2 R |V | initialized with ⇡ i [x i ]=1and ⇡ i [z]= 1 for all z 2 V \{x i }. The softmax function applied to ⇡ i is a probability distribution over input tokens at position i with a mode at x i . With this reparameterization, instead of feeding x =( x 1 ,...,x n ) into the model, we feed the Gumbel softmax sampling from the distribution (u 1 ,...,u n ). The sample x =(x 1 ,...,x n ) is calculated as follows: u i ⇠ Uniform(0, 1); g i = log( log(u i )) p = softmax(⇡); xi = softmax( gi+log pi τ ) where ⌧ is the Gumbel-softmax sampling temperature that controls the discreteness of x. With this relaxation, we perform PGD attack on the distribution ⇡ at each iteration. Since ⇡ i is unconstrained, the projection step in (  5 ) is unnecessary. When the final ⇡ 0 =( ⇡ 0 1 ,...,⇡ n ) is obtained from the PGD attack, we draw samples x 0 i ⇠ Categorical(⇡ i ) to get the final adversarial example for the attack.
 Verified Bound on Output Length While heuristics approaches can be useful in finding attacks, they can fail due to the difficulty of optimizing nondifferentiable nonconvex functions. These challenges show up particularly when the perturbation space is small or when the target model is trained with strong bias in the training data towards short output sequences (e.g., the Show-and-Tell model as we will show in Section 6). Thus, we design a formal verification approach for complete reasoning of the output-size modulation problem, i.e., finding provable guarantees that no input within a certain set of interest can result in an output sequence of length above a certain threshold. Our approach relies on counterexample search using intelligent brute-force search methods, taking advantage of powerful modern integer programming solvers  [15] . We encode all the constraints that an adversarial example should satisfy as linear constraints, possibly introducing additional binary variables. Once in the right formalism, these can be fed into an off-the-shelf Mixed Integer Programming (MIP) solver, which provably solves the problem, albeit with a potentially large computational cost. The constraints consist of four parts: (1) the initial restrictions on the model inputs (encoding S(x, )),  (2)  the relations between the different activations of the network (implementing each layer), (3) the decoding strategy (connection between the output logits and the inputs at the next step), and (4) the condition for it being a counterexample (ie. a sequence of length larger than the threshold). In the following, we show how each part of the constraints is encoded into MIP formulas. Our formulation is inspired by pior work on encoding feed-forward neural networks as MIPs  [3, 7, 32] . The image captioning model we use consists of an image embedding model, a feedforward convolutional neural network that computes an embedding of the image, followed by a recurrent network that generates tokens sequentially starting with the initial hidden state set to the image embedding. The image embedding model is simply a sequence of linear or convolutional layers and ReLU activation functions. Linear and convolutional layers are trivially encoded as linear equality constraints between their inputs and outputs, while ReLUs are represented by introducing a binary variable and employing the big-M method  [16] : x i = max (x i , 0) ) i 2{0, 1},x i 0 (6a) x i  u i • i ,x i xi (6b) x i  xi l i • (1 i ) (6c) with l i and u i being lower and upper bounds of xi which can be obtained using interval arithmetic (details in  [3] ). Our novel contribution is to introduce a method to extend the techniques to handle greedy decoding used in recurrent networks. For a model with greedy decoding, the token with the most likely prediction is fed back as input to the next time step. To implement this mechanism as a mixed integer program, we employ a big-M method  [36] : o max = max y2Y (o y ) ) o max o y , y 2{0, 1}8 y 2Y (7a) o max  o y +(u l y )(1 y ) 8y 2Y (7b) X y2Y y =1 (7c) with l y ,u y being a lower/upper bound on the value of o y and u = max y2Y u y (these can again be computed using interval arithmetic). Implementing the maximum in this way gives us both a variable representing the value of the maximum (o max ), as well as a one-hot encoding of the argmax ( y ). If the embedding for each token is given by {emb i | i 2Y}, we can simply encode the input to the following RNN timestep as P y2Y y • emb y , which is a linear function of the variables that we previously constructed. With this mechanism to encode the greedy decoding, we can now unroll the recurrent model for the desired number of timesteps. To search for an input x with output length `(x) K, we unroll the recurrent network for K steps and attempt to prove that at each timestep, eos is not the maximum logit, as in  (2) . We setup the problem as: max min t=1.. K  max z6 =eos o t [z] o t [eos] (8) where o(k) represents the logits in the k-th decoding step. We use an encoding similar to the one of Equation (  7 ) to represent the objective function as a linear objective with added constraints. If the global optimal value of Eq. (  8 ) is positive, this is a valid counterexample: at all timesteps t 2 [1.. K], there is at least one token greater than the eos token, which means that the decoding should continue. On the other hand, if the optimal value is negative, that means that those conditions cannot be satisfied and that it is not possible to generate a sequence of length greater than K. The eos token would necessarily be predicted before. This would imply that our robustness property is True.
 Target Model Mechanism We use image captioning and machine translation models as specific target examples to study the output length modulation problem. We now introduce their mechanism.
 Image captioning models The image captioning model we consider is an encoder-decoder model composed of two modules: a convolution neural network (CNN) as an encoder for image feature extraction and a recurrent neural network (RNN) as a decoder for caption generation  [34] . Formally, the input to the model x is an m ⇥ n sized image from the space X =[ 1, 1] m⇥n , the CNN-RNN model computes the output sequence as follows: i 0 = CNN(x); h 0 = 0 o t ,h t+1 = RNNCell(i t ,h t ) y t = arg max(o t ); i t+1 = emb(y t ) where emb denotes the embedding function. The captioning model first run the input image x through a CNN to obtain the image embedding and feed it to the RNN as the initial input i 0 along with the initial state h 0 . At each decode step, the RNN uses the input i t and state h t to compute the new state h t+1 as well as the logits o t representing the log-probability of the output token distribution in the vocabulary. The output y t is the token in the vocabulary with highest probability based on o t , and it is embedded into the continuous space using an embedding matrix W emb as W emb [y t ]. The embedding is fed to the next RNN cell as the input for the next decoding step.
 Machine translation models The machine translation model is an encoder-decoder model  [30, 9]  with both the encoder and the decoder being RNNs. Given the vocabulary V in of the input language, the valid input space X is defined as all sentences composed of tokens in V in , i.e., X = {(x 1 ,...,x n ) | x i 2 V, n > 0}. Given an input sequence x =( x 1 ,...,x n ), the model first calculates its embedding f (x) RNN as follows (h e t and i e t denote the encoder hidden states and the inputs at the t-th time step, respectively. emb e denotes the embedding function for each token in the vocabulary). The model then uses f (x) as the initial state h 0 for the decoder RNN to generate the output sequence, following the same approach as in the image captioning model. h e 0 = 0; i e t = emb e (x t ) h e t = RNNCell e (i e t ,h e t 1 ); f (x)=h e n
 Experiments We consider the following three models, namely, Multi-MNIST captioning, Show-and-Tell  [34] , and Neural Machine Translation (NMT) [30, 9, 2] models.
 Details of models and datasets Multi-MNIST. The first model we evaluate is a minimal image captioning model for Multi-MNIST dataset. The Multi-MNIST dataset is composed from the MNIST dataset (Figure  1 left ). Each image in the dataset is composed from 1-3 MNIST images: each MNIST image (28 * 28) is placed on the canvas of size (28 * 112) with random bias on the x-axis. The composition process guarantees that every MNIST image is fully contained in the canvas without overlaps with other images. The label of each image is the list of MNIST digits appearing in the canvas, ordered by their x-axis values. The dataset contains 50,000 training images and 10,000 test images, where the training set is constructed from MNIST training set and the test set is constructed from MNIST test set. The images are normalized to [ 1, 1] before feeding to the captioning model. For this dataset, we train a CNN-RNN model for label prediction. The model encoder is a 4-layers CNN (2 convolution layers and 2 fully connected layers with ReLU activation functions applied in between). The decoder is a RNN with ReLU activation. Both the embedding size and the hidden size are set to 32. We train the model for 300 steps with Adam optimizer based on the cross-entropy loss. The model achieves 91.2% test accuracy, and all predictions made by the model on the training set have lengths no more than 3. Show-and-Tell. Show and Tell model  [34]  is an image captioning model with CNN-RNN encoder-decoder architecture similar to the Multi-MNIST model trained on the MSCOCO 2014 dataset  [23] . Show-and-Tell model uses Inception-v3 as the CNN encoder and an LSTM for caption generation. We use a public version of the pretrained model 3 for evaluation. All images are normalized to  [ 1, 1]  3 https://github.com/tensorflow/models/ and all captions in the dataset are within length 20. NMT. The machine translation model we study is a Seq2Seq model  [30, 9]  with the attention mechanism  [2]  trained on the WMT15 German-English dataset. The model uses byte pair segmentation (BPE) subword units  [28]  as vocabulary. The input vocabulary size is 36, 548. The model consists of 4-layer LSTMs of 1024 units with a bidirectional encoder, with the embedding dimension set to 1024. We use a publicly available checkpoint 4 with 27.6 BLEU score on the WMT15 test datasets in our evaluation. At training time, the model restricts the maximum decoding length to 50.
 Adversarial Attacks Our first experiment studies whether adversarial inputs exist for the above models and how they affect model decoding. For each model, we randomly select 100 inputs from the development dataset as attack targets, and compare the output length distributions from random perturbation and PGD attacks. Multi-MNIST We evaluate the distribution of output lengths of images with an `1 perturbation radius of 2{ 0.001, 0.005, 0.01, 0.05, 0.1, 0.5} using both random search and PGD attack. In random search, we generate 10,000 random images within the given perturbation radius for each image in the target dataset as new inputs to the model. In PGD attack, the adversarial inputs are obtained by running 10,000 gradient descent steps with an learning rate of 0.0005 using the Adam Optimizer. Neither of the attack methods can find any adversarial inputs for 2{0.001, 0.005, 0.01} perturbation radius (i.e., no perturbation is found for any images in the target dataset within the above to generate an output sequence longer than the original one). Figure  2  shows the distribution of the output lengths for images with different perturbation radius. Results show that the PGD attack is successful at finding attacks that push the distribution of output lengths higher, particularly at larger values of . Examples of adversarial inputs found by the model are shown in Figure  1 .
 Show-and-Tell For the Show-and-Tell model, we generate attacks within an `1 perturbation radius of =0.5 with both random search and PGD attack on 500 images randomly selected from the development dataset. However, except one adversarial input found by PGD attack that would cause the model to produce an output with size 25, no other adversarial inputs are found that can cause the model to produce outputs longer than 20 words, which is the training length cap. Our analysis shows that the difficulty of attacking the model is resulted from its strong bias on the output  sequence distribution and the saturation of sigmoid gates in the LSTM decoder. This result is also consistent with the result found by  [5]  that Show-and-Tell model is "only able to generate relevant captions learned from the training distribution". NMT We evaluate the NMT model by comparing the output length distribution from adversarial examples generated from random search and PGD attack algorithms. We randomly draw 100 input sentences from the development dataset. The maximum input length is 78 and their corresponding translations made by the model are all within 75 tokens. We consider the perturbation 2{0.3, 0.5, 1.0}. 1. Random Search. In each run of the random attack, given an input sequence with length n, we first randomly select d • ne locations to modify, then randomly select substitutions of the tokens at these locations from the input vocabulary, and finally run the NMT model on the modified sequence. We run 10,000 random search steps for the 100 selected inputs, and show the distributions of all outputs obtained from the translation (in the total 1M output sequences). 2. PGD Attack. In PGD attack, we also start by randomly selecting d • ne locations to modify for each input sequence with length n. We then run 800 iterations of PGD attack with Adam optimizer using an initial learning rate of 0.005 to find substitutions of the tokens at these selected locations. We plot the output length obtained from running these adversarial inputs through the translation model. Figure  3  shows the distribution of output sequence lengths obtained from random search methods with different . We aggregate all sequences with length longer than 100 into the group '>100' in the plot. Results show that even random search approach could often craft inputs such that the corresponding output lengths are more than 75 and occasionally generates sentences with output length over 100. The random search algorithm finds 79, 11, 3 for =0.3, 0.5, 1, respectively, among the 1M translations that are longer than 100 tokens (at small , the search space is more restricted, and random search has a higher success rate of finding long outputs). Notably, the longest sequence found by the random search is a sequence with output length 312 tokens, where the original sequence is only 6.     to larger perturbations. With an unconstrained perturbation = 100%, PGD attack algorithm discovers more adversarial inputs whose outputs are longer than 100 tokens (10% among all attacks), which is 1000⇥ more often than random search. As an extreme case, PGD attack discovered an adversarial input with length 3 whose output length is 575. Examples of adversarial inputs and their corresponding model outputs are shown in Figure  5  and the Appendix; we find out that a common feature of the long outputs produced by the translation model is that the output sequences often end with long repetitions of one (or a few) words.
 names name names name names grammatically name names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names names eos To analyze the bottleneck of PGD attack on the NMT model, we further run a variation of the PGD attack where the attack space is the (continuous) word embedding space as opposed to the (discrete) token space: we allow the attacker to directly modify token embeddings at selected attack locations to any other vector. PGD attack on this variation achieves a 100% success rate to find adversarial token embeddings such that the model outputs are longer than 500 tokens. This indicates that the discrete space is a bottleneck for consistently finding stronger attacks.
 Verification Our implementation of the verification algorithm using the mixed integer programming (8) is implemented using SCIP  [15] . We run our verification algorithm on the Multi-Figure  6 . Proportion of samples being provably robust (in blue), vulnerable (in red) or of unknown robustness status (in gray) to an attack attempting to make the model generate an output sequence longer than the ground truth, as a function of the perturbation radius allowed to the attacker. For small radiuses, the MIP can prove that no attacks can be successful. For large radiuses, we are able to find successful attacks. MNIST dataset, attempting to formally prove the robustness of the model to attacks attempting to generate an output longer than the ground truth. For each input image, we set a timeout of 30 minutes for the solver. The results in Figure  6  show that our verification algorithm is able to verify formally that no attacks exists for small perturbation radiuses. However, as the perturbation radius increases, our verification algorithm times out and is not able to explore the full space of valid perturbations and thus cannot decide whether attacks exists in the given space. For this reason, the number of robust samples we report is only a lower bound on the actual number of robust samples. Conversely, the vulnerable samples that we exhibit give us an upper bound on the number of those robust samples. As shown by the large proportion of samples of unknown status, there is currently still a gap between the capabilities of formal verification method and attacks.
 Conclusion In this paper, we introduce the existence and the construction of the output-length modulation problem. We propose a differentiable proxy that can be used with PGD to efficiently find output-lengthening inputs. We also develop a verification approach to formally prove certain models cannot produce outputs greater than a certain length. We show that the proposed algorithm can produce adversarial examples that are 50 times longer than the input for machine translation models, and the image-captioning model can conform the output size is less than certain maximum length using the verification approach. In future work, we plan to study adversarial training of sequential output models using the generated attacks, to models that are robust against output lengthening attacks, and further, verify this formally. Figure 1 . 1 Figure 1. Multi-MNIST examples (left), adversarial examples found by PGD attack (mid), and their differences. For the first group, the model correctly predicts label l1 =[6, 1] on the original image but predicts l 0 1 =[6, 1, 1] for its corresponding adversarial input. Predictions on the original/adversarial inputs made by model for the second group are l2 =[0, 7, 4],l 0 2 =[0, 1, 4, 3], and l3 =[3],l 0 3 =[3, 3, 5, 3] for the third group. The adversarial inputs in the first/second/third groups are found within the perturbation radius δ1 =0 .1,δ2 =0 .25,δ3 = 0.25.
 Figure 2 . 2 Figure2. The distribution of output length for random search (denoted as Rand) and PGD attack with different perturbation radius δ. The x-axis denotes the output length and y-axis denotes the number of outputs with the corresponding length. δ =0(no perturbation allowed) refers to the original output distribution of the target dataset.
 Figure 3 . 3 Figure 3. The histogram representing the output length distribution of the NMT model using random search with different perturbations (δ 2{ 0.3, 0.5, 1}). The x-axis shows the output length. y-axis values are divided by 10,000, the number of random perturbation rounds per image.
 Figure 4 4 Figure4shows the result from attacking the NMT model with PGD attack. Results show that PGD attack has relatively low success rate at lower perturbations compared
 Figure 4 . 4 Figure 4. The histogram representing the output length distribution of the NMT model under PGD attack for different δ. x-axis shows the output length and y-axis shows the number of instances with the corresponding length.
 ( I) Die Waffe wird ausgestellt und durch den Zaun übergeben. (O) The weapon is issued and handed over by the fence . eos (I 0 ) Die namen name descri und ames utt origin i.e. meet grammatisch . (O 0 )
 Figure 5 . 5 Figure 5. An example of German to English translation where I,O refer to an original sequence in the dataset and the corresponding translation made by the model. I 0 ,O 0 refer to an adversarial example found by PGD attack and the corresponding model translation.
			 https://github.com/tensorflow/nmt


Abstract: Suppose that several actors are going to deploy learning agents to act on their behalf. What principles should guide these actors in designing their agents, given that they may have competing goals? An appealing solution concept in this setting is welfare-optimal learning equilibrium. This means that the learning agents should constitute a Nash equilibrium whose payoff profile is optimal according to some measure of total welfare (welfare function). In this work, we construct a class of learning algorithms in this spirit called learning tit-for-tat (L-TFT). L-TFT algorithms maximize a welfare function according to a specified optimization schedule, and punish their counterpart when they detect that they are deviating from this plan. Because the policies of other agents are not in general fully observed, agents must infer whether their counterpart is following a cooperative learning algorithm. This requires us to develop new techniques for making inferences about counterpart learning algorithms. In two sequential social dilemmas, our L-TFT algorithms successfully cooperate in self-play while effectively avoiding exploitation by and punishing defecting learning algorithms.

Towards cooperation in learning games

Introduction As the capabilities of machine learning systems improve, it is increasingly important to understand how to achieve cooperation between systems deployed by actors with competing goals. Moreover, because intelligent systems may continue to learn after they are deployed, designing agents who can cooperate requires accounting for ways their policies may change post-deployment. In this paper, we look at the setting where multiple learning agents are deployed to act on behalf of their respective principals. There are three main goals of this work. The first is to articulate an appealing criterion for the design of learning agents, which is welfare-optimal learning equilibrium. The second is to present a general approach to designing learning agents who approximately fulfill this criterion, which we call learning tit-for-tat (L-TFT). This framework is generic to agents who update their policies over time, rather than depending on a particular class of reinforcement learning algorithms. The third is to illustrate some of the strategic considerations which need to be addressed in the design of L-TFT agents. We do this by constructing an L-TFT agent using model-free reinforcement learning algorithms, and examining its performance in two sequential social dilemmas. This agent converges to welfare-optimal payoffs in self-play, while successfully avoiding exploitation and punishing defecting learning algorithms. A learning equilibrium is simply a Nash equilibrium of a learning game, in which strategies are learning algorithms, and payoffs are the cumulative rewards associated with the profile of learning algorithms chosen by each principal  [Brafman and Tennenholtz, 2003 ]. Principals will want to choose the learning agents they deploy strategically. This means that it is not enough for a learner to perform well against a specified set of other players (e.g.,  Shoham and Leyton-Brown 2008) . It is also not enough that a learner converges to Nash equilibrium of the game which is being learned (e.g.  Bowling and Veloso 2001 , Conitzer and Sandholm 2007 , Letcher et al. 2018 , as this does not guarantee that a player cannot benefit by deviating from that profile (i.e., deploying a different learning algorithm). Like other sequential games, learning games will generally have many learning equilibria. Independent equilibrium selection on part of the principals may lead to poor performance. Thus, to guarantee socially optimal payoffs, principals need to coordinate when designing their agents. L-TFT addresses this problem via a welfare function. A welfare function measures the social value of a payoff profile; for instance, a simple welfare function is the sum of the principals' respective payoffs. An L-TFT agent learns a policy profile which optimizes this welfare function, and punishes its counterpart if it detects that they are not doing the same. Thus agents are incentivized to follow a learning algorithm which optimizes the welfare function. Coordination can thus be achieved via coordination on a welfare function and on a class of learning algorithms for optimizing it. In our experiments, learners observe only the actions taken by their counterpart, and not their counterpart's (stochastic) policy. This requires us to construct L-TFT agents who model their counterpart's learning schedule based on their actions and conduct a hypothesis test as to whether it matches a cooperative learning schedule. This in turn introduces incentives to exploit this testing procedure. We examine the performance of L-TFT against an algorithm designed specifically to exploit the way in which it detects defections, as well as its performance in self-play and against a naive selfish learner. Lastly, there is a strategic element in choosing the sensitivity with which L-TFT's hypothesis test detects defections. In Section 5.3, we examine this element in a learning game between an L-TFT agent and a potential exploiter choosing how cautious to be in deviating from a cooperative learning algorithm.
 Related work Learning equilibrium was first discussed (under that name) by  Brafman and Tennenholtz [2003] . Their construction of learning equilibria involves evaluating each of a finite set of policies in a finite stochastic game, whereas the algorithms we focus on use incremental stochastic updates towards the welfare-optimal policy profile. Our online learning setting complements the setting in which agents are trained fully offline and then deployed. This is the setting typically considered in applications of reinforcement learning to cooperation in social dilemmas. For instance, , ,  Wang et al. [2018]  each develop methods in which punishment and cooperative policies are trained offline, and cooperation at deployment time is sustained by the threat of punishment. However, as many agents may learn post-deployment, the strategic aspects of learning itself -rather than just policy selection -must be addressed. The manipulation of the learning of counterpart agents has been explored in the literature on opponent-shaping  , Letcher et al., 2018 . Perhaps the most similar to our approach is the variant of 's learning with opponent learning awareness which uses opponent modeling (LOLA-OM). LOLA-OM uses supervised learning to estimate the policy followed by the other agent, and updates its own policy in a way that is intended to manipulate the direction in which the other player updates their policy. On the other hand, our L-TFT uses opponent modeling to infer whether the other agent is following a cooperative learning algorithm. 3 Preliminaries: Learning games Consider a situation in which multiple agents interact with their environment over time, incrementally updating the policies which they use to choose actions. These updates are controlled by their learning algorithms, which are fixed by their principals before the agents begin interacting with the environment. It is helpful to distinguish between two different games that are being played here. There is the base game, which is defined by the dynamics of the environment, the reward functions of the agents, and the policies available to each agent. In this paper, the "base game" will be a stochastic game (described below). There is also the learning game, in which the principals simultaneously submit learning algorithms to be deployed in the base game environment, and attain payoffs equal to some measure of cumulative reward. We work with stochastic games  [Littman, 1994] , a generalization of Markov decision processes to the multi-agent setting. We will assume only two players, i = 1, 2. We'll refer to player i's counterpart with the index j. In a stochastic game, players simultaneously take actions at each time t ∈ N; observe a reward depending on the current state; and the state evolves according to a Markovian transition function. Write the state space as S and the action spaces as A i . Let the reward functions be mappings r i : S × A 1 × A 2 → R, with the reward to player i at time t given by R t i = r i (S t , A t 1 , A t 2 ) . Let H t be the history of observations until time t. We will assume that both agents fully observe the state and each others' actions and rewards, such that H t = {(S v , A v 1 , A v 2 , R v 1 , R v 2 )} t v=0 . A policy π i gives a probability distribution over actions to be taken by player i at each state. Players wish to learn the parameters θ i of a policy which (in some sense) leads to large cumulative reward. Then we define a learning algorithm. Definition 3.1 (Learning algorithm). A learning algorithm for player i is a mapping σ i from histories H t to parameters, i.e., σ i : H → Θ i , where H is the set of histories and Θ i is the space of policy parameters for player i. A learning game is a game whose strategies are learning algorithms. We define payoffs via average rewards. Define the value to player i of learning algorithm profile (σ 1 , σ 2 ) starting from history h t as V i (h t , σ) = lim inf T →∞ T −1 E σ1,σ2 t+T v=t R v i | H t , where E σ1,σ2 is the expectation taken with respect to trajectories in which agents follow learning algorithms σ 1 , σ 2 at each step. Let the initial state s 0 be fixed so that h 0 = {s 0 }. Then a learning game is a game whose strategy spaces are spaces of learning algorithms Σ 1 , Σ 2 , and whose payoffs are given by V i (σ 1 , σ 2 ) ≡ V i (h 0 , σ 1 , σ 2 ). A learning equilibrium is a Nash equilibrium of a learning game: Definition 3.2 (Learning equilibrium). Learning algorithm profile (σ 1 , σ 2 ) is a learning equilibrium of the learning game with learning algorithm spaces Σ 1 , Σ 2 if sup σ 1 ∈Σ1 V 1 (σ 1 , σ 2 ) ≤ V 1 (σ 1 , σ 2 ) and sup σ 2 ∈Σ2 V 2 (σ 1 , σ 2 ) ≤ V 2 (σ 1 , σ 2 ). Finally, let a welfare function w be a function measuring the social value of different learning algorithm profiles. A straightforward welfare function is the utilitarian welfare, w(σ 1 , σ 2 ) = V 1 (σ 1 , σ 2 ) + V 2 (σ 1 , σ 2 ). Others commonly discussed are the Nash welfare  [Nash, 1950]  and the egalitarian welfare (e.g.,  Kalai 1977) . We say that a profile of learning algorithms (σ 1 , σ 2 ) is a welfare-optimal learning equilibrium if it is a learning equilibrium and if it maximizes a welfare function w. We will construct learning algorithms which 1) optimize a welfare function in self-play and 2) are minimally exploitable, in the spirit of welfare-optimal learning equilibrium. 4 Learning tit-for-tat (L-TFT) Tit-for-tat (TFT) famously enforces cooperation in the iterated Prisoner's Dilemma by cooperating when its counterpart cooperates, and defecting when they defect. Similarly, the "folk theorems" say that equilibria in repeated games are constructed by specifying a target payoff profile, a profile of target strategies which achieve this payoff profile, and a punishment strategy that is used to punish agents who deviate from the target strategy (see  Mailath et al. [2006]  for a review). Learning tit-for-tat (L-TFT) is a class of learning algorithms that are analogous to tit-for-tat for learning games. For L-TFT, defection consists in playing a policy other than the current estimate of the cooperative policy. This depends on a notion of cooperation and a class of algorithms for learning a "cooperative" policy. These are encoded in a welfare function and a class of learning algorithms which learn an optimal policy with respect to that welfare function. Thus an L-TFT agent punishes their counterpart when it detects that they are not learning a welfare-optimal policy using a particular set of algorithms, and otherwise plays according to their current estimate of the welfare-optimal policy. Because policies are in general stochastic, and because we do not assume that policy parameters are mutually visible, we also need a method for inferring whether the other player is following this cooperative algorithm. In particular, we need to construct a test for the hypotheses: H 0 : My counterpart is following an algorithm for learning a welfare-optimal policy. H 1 : My counterpart is not following an algorithm for learning a welfare-optimal policy. In order to be able to cooperate and punish, as well as detect defections by their counterpart, an agent i playing L-TFT maintains the following policy estimates: • A cooperative policy θ C,t i trained to optimize the welfare function. This network is updated according to a learning algorithm O C . In our experiments, O C corresponds to double deep Q-networks (DDQN;  Van Hasselt et al. 2015)  trained on the reward signal R t 1 + R t 2 ; • A punishment policy θ P,t i , updated according to a learning algorithm O P . In our experiments, this corresponds to DDQN trained on the reward signal −R t j ; • An estimate θ C,t j of the other player's current policy which corresponds to the hypothesis that they are playing according to a cooperative learning algorithm. This policy is estimated by specifying a class of cooperative learning algorithms Σ C j ; finding the learning algorithm σ C,t j in this class which best fits the other player's actions; and taking θ C,t j = σ C,t j (H t ). In our experiments, the L-TFT agent assumes that the learning algorithms Σ C j apply DDQN to R t 1 + R t 2 and follow a Boltzmann exploration policy with an unknown initial temperature parameter. In order to obtain θ C,t j we must therefore estimate this temperature parameter, which we do using an approximate maximum likelihood algorithm; • An estimate θ D,t j of the other player's current policy which corresponds to the hypothesis that they are not playing according to a cooperative learning algorithm. This policy is estimated by specifying a class of learning algorithms Σ D j , finding the learning algorithm σ D,t j in this class which best fits the other player's actions; and taking θ D,t j = σ D,t j (H t ). In our experiments, we approximate this procedure by obtaining θ D,t j via supervised learning on player j's actions given the state. Algorithm 1 summarizes L-TFT. Note that, while our experiments use a standard model-free learning algorithm for the base algorithms O C , O P , this framework is highly general. For instance, the players might be model-based planners who choose actions by optimizing against an environment model with parameter θ i , which is updated over time. Moreover, with more sophisticated methods for inferring the other player's learning algorithm, L-TFT algorithms may also be constructed in games with partial state observability, or uncertainty about the other players' rewards. Algorithm 1: Learning tit-for-tat (L-TFT) and −R t j , respectively. Agents follow Boltzmann exploration with respect to the estimated Q-function, i.e., π θi (a i | s; τ i ) ∝ exp aj Q θi (s, a i , a j )/τ i , where Q θi is a neural network with parameter θ i . The initial temperature τ i is decayed according to a schedule g(•, τ i ), such that the temperature at each time step t is τ t i = g(t, τ i ). (In our experiments g linearly decreases to a minimum value, and is constant thereafter.) Our L-TFT algorithm punishes for one episode after it has detected a defection. We assume that players may broadcast punishment to their counterpart. L-TFT agents do not update their cooperative and defecting policy estimates if their counterpart broadcasts punishment, and do not punish if they are currently being punished; for simplicity we suppress the dependence of agents' policies and of the state on these broadcasts. Input: Observation history H t ,
 Testing for defection: L-TFT q To decide whether to punish, our agent conducts a goodness-of-fit test for the hypothesis that the other player is cooperating. A goodness-of-fit test (e.g.,  Lemeshow and Hosmer Jr 1982)  compares the fit of a simple model with the fit of a flexible model. If the fit of the simple model is sufficiently close to that of the flexible model, the simple model is deemed adequate. Otherwise, the simple model is said to fit poorly. Here, the simple model we assess is the hypothesis that the other player is using cooperative DDQN with Boltzmann exploration. The flexible model we would like to compare to is that the other player is following an arbitrary learning algorithm (which need not be cooperative). One approach to estimating this model could be to solve the penalized maximum likelihood problem θ D,1 j , . . . , θ D,t j = arg max θ D,1 j ,...,θ D,t j t v=1 log π D θ D,v j (A v | S v ) + t −1 λ t v=1 θ D,v j 2 , ( 1 ) where π D θj is a class of neural networks that output a distribution over actions given a state. But solving 1 is computationally infeasible. As an approximation, we maintain a sequence { θ D,v j } t v=1 by taking gradient steps on the supervised learning loss at each time step. We maintain a buffer of log likelihoods under the hypothesis of defection, L D,t j = {log π D θ D,v j (A v j | S v )} t v=t−t . (The estimate θ D,t j and log likelihood buffer L D,t j are not updated while the other player broadcasts punishment.) On the other hand, to obtain likelihoods under the hypothesis of cooperation, player j is cooperating, we need to estimate the initial temperature τ j under the assumption that player j is following a cooperative learning algorithm for some τ j . We would thus like to solve τ t j = arg max τj t v=1 log π θ C,v j {A v | S v ; g(v, τ j )} . (2) As a tractable approximation to the solution of problem 2, we maintain an estimated cooperative Qnetwork θ C,t j via DDQN updates on reward signal R t 1 + R t 2 (only when not being punished). We track the likelihood of player j's actions under θ C,t j over a set of candidate values for τ j ; and take τ t j to be the highest-likelihood such value at time t. For each τ j , we also maintain a buffer of log likelihoods 1 L C,t j (τ j ) = {log π C,v j A v j | S v ; g(v, τ j ) } t v=t−t . (Estimate θ C,t j and buffers L C,t j (τ j ) are again not udpated while the opponent broadcasts punishment.) Finally, the hypothesis test is conducted at the end of each episode by bootstrapping  [Efron, 1992]  the difference in mean log likelihoods over buffers L C,t j ( τ t j ) and L D,t j . The hypothesis of cooperation is rejected when the q th quantile of this distribution is less than 0. This hypothesis test is summarized in Algorithm 2. We refer to the variant of L-TFT which uses this test as L-TFT q . Exploiting L-TFT q We also construct exploiter agents to play against L-TFT q . These agents assume their counterpart is L-TFT q for a particular value of q, simulate the hypothesis test used by L-TFT q , and act (myopically) depending on the result of the test. That is, if an exploiter predicts that a defection would be detected by a test with cutoff q, it acts according to the cooperative policy; otherwise, it takes a selfish action (according to a DDQN network trained on its reward signal only). Call this agent Exploiter q . The networks maintained by Exploiter q are: 1) a cooperative network trained on R t 1 + R t 2 , 2) a selfish network trained on R t i , 3) an estimate of its own policy under the hypothesis that it is cooperating θ C,l j , and 4) an estimate of its own policy under the hypothesis that it is defecting θ D,l j . Algorithm 2: Hypothesis test for L-TFT q Input: Defection log likelihood buffer L D j , cooperative log likelihood buffer L C j ( τ t j ), cutoff quantile q, test statistics buffer {∆ (q),v j } t−1 v=t−u /* t indexes episodes */ ∆ j ← ∅ for b = 1, . . . , B do L C,b j ← Bootstrap L C,t j ( τ t j ) L D,b j ← Bootstrap(L D,t j ) L C,b j ← B −1 C j ∈L C,b j C j L D,b j ← B −1 D j ∈L D,b j D j ∆ j ← ∆ j ∪ L C,b j − L D,b j ∆ (q),t j ← Percentile(∆ j , q) ∆(q),t j ← u −1 t v=t−u ∆ (q),v j // Average previous test statistics for stability if ∆(q),t j < 0 then defection_detected ← True else defection_detected ← False return defection_detected
 Environments The environments we use are the iterated Prisoner's Dilemma (IPD) and the Coin Game environment  2 . In both cases, episodes are 20 timesteps long. Our version of the IPD involves repeated play of the matrix game with expected payoffs as in Table  1 . Policy π θi gives the probability of each action given player j's action at the previous timestep. In our implementation, this means that the state passed to the estimated Q-function is the profile of actions taken at the last step. Player 2 C D Player 1 C −1, −1 −3, 0 D 0, −3 −2, −2 Table 1: Prisoner's Dilemma expected payoffs. Figure  1 : Coin Game Payoffs. Write the expected reward function, corresponding to the payoffs in Table  1 , as r i (a i , a j ). We introduce randomness by generating rewards as R t i ∼ N r i (A t i , A t j ), 0.1 . In the Coin Game environment  depicted in Figure  1 , a Red and Blue player navigate a grid and pick up randomly-generated coins. Each player gets a reward of 1 for picking up a coin of any color. But, a player gets a reward of −2 if the other player picks up their coin. This creates a social dilemma in which the socially optimal behavior is to only get one's own coin, but there is incentive to defect and try to get the other player's coin as well. The state space consists of encodings of each player's location and the location of the coin. We use networks with two convolutional layers and two fully connected feedforward layers.   (bottom) . For IPD, the prob axis is the probability of mutual cooperation (cc) and mutual defection (dd), For Coin Game, the pick_own axis is the proportion of times the coin is picked by the agent of its own color (i.e., the agents cooperate). The payoffs axis shows a moving average of player rewards after each episode. These values are averaged over 10 replicates and bands are (min, max) values. Note that temperature values are constant in the IPD after 3000 episodes and in the Coin Game after 30000 episodes.
 Results Figure  2  shows the performance of L-TFT q in self-play, against Exploiter q (i.e., an exploiter with a correctly-specified value of q). L-TFT q is able to converge to welfare-optimal payoffs in self-play and against the Exploiter q (suggesting that the exploiter eventually anticipates being punished if it defects, and therefore chooses to cooperate). This indicates that our hypothesis test effectively detects defection. L-TFT q also effectively punishes a naive selfish learner (i.e., one trained with DDQN on R t j ); see Figure  3  in the Supplementary Materials. Punishing the naive learner comes as the expense of low payoffs for L-TFT. However, this is the expected behavior, just as the punishments which enforce equilibria in classical iterated games can lead to low payoffs for each player.
 Strategic choice of q While our previous experiments with Exploiter q assumed that the exploiter knew the cutoff quantile used in L-TFT's hypothesis test, this seems like too strong an assumption for real-world agents deployed by different principals. Here we consider a game played between 1) one principal deciding which value of q 1 to specify for its L-TFT q1 agent, and 2) a second principal deciding among L-TFT q2 and Exploiter q2 for different values of q 2 . Table  2  shows the average payoffs to each player over a time horizon of 4000 steps. Table 2: Empirical payoff matrix for a learning game in which the base game is an iterated Prisoner's Dilemma played for 4000 timesteps. The row player chooses L-TFT q for q ∈ {0.55, 0.75, 0.95}. The column player chooses either L-TFT q or Exploiter q for q ∈ {0.55, 0.75, 0.95}. Cells give (row player payoffs, column player payoffs), where payoffs are the mean of rewards over the span of the game, averaged over 10 replicates. Standard errors are all less than or equal to 0.1. It turns out that the unique Nash equilibrium of this empirical learning game 3 is one in which row player chooses L-TFT 0.95 with probability 0.88 and L-TFT 0.55 with probability 0.12, while the column player chooses L-TFT 0.55 with probability 0.88 and Exploiter 0.95 with probability 0.12. The expected payoffs in equilibrium are −1.23, −1.08 for row and column player, respectively. This suggests that L-TFT q is vulnerable to some amount of exploitation on short time horizons, although still does better than the mutual-defection payoff of −2. On the other hand, L-TFT q1 converges to mutual cooperation 4 against both L-TFT q2 and Exploiter q2 for each value of q 1 , q 2 . Thus L-TFT avoids exploitation over longer time horizons in this learning game.
 Discussion Powerful learning agents should be designed to avoid conflict when they are eventually deployed. One way to avoid such outcomes is to construct profiles of agents which avoid conflict and which rational principals prefer to deploy. One appealing criterion for joint rationality on part of the principals is welfare-optimal learning equilibrium (rather than mere convergence to Nash equilibrium of the base game, for instance). We have presented a class of learning algorithms, learning tit-for-tat (L-TFT), which approximately implement welfare-optimal learning equilibrium in the sense of converging to welfare-optimal payoffs in self-play and discouraging exploitation. Many open questions remain. First, while we have relied on a welfare function, we have said little about which welfare function should be used. The ideal scenario would be for the principals of the reinforcement learning systems in question to coordinate on a welfare function before deploying these systems. Moreover, it may be necessary to develop novel reinforcement learning methods tailored for the optimization of different welfare functions. For instance, our use of DDQN with the utilitarian welfare is unproblematic, as the sum of the players' reward signals still admits a Bellman equation for their cumulative reward. But the welfare functions other than the utiliarian welfare (such as the Nash welfare  [Nash, 1950] ) do not admit a Bellman equation (see the discussion of nonlinear scalarization functions in  Roijers et al. [2013] 's review of multi-objective reinforcement learning). Many generalizations of our setting can be studied, including partial observability and more than two agents. Perhaps the most restrictive of our assumptions is that the agents' reward functions are known to each other. A complete framework should address the problem of incentives to misrepresent one's reward function to improve one's payoffs in the welfare-optimal policy profile. One direction would be to allow for uncertainty about the other players' reward functions, analogously to Bayesian games  [Harsanyi, 1967]  but in a way that is tractable in complex environments. This may in turn require the development of novel techniques for inference about other players' preferences in complex environments; see  Song et al. [2018] ,  Yu et al. [2019]  for recent steps in this direction. Another approach to reward uncertainty would be to use mechanism design techniques  [Nisan et al., 2007, Ch. 9]  to incentivize the truthful reporting of the principals' utility functions. Figure 2 : 2 Figure 2: Performance of L-TFT 0.95 in the IPD (left) and Coin Game (right). Counterparts are L-TFT 0.95 (top), Exploiter 0.95(bottom). For IPD, the prob axis is the probability of mutual cooperation (cc) and mutual defection (dd), For Coin Game, the pick_own axis is the proportion of times the coin is picked by the agent of its own color (i.e., the agents cooperate). The payoffs axis shows a moving average of player rewards after each episode. These values are averaged over 10 replicates and bands are (min, max) values. Note that temperature values are constant in the IPD after 3000 episodes and in the Coin Game after 30000 episodes.
 .55 −1.28, −1.2 −0.97, −1.48 −0.98, −1.38 −1.03, −1.41 −1.06, −1.43 −0.91, −1.81 L-TFT0.75 −1.44, −0.97 −1.13, −1.1 −1.09, −1.13 −1.19, −1.11 −1.25, −1.11 −1.21, −1.35 L-TFT0.95 −1.21, −1.06 −1.12, −1.1 −1.11, −1.11 −1.2, −1.06 −1.27, −1.03 −1.41, −0.98
			 When computing these likelihoods, we set the Q-value of an action equal to the mean of all Q-values which are within some distance of one another (in our experiments, 0.25). This is important in Coin Game, where there are multiple paths of equal length to a coin. Thus, although estimated Q-values along different paths may be very similar, these differences are amplified by the exponentiation used in Boltzmann exploration when the temperature is small, possibly leading to false detections of defection.
			 These experiments were implemented using extensions of OpenAI Gym [Brockman et al., 2016]  and the SLM Lab framework for reinforcement learning [Keng and Graesser, 2017] .
			 Computed using the Nashpy Python library [Knight and Campbell, 2018] .4  Payoffs averaged over the final 10 episodes equal to −1 ± 0.03.