Learning to Make Decisions via Submodular Regularization

1 INTRODUCTION . In real-world automated decision making tasks we seek the optimal set of actions that jointly achieve the maximal utility . Many of such tasks — either deterministic/non-adaptive or stochastic/adaptive — can be viewed as combinatorial optimization problems over a large number of actions . As an example , consider the active learning problem where a learner seeks the maximally-informative set of training examples for learning a classifier . The utility of a training set could be measured by the mutual information ( Lindley , 1956 ) between the training set and the remaining ( unlabeled ) data points , or by the expected reduction in generation error if the model is trained on the candidate training set . Similar problems arise in a number of other domains , such as experimental design ( Chaloner and Verdinelli , 1995 ) , document summarization ( Lin and Bilmes , 2012 ) , recommender system ( Javdani et al. , 2014 ) , and policy making ( Runge et al. , 2011 ) . Identifying the optimal set of actions ( e.g. , optimal training sets , most informative experiments ) amounts to evaluating the expected utility over a combinatorial number of candidate sets . When the underlying model class is complex and the evaluation of the utility function is expensive , these tasks are notoriously difficult to optimize ( Krause and Guestrin , 2009 ) . For a broad class of decision making problems whose optimization criterion is to maximize the decision-theoretic value of information ( e.g. , active learning and experimental design ) , it has been shown that it is possible to design surrogate objective functions that are ( approximately ) submodular while being aligned with the original objective at the optimal solutions ( Javdani et al. , 2014 ; Chen et al. , 2015b ; Choudhury et al. , 2017 ) . Here , the information gathering policies no longer aim to directly optimize the target objective value , but rather choose to follow a greedy trajectory governed by the surrogate function that is much cheaper to evaluate . These insights have led to principled algorithms that enable significant gains in the efficiency of the decision making process , while enjoying strong performance guarantees that are competitive with the optimal policy . Despite the promising performance , a caveat for these “ submodular surrogate ” -based approaches is that it is often challenging to engineer such a surrogate objective without an ad-hoc design and analysis that requires trial-and-error ( Chen et al. , 2015b ; Satsangi et al. , 2018 ) . Furthermore , for certain classes of surrogate functions , it is NP-hard to compute/evaluate the function value ( Javdani et al. , 2014 ) . In such cases , even a greedy policy , which iteratively picks the best action given the ( observed ) history , can be prohibitively costly to design or run . Addressing this limitation requires more automated or systematic ways of designing ( efficient ) surrogate objective functions for decision making . Overview of main results . Inspired by contemporary work in data-driven decision making , we aim to learn a greedy heuristic for sequentially selecting actions . This heuristic acts as a surrogate for invoking the expensive oracle when evaluating an action . Our key insight is that many practical algorithms can be interpreted as greedy approaches that follow an ( approximate ) submodular surrogate objective . In particular , we focus on the class of combinatorial problems that can be solved via submodular maximization ( either directly on the objective function or via a submodular surrogate ) . We highlight some of the key results below : • Focusing on utility-based greedy policies , we introduce a data-driven optimization framework based on the “ submodular-norm ” loss , which is a novel loss function that encourages learning functions that exhibit “ diminishing returns ” . Our framework , called LEASURE ( Learning with Submodular Regularization ) , outputs a surrogate objective that is efficient to train , approximately submodular , and can be made permutation-invariant . The latter two properties allow us to prove approximation guarantees for the resulting greedy heuristic . • We show that our approach can be easily integrated with modern imitation learning pipelines for sequential prediction tasks . We provide a rigorous analysis of the proposed algorithm and prove strong performance guarantees for the learned objective . • We demonstrate the performance of our approach on a variety of decision making tasks , including set cover , active learning for classification , and data-driven protein design . Our results suggest that , compared to standard learning-based baselines : ( a ) at training time , LEASURE requires significantly fewer oracle calls to learn the target objective ( i.e. , to minimize the approximation error against the oracle objective ) ; and ( b ) at test time , LEASURE achieves superior performance on the corresponding optimization task ( i.e. , to minimize the regret for the original combinatorial optimization task ) . In particular , LEASURE has shown promising performance in the protein design task and will be incorporated into a real-world protein design workflow . 2 RELATED WORK . Near-optimal decision making via submodular optimization . Submodularity is a property of a set function that has a strong relationship with diminishing returns , and the use of submodularity has wide applications from information gathering to document summarization ( Leskovec et al. , 2007 ; Krause et al. , 2008 ; Lin and Bilmes , 2011 ; Krause and Golovin , 2014 ) . The maximization of a submodular function has been an active area of study in various settings such as centralized ( Nemhauser et al. , 1978 ; Buchbinder et al. , 2014 ; Mitrovic et al. , 2017 ) , streaming ( Badanidiyuru et al. , 2014 ; Kazemi et al. , 2019 ; Feldman et al. , 2020 ) , continuous ( Bian et al. , 2017b ; Bach , 2019 ) and approximate ( Horel and Singer , 2016 ; Bian et al. , 2017a ) . Variants of the greedy algorithm , which iteratively selects an element that maximizes the marginal gain , feature prominently in the algorithm design process . For example , in the case of maximizing a monotone submodular function subject to a cardinality constraint , it is shown that the greedy algorithm achieves an approximation ratio of ( 1− 1/e ) of the optimal solution ( Nemhauser et al. , 1978 ) . In applications where we need to make a sequence of decisions , such as information gathering , we usually need to adapt our future decisions based on past outcomes . Adaptive submodularity is the corresponding property where an adaptive greedy algorithm enjoys a similar guarantee for maximizing an adaptive submodular function ( Golovin and Krause , 2011 ) . Recent works have explored optimizing the value of information ( Chen et al. , 2015b ) and Bayesian active learning ( Javdani et al. , 2014 ; Chen et al. , 2017a ) with this property . Another line of related work is online setting ( typically bandits ) , which is grounded in minimizing cumulative regret ( Radlinski et al. , 2008 ; Streeter et al. , 2009 ; Yue and Guestrin , 2011 ; Ross et al. , 2013 ; Yu et al. , 2016 ; Hiranandani et al. , 2020 ) . Learning submodular functions . Early work focused on learning non-negative linear combinations of submodular basis functions ( Yue and Joachims , 2008 ; El-Arini et al. , 2009 ; Yue and Guestrin , 2011 ; Sipos et al. , 2012 ) , which was later generalized to mixtures of “ submodular shells ” ( Lin and Bilmes , 2012 ) . Deep submodular functions ( Dolhansky and Bilmes , 2016 ) extend these ideas to more expressive compositional function classes by using sums of concave composed with modular functions . The theoretical question of the learnability of general submodular functions is analyzed in Balcan and Harvey ( 2018 ) . Our goal is to encourage submodularity via regularization , rather than via hard constraints on the function class design . Learning to optimize via imitation learning . Rather than first learning a submodular function and then optimizing it , one can instead learn to directly make decisions ( e.g. , imitate the oracle greedy algorithm ) . This area builds upon imitation learning , which learns a policy ( i.e. , a mapping from states to actions ) directly from examples provided by an expert ( e.g. , an expensive computational oracle , or a human instructor ) ( Chernova and Thomaz , 2014 ) . Classic work on imitation learning ( e.g. , the Dataset Aggregation ( DAgger ) algorithm ( Ross et al. , 2011 ) ) reduce the policy learning problem to the supervised learning setting , which has been extended to submodular optimization by imitating the greedy oracle method ( Ross et al. , 2013 ) . More generally , learning to optimize has been applied generically to improve combinatorial optimization solvers for focused distributions of optimization problems ( He et al. , 2014 ; Song et al. , 2018 ; Khalil et al. , 2016 ; Balunovic et al. , 2018 ; Gasse et al. , 2019 ; Song et al. , 2020 ) . Our approach bridges learning to optimize and learning submodular functions , with a focus on learning surrogate utilities using submodular regularization . Learning active learning . Our approach is applicable to active learning , and so is related to work on learning active learning . The closest line of work learns a utility function as a surrogate for improvement in classifier accuracy ( Konyushkova et al. , 2017 ; Liu et al. , 2018 ) , which is then used as the decision criterion . However , prior work either used restricted function classes ( Konyushkova et al. , 2017 ) , or very expressive function classes that can be hard to fit well ( Liu et al. , 2018 ) . Our work can be viewed as a direct extension of this design philosophy , where we aim to reliably learn over expressive function classes using submodular regularization . Other related work do not directly learn an active learning criterion , instead encouraging sample diversity using submodularity ( Wei et al. , 2015 ) or the gradient signal from the classifier ( Ash et al. , 2020 ) . 3 BACKGROUND AND PROBLEM STATEMENT . 3.1 DECISION MAKING VIA SUBMODULAR SURROGATES . Given a ground set of items V to pick from , let u : 2V → R be a set function that measures the value of any given subset1 A ⊆ V . For example , for experimental design , u ( A ) captures the utility of the output of the best experiment ; for active learning u ( A ) captures the generalization error after training with set A . We denote a policy π : 2V → V to be a partial mapping from the set/sequence of items already selected , to the next item to be picked . We use Π to denote our policy class . Each time a policy picks an item e ∈ V , it incurs a unit cost . Given the ground set V , the utility function u , and a budget k for selecting items , we seek the optimal policy π that achieves the maximal utility : π∗ ∈ arg max π∈Π u ( Sπ , k ) . ( 1 ) Sπ , k is the sequence of items picked by π : Sπ , i = Sπ , i−1 ∪ { π ( Sπ , i−1 ) } for i > 0 and Sπ,0 = ∅ . As we have discussed in the previous sections , many sequential decision making problems can be characterized as constrained monotone submodular maximization problem . In those scenarios u is : • Monotone : For any A ⊆ V and e ∈ V \A , u ( A ) ≤ u ( A ∪ { e } ) . • Submodular : For any A ⊆ B ⊆ V and e ∈ V \B , u ( A ∪ { e } ) − u ( A ) ≥ u ( B ∪ { e } ) − u ( B ) . 1For simplicity , we focus on deterministic set functions in this section . Note that many of our results can easily extent to the stochastic , by leveraging the theory of adaptive submodularity ( Golovin and Krause , 2011 ) In such cases , a mypopic algorithm following the greedy trajectory of u admits a near-optimal policy . However , in many real-world applications , u is not monotone submodular . Then one strategy is to design a surrogate function f : 2V → R which is : • Globally aligning with u : For instance , f lies within a factor of u : f ( A ) ∈ [ c1 · u ( A ) , c2 · u ( A ) ) ] for some constants c1 , c2 and any set A ⊆ V ; or within a small margin with u : f ( A ) ∈ [ u ( A ) − , u ( A ) + ] for a fixed > 0 and any set A ⊆ V ; • Monotone submodular : Intuitively , a submodular surrogate function encourages selecting items that are beneficial in the long run , while ensuring that the decision maker does not miss out any actions that are “ surprisingly good ” by following a myopic policy ( i.e. , future gains for any item are diminishing ) . Examples that fall into this category include machine teaching ( Singla et al. , 2014 ) , active learning ( Chen et al. , 2015a ) , etc . We argue that in real-world decision making scenarios—as validated later in Section 6—the decision maker is following a surrogate objective that aligns with the above characterization . In the following context , we will assume that such surrogate function exists . Our goal is thus to learn from an expert policy that behaves greedily according to such surrogate functions .

Summary: This paper combines combines submodular surrogates for sequential decision making with imitation learning. Specifically, it proposes to learn an acquisition function g by imitating an expert which is assumed to be following a greedy policy wrt a general submodular surrogate f. This is accomplished by regularizing g to encourage diminishing returns and monotonicity. The learning algorithm is a modified version of DAgger which is consistent with the expert and provably near-optimal utility. Results outperform baselines on various sequential decision making tasks.