UNIVERSIDADE TÉCNICA
DE LISBOAINSTITUTO SUPERIOR TÉCNICOCARNEGIE MELLON UNIVERSITY

The Geometry of Constrained Structured Prediction:
Applications to Inference and Learningof Natural Language Syntax
André Filipe Torres Martins

Thesis Committee:
Prof. Mário Alexandre Teles Figueiredo,
Prof. Noah Ashton Smith,Prof. Pedro Manuel Quintas Aguiar,
Prof. Eric Poe Xing,
Prof. Isabel Maria Martins Trancoso,Prof. Carlos Ernesto Guestrin,
Prof. Michael Collins,

Instituto Superior Técnico
Carnegie Mellon University
Instituto Superior Técnico
Carnegie Mellon University
Instituto Superior Técnico
Carnegie Mellon University
Columbia University

Thesis specifically prepared to obtain the Dual PhD Degree in
Language Technologyc 2012, André F. T. Martins

Resumo
Esta tese propõe novos modelos e algoritmos para a predição de variáveis estruturadas, comênfase em aplicações no processamento de linguagem natural. Apresentam-se contribuições
em duas frentes: no problema de inferência, cujo objectivo é fazer uma predição a partir deum modelo estatı́stico, e no problema de aprendizagem, onde o modelo é treinado a partir
dos dados.
No primeiro problema, propõe-se uma mudança de paradigma, considerando-se modelos expressivos com caracterı́sticas e restrições globais, representáveis como modelos gráficos
restritos. Introduz-se um novo algoritmo de inferência aproximada, o qual ignora os efeitos
globais causados pelos ciclos do grafo. Esta metodologia é depois aplicada ao problema daanálise sintáctica de texto, originando novos analisadores de dependências a que chamamos“turbo parsers”, os quais atingem resultados ao nı́vel do estado-da-arte.
No segundo problema, considera-se uma famı́lia de funções de custo que inclui os campos
aleatórios condicionais, as máquinas de vectores de suporte, e o perceptrão estruturado, paraa qual apresentamos novos algoritmos em linha que dispensam a especificação de uma taxa
de aprendizagem. Em seguida, debruçamo-nos sobre o regularizador, o qual é utilizado parapromover esparsidade estruturada e para aprender classificadores estruturados com múltiplos
núcleos. Para o efeito, introduzem-se novos algoritmos do tipo gradiente-proximal, os quais
têm a capacidade de explorar grandes espaços de caracterı́sticas de forma eficiente, com
reduzido consumo de memória. Como resultado, obtém-se um novo procedimento para aselecção de caracterı́sticas-padrão, o qual conduz a modelos compactos e precisos.

Palavras-chave: Aprendizagem estatı́stica, classificação estruturada, processamento de linguagem natural, sintaxe de dependências, modelos gráficos probabilı́sticos, programaçãolinear inteira, decomposição dual, método das direcções alternadas, algoritmos gradienteproximal, esparsidade estruturada.

i

ii

Abstract
This thesis proposes new models and algorithms for structured output prediction, with anemphasis on natural language processing applications. We advance in two fronts: in theinference problem, whose aim is to make a prediction given a model, and in the learning
problem, where the model is trained from data.
For inference, we make a paradigm shift, by considering rich models with global featuresand constraints, representable as constrained graphical models. We introduce a new approximate decoder that ignores global effects caused by the cycles of the graph. This methodology
is then applied to syntactic analysis of text, yielding a new framework which we call “turbo
parsing,” with state-of-the-art results.
For learning, we consider a family of loss functions encompassing conditional random
fields, support vector machines and the structured perceptron, for which we provide new
online algorithms that dispense with learning rate hyperparameters. We then focus on theregularizer, which we use for promoting structured sparsity and for learning structured predictors with multiple kernels. We introduce online proximal-gradient algorithms that can
explore large feature spaces efficiently, with minimal memory consumption. The result is a
new framework for feature template selection yielding compact and accurate models.

Keywords:

Machine learning, structured classification, natural language processing, depen-dency parsing, probabilistic graphical models, integer linear programming, dual decomposition, alternating directions method of multipliers, proximal gradient algorithms, structured
sparsity.

iii

Acknowledgments
This thesis would not exist without the help of many.
First of all, I would like to thank my quadruplet of advisors at IST and CMU, who havealways pointed me not just to the North, but to all four cardinal directions: Mário Figueiredo,
Pedro Aguiar, Noah Smith, and Eric Xing. Thanks to their diverse areas of expertise, I could
expand horizons and learn about many different topics, which ended up cross-fertilizing
and producing this thesis. I have been working with Mário and Pedro since the beginning of
my research career. They have always provided truly insightful advice, whether the topic is
computer vision, kernel methods, or optimization algorithms. Noah has always been there
for enthusiastic discussions, either physically at CMU, or via Skype meetings while I was at
IST. It was him who introduced me the problem of dependency parsing and taught me all I
know about statistical NLP. I’m grateful to Eric for pointing out structured prediction as an
exciting research topic. He has a tremendous insight about what are the open problems inmachine learning that are worthwhile for pursuing research, and has been very encouraging
and supportive of my research.
Thanks also to Carlos Guestrin and Michael Collins for agreeing being in my thesis
committee. I have greatly benefited from discussions with both of them that materialized
in this thesis. Carlos’ work in optimization and graphical models has been a source ofinspiration, and so has been the work done by Mike and his group on dual decompositionand statistical parsing.
During my two-year stay at CMU, I was very fortunate to participate in the ARK and
SAILING research groups. I learned a lot from uncountable discussions with Shay Cohen,
Dipanjan Das, Kevin Gimpel, Mike Heilman, Tae Yano, Nathan Schneider, Chris Dyer, Brendan O’Connor, Dani Yogatama, Jacob Eisenstein, Mladen Kolar, Kriti Puniyani, Amr Ahmed,
Le Song, and Pradipta Ray, among others. Special thanks to Dipanjan and Kevin for collaborating with me in several projects. It was also a remarkable experience to interact withseveral other people in the LTI: Alan Black, Alon Lavie, Lori Levin, Stephen Vogel, William
Cohen, Gopala Anumanchipalli, Sourish Chaudhuri, Ni Lao, Vı́tor Carvalho, Andreas Zollmann, Luis Marujo, Matthew Marge, and Konstantin Salomatin are just some of them. I am
thankful to Bob Frederking and José Moura for helping to clarify several questions about the
CMU-Portugal program, allowing me to make the most of this two-year experience. Thanks
also to Dana Houston, Stacey Young, Radha Rao, Michelle Martin, Diane Stidle, Lori Spears,
Nicole Hillard-Hudson, and remaining staff at LTI, MLD and ICTI. Special thanks to all
my friends for the good times we have spent in Pittsburgh, in particular Vasco and Maria
Calais Pedro, Luis Pedro Coelho, Rita Reis, the corresponding families, and the remaining
Portuguese crew.
During the period of my PhD in Portugal, I had the opportunity of interacting withv

many colleagues working in the fields of machine learning and NLP. At IST, I learned a lot
from discussions with João Xavier, José Bioucas Dias, Ana Fred, Mariana Sá Correia, André
Lourenço, João Graça, Ana Mendes, João Mota, Manya Afonso, and many others. JoãoPaulo Costeira, Victor Barroso and Isabel Trancoso answered several questions about the
CMU-Portugal program. Special thanks to João Graça and Luis Sarmento for co-organizing
with me the first LxMLS (Lisbon Machine Learning School), a remarkable experience! I am
also very grateful to Paulo Almeida, for teaching me so much about mathematics, geometry,and language. Now I realize that I need to learn much more.
I was fortunate to spend a semester interning at Google Research in New York. I amvery thankful to my host, Slav Petrov, as well as the remaining people in the NLP group:
Ryan McDonald, Kuzman Ganchev, Keith Hall, and Hao Zhang, with whom I had so many
interesting discussions. During that period I was also lucky to interact with Sasha Rush,
who was interning, Yoav Goldberg and Percy Liang, who were doing a post-doc, Afshin
Rostamizadeh and Umar Syed, who had just started as research scientists, and Stephen
Boyd, who was visiting.
Along the five years of my PhD, I benefited from many interesting discussions withresearchers from other labs and universities, such as Jason Eisner, David Smith, David
Sontag, Ben Taskar, Xavier Carreras, Terry Koo, David Hall, Taylor Berg-Kirkpatrick, John
Blitzer, John DeNero, Sebastian Riedel, James Clarke, Ming-Wei Chang, Scott Yih, Kristina
Toutanova, Hoifung Poon, Chris Meek, Chris Quirk, Partha Talukdar, Guy Lebanon, Cheng
Soon Ong, Manuele Bicego, Geoff Gordon, Ben Recht, Reza Zadeh, Ofer Meshi, and Gholamreza Haffiri.
I would like to thank Priberam for all the support during the PhD years, in particularAfonso Mendes, who encouraged me to apply to the PhD program, and also Carlos Amaral
and João Prieto for believing that research is an added value. Priberam has also sponsoredthe machine learning seminars at IST which I have been organizing since 2009.
My PhD studies were funded by a grant from Fundação para a Ciência e Tecnologia(FCT) and Information and Communication Technology Institute (ICTI) through the CMUPortugal Program (SFRH/BD/36737/2007). Some work was partially supported by the FET
programme (EU FP7), under the SIMBAD project (contract 213250), and by a FCT grant
PTDC/EEA-TEL/72572/2006.
On a personal note, I am deeply grateful to my parents Teresa and Leonel Martins, whohave always motivated me to study since my first day of school, an opportunity which they
never had in their lifetimes. I am also thankful to my brother Luis. Obrigado por tudo!
Finally, I thank my loving and supportive family for sharing this journey with me. In
particular, my wife Sarah, who encouraged me to apply to this PhD program and has madeunconditional sacrifices to join me in Pittsburgh. Thank you for all your love and patience!
I dedicate this thesis to my son Samuel, who gave his first steps in our house in Greenfield,
and to my daugther Mariana, who was born after we came back, just a couple of days before
the first LxMLS.

Contents
Resumoi

Abstractiii

1

Introduction
1.1 Motivation and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.1 Inference and Graphical Models . . . . . . . . . . . . . . . . . . . . . . .
1.1.2 Learning Structured Predictors . . . . . . . . . . . . . . . . . . . . . . . .
1.1.3 Natural Language Processing . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Previous Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Contributions and Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1
2
3
3
4
5
5
7

I

Background92 Structure in Natural Language
2.1 Tagging and Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Sequence Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 Tagging Problems in Natural Language Processing . . . . . . . . . . . .
2.2 Phrase-Structure Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Dependency Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11
12
12
14
15
21
26

3 Structured Prediction and Learning Algorithms
3.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Linear Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.2 Loss Functions for Discriminative Learning . . . . . . . . . . . . . . . .
3.3 What is Structured Prediction? . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Discriminative Learning of Structured Predictors . . . . . . . . . . . . . . . . .
3.4.1 Structured Cost Functions . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 Conditional Random Fields . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.3 Structured Support Vector Machines . . . . . . . . . . . . . . . . . . . . .
3.5 Online and Stochastic Learning Algorithms . . . . . . . . . . . . . . . . . . . . .
3.5.1 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . .

27
27
29
31
31
33
35
36
36
37
38
39

vii

Stochastic Subgradient Descent . . . . . . . . . . . . . . . . . . . . . . . .
The Perceptron Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . .
Online Passive-Aggressive Algorithms . . . . . . . . . . . . . . . . . . .

40
41
42

4 Graphical Models and Inference Algorithms
4.1 Undirected Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1.1 Markov Random Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1.2 Factor Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Message-Passing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Graphs with Cycles and Approximate Inference . . . . . . . . . . . . . .
4.3 The Geometry of Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Graphical Models as Exponential Families . . . . . . . . . . . . . . . . .
4.3.2 Mean Parametrization and Marginal Polytope . . . . . . . . . . . . . . .
4.3.3 MAP Inference as a Linear Program . . . . . . . . . . . . . . . . . . . . .
4.3.4 Marginal Inference and Conjugate Duality . . . . . . . . . . . . . . . . .
4.4 Local Polytope Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Bethe’s Variational Principle and Loopy Belief Propagation . . . . . . . . . . . .
4.6 Linear Programming Relaxation for MAP Inference . . . . . . . . . . . . . . . .
4.6.1 Block Coordinate Descent Algorithms . . . . . . . . . . . . . . . . . . . .
4.6.2 Dual Decomposition with the Projected Subgradient Algorithm . . . . .

45
45
46
47
48
49
50
50
53
54
55
56
57
59
60
61

II

65

3.5.2
3.5.3
3.5.4

Inference

5 Constrained Structured Prediction
5.1 Motivation and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Constrained Graphical Models and Their Geometry . . . . . . . . . . . . . . . .
5.2.1 Marginal and Local Polytopes . . . . . . . . . . . . . . . . . . . . . . . .
5.2.2 Duality and Variational Representations . . . . . . . . . . . . . . . . . .
5.3 An Inventory of Hard Constraint Factors . . . . . . . . . . . . . . . . . . . . . .
5.3.1 One-hot XOR and Uniqueness Quantification . . . . . . . . . . . . . . .
5.3.2 OR and Existential Quantification . . . . . . . . . . . . . . . . . . . . . .
5.3.3 Negations and De Morgan’s law . . . . . . . . . . . . . . . . . . . . . . .
5.3.4 Logical Variable Assignments: XOR-with-output and OR-with-output .
5.3.5 AND-with-output and Expressions with Universal Quantifiers . . . . .
5.4 Computing Messages and Solving Local Subproblems . . . . . . . . . . . . . .
5.4.1 Sum-Product Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.2 Max-Product Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.3 Negations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.4 Generalized Message-Passing Algorithms . . . . . . . . . . . . . . . . .
5.4.5 Local Subproblems in Dual Decomposition . . . . . . . . . . . . . . . . .
5.5 Approximate Inference in Constrained Factor Graphs . . . . . . . . . . . . . . .
5.5.1 Binarization of a Factor Graph . . . . . . . . . . . . . . . . . . . . . . . .
5.5.2 Local Polytope Approximation and LP-MAP Inference . . . . . . . . . .
5.5.3 Bethe Entropy Approximation and Marginal Inference . . . . . . . . . .

67
68
68
69
71
74
76
77
78
79
79
81
81
82
86
89
92
94
95
95
97
99

5.6
5.7
6

Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Alternating Directions Dual Decomposition
105
6.1 Motivation and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.2 Augmented Lagrangian Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.2.1 Method of Multipliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.2.2 Alternating Direction Method of Multipliers . . . . . . . . . . . . . . . . 109
6.3 LP-MAP Inference with the AD3 Algorithm . . . . . . . . . . . . . . . . . . . . 110
6.3.1 Derivation of AD3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.3.2 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4 Solving the Local Subproblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.4.1 Binary pairwise factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.4.2 Hard constraint factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.4.3 Larger factors and multi-valued variables . . . . . . . . . . . . . . . . . . 120
6.5 An Active Set Method for Handling Arbitrary Factors . . . . . . . . . . . . . . 121
6.6 Exact Inference with Branch-and-Bound . . . . . . . . . . . . . . . . . . . . . . . 125
6.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.7.1 Binary Pairwise MRFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.7.2 Multi-valued Pairwise MRFs . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.7.3 Protein Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.7.4 Frame Semantic Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.8 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

7 Turbo Parsers
135
7.1 Motivation and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.2 Dependency Parsing as an Integer Linear Program . . . . . . . . . . . . . . . . 137
7.2.1 The Arborescence Polytope . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.2.2 A Cycle-Based Representation . . . . . . . . . . . . . . . . . . . . . . . . 140
7.2.3 A Single Commodity Flow Representation . . . . . . . . . . . . . . . . . 141
7.2.4 A Multi-Commodity Flow Representation . . . . . . . . . . . . . . . . . 142
7.2.5 Arc-Factored Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.2.6 Pairwise Interactions: Siblings, Grandparents and Head Bigrams . . . . 145
7.2.7 Head Automata and Consecutive Siblings . . . . . . . . . . . . . . . . . 147
7.2.8 Valency Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.2.9 Directed Path Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.2.10 Nonprojective Arc Indicators . . . . . . . . . . . . . . . . . . . . . . . . . 151
7.3 Turbo Parsers and Their Factor Graphs . . . . . . . . . . . . . . . . . . . . . . . 151
7.3.1 A Tree-Based Factor Graph . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.3.2 A Flow-Based Factor Graph . . . . . . . . . . . . . . . . . . . . . . . . . . 155
7.4 Dependency Parsing with AD3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.4.1 Related Work: Dual Decomposition in NLP . . . . . . . . . . . . . . . . 159
7.4.2 Dual Decomposition With Many Overlapping Components . . . . . . . 160
7.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

7.6

III
8

Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Learning

171

Learning with Dual Coordinate Ascent173
8.1 Motivation and Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
8.2 A Family of Loss Functions for Structured Classification . . . . . . . . . . . . . 175
8.3 Loss Evaluation and Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.4 Online Learning with Dual Coordinate Ascent . . . . . . . . . . . . . . . . . . . 177
8.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
8.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

9 Online Learning with Multiple Kernels
185
9.1 Motivation and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
9.2 Multiple Kernel Learning and Group Sparsity . . . . . . . . . . . . . . . . . . . 187
9.2.1 Inference and Learning with Kernels . . . . . . . . . . . . . . . . . . . . 187
9.2.2 Block-Norm Regularization and Kernel Learning . . . . . . . . . . . . . 188
9.2.3 Learning the Kernel in Structured Prediction . . . . . . . . . . . . . . . . 189
9.2.4 Existing MKL Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
9.3 Online Proximal Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
9.3.1 Proximity Operators and Moreau Projections . . . . . . . . . . . . . . . 191
9.3.2 An Online Proximal Gradient Scheme . . . . . . . . . . . . . . . . . . . . 192
9.3.3 Proximity Operators of Block-Norm Regularizers . . . . . . . . . . . . . 194
9.3.4 Regret, Convergence, and Generalization Bounds . . . . . . . . . . . . . 195
9.3.5 SPOM: Structured Prediction with Online MKL . . . . . . . . . . . . . . 198
9.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.4.1 Handwriting Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.4.2 Online Binary Classification . . . . . . . . . . . . . . . . . . . . . . . . . . 201
9.5 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
9.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
10 Structured Sparsity for Structured Prediction
205
10.1 Motivation and Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
10.2 Sparse Modeling in Structured Prediction . . . . . . . . . . . . . . . . . . . . . . 206
10.3 Structured Sparsity: Group-Lasso Regularization . . . . . . . . . . . . . . . . . 207
10.3.1 The Group Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
10.3.2 Bayesian Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
10.3.3 Prox-operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
10.4 Online Proximal Gradient Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 211
10.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
10.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
10.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

IV

Conclusions

221

11 Conclusions
223
11.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
11.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
11.2.1 Broader Constrained Formalisms and Better Entropy Approximations . 224
11.2.2 Applying AD3 to Other Combinatorial Problems . . . . . . . . . . . . . 225
11.2.3 Sparse Structured Prediction . . . . . . . . . . . . . . . . . . . . . . . . . 225
11.2.4 Turbo Parsers and Other Syntactic Formalisms . . . . . . . . . . . . . . 226
11.2.5 Learning With Approximate Inference . . . . . . . . . . . . . . . . . . . 226
11.2.6 Structured Sparsity in NLP . . . . . . . . . . . . . . . . . . . . . . . . . . 226

V

Appendices229

A Background Proofs
231
A.1 Proof of Proposition 2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
A.2 Derivation of the MPLP Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 232
B Convex Analysis and Optimization
235
B.1 Convex Sets, Hulls, and Polytopes . . . . . . . . . . . . . . . . . . . . . . . . . . 235
B.2 Convex Functions, Subdifferentials, Proximity Operators, and Moreau Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
C Derivation of Messages for Logic Factors
239
C.1 Sum-Product Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
C.2 Max-Product Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
D Proof of Convergence Rate of ADMM and AD3251

E Derivation of Solutions for Quadratic Problems in AD3257
E.1 Binary pairwise factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
E.2 Hard constraint factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
E.2.1 Sifting Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
E.2.2 OR-with-output Factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
E.3 Proof of Proposition 6.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
F Proofs for Online Proximal Gradient Algorithms
263
F.1 Proof of Proposition 9.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
F.2 Proof of Corollary 9.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
F.3 Proof of Proposition 9.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
F.4 Proof of Proposition 9.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
F.5 Proof of Lemma 9.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
F.6 Proof of Proposition 9.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
F.7 Lipschitz Constants of Some Loss Functions . . . . . . . . . . . . . . . . . . . . 266
F.8 Computing the proximity operator of the squared L1 . . . . . . . . . . . . . . . 267

Table of Notation
R
R+
R++
ε
Σ
Σ∗
[[π ]]
A, B, C, . . .
a, b, c, . . .
A, B, C, . . .
X
Y
h:X→Y
Y( x )f ( x, y)w
f p ( x, y p )Z (w, x )
θ p (y p )θµG := (V, F, E)
MARG(G )LOCAL(G )The set of real numbers
The set of non-negative real numbers
The set of strictly positive real numbers
The empty string
An alphabet
The Kleene closure of Σ, Σ∗ = {ε} ∪ Σ ∪ Σ2 ∪ . . .
1 if predicate π is true, 0 otherwise
Arbitrary sets
Arbitrary vectors of real numbers
Arbitrary matrices over the reals
The input set
The output set
A classifier
The set of admissible outputs for x ∈ X
A joint feature vector
The weight vector
A local feature vector at a place p
The partition function of Pw (y| x )A local score for a part ( p, y p )A score vector, corresponding to the canonical parameters
A marginal vector, corresponding to the mean paramaters
A factor graph with variable nodes V, factor nodes F, and edges E
The marginal polytope of a factor graph G
The local polytope of a factor graph G

xiii

Chapter 1

Introduction
The last years have witnessed a remarkable progress in language technologies, well patentin the ever-increasing quality of search engines, question-answering systems, speech recognizers, and automatic translators. All these systems perform tasks that have to deal withstructured and ambiguous textual data, where the “structure” emanates from the complex
network of syntactic and semantic phenomena that underlies natural language. Despite thetechnological advances, we are far from the point where those problems can be declared“solved.” To make computers capable of understanding and extracting meaning from text,
a new generation of methods, models and algorithms is likely to be necessary.
So, what is the driving force behind the aforementioned progress? Essentially, it is thealliance of two important factors: the massive amount of data that became available withthe advent of the Web, and the success of machine learning techniques to extract statisticalmodels from the data (Mitchell, 1997; Manning and Schütze, 1999; Schölkopf and Smola,
2002; Bishop, 2006; Smith, 2011). As a consequence, a new paradigm has emerged in thelast couple of decades, which directs attention to the data itself, as opposed to the explicit
representation of knowledge (Abney, 1996; Pereira, 2000; Halevy et al., 2009). This datacentric paradigm has been extremely fruitful in natural language processing (NLP), and cameto replace the classic knowledge representation methodology which was prevalent until the
1980s, based on symbolic rules written by experts. The increasing availability of humanannotated corpora—of which the Penn Treebank (Marcus et al., 1993) is perhaps the most
famous example—opened the door for the use of supervised machine learning methods.
While earlier statistical models, such as hidden Markov models (Jelinek, 1997; Rabiner, 1989)
and probabilistic context-free grammars (Charniak, 1996b; Baker, 1979) were over-simplistic,
relying on too stringent independence assumptions, significant progress has been made recently using feature-based models for structured prediction. As a consequence, increasingly
accurate models have been devised, with applications to syntax, semantics, and machinetranslation.
Structured prediction (Bakır et al., 2007) is a machine learning framework that provides aunified treatment for dealing with structured output variables, rooted in seminal work by
Lafferty et al. (2001); Collins (2002a); Taskar et al. (2003); Altun et al. (2003); Tsochantaridis
et al. (2004), among others. This framework paved the way for the development of increasingly sophisticated statistical language models, with fewer independence assumptions thanearlier models, at the cost of more expensive estimation procedures. Today, there is a clear
demand for more powerful models, motivated by the perception, from the scientific com1

CHAPTER 1. INTRODUCTION

2

munity, that there is a long tail of linguistic phenomena which can only be captured byincorporating global features and constraints into the model.1 A current limitation is that theinclusion of this sort of features and constraints has a negative effect on the simplicity andefficiency of inference algorithms. For this reason, the progress in this area has been shapedby the ability of extending the existing algorithms, rather than being oriented by the designof features that can have impact on the quality of the model, as would be desirable. That is,
the model design is typically subjugated by algorithmic considerations.
This thesis contributes to a paradigm shift by embracing principled forms of approximate
inference. A general framework is proposed, based on the formalism of graphical models,
which allows the easy incorporation of global features and declarative constraints as factors
in the model. Then, new inference algorithms are proposed that are tailored to solve a linear
relaxation of the original problem, while exploiting the structure of the graphical model. This
framework is compatible with recently proposed formalisms that ally the robustness andsemantics of statistical models and the expressive power of first order logic (Roth and Yih,
2004; Richardson and Domingos, 2006), bridging the gap between statistical and symbolic
methods.
More broadly, we contribute new methods, models, and algorithms for statistical learningand inference with structured data. We give particular emphasis to applications in NLP,
such as syntactic parsing, text segmentation, and recognition of named entities, all of which
involve predicting structured output variables, such as trees or sequences. For modelingthe syntax of languages, we introduce rich models with global features and constraints andembrace approximate inference through a new dual decomposition algorithm that is suitablefor handling declarative constraints. We also consider the structure in the feature space,through the proposal of new learning algorithms that select feature templates and identify
relevant patterns in the data.
Note, however, that structured data is by no means exclusive of natural language. Thereis a wide range of important problems, in areas such as computer vision, communication
theory, computational biology, signal processing, and others, which are also characterized
by the need of handling structured data and identifying complex interactions. The methods
that we propose are quite general; it is thus likely that the contributions presented in thisthesis will benefit other areas dealing with structured prediction.

1.1 Motivation and Related Work
There are several aspects that characterize structured prediction problems: the representational framework and its interplay with the inference procedure, the learning procedure thatis employed for training the model, and the particular properties of the application that isbeing tackled. We next describe our motivations in each of these aspects and briefly pinpoint some of the related work; a more thorough description of prior work will appear in
subsequent chapters, when we discuss each aspect in full detail. We will also highlight someof the open problems that this thesis addresses.
1 By “global,” in this context, we mean features capable of modeling long-range dependencies, which are noteasy to incorporate in low-order Markov models.

1.1. MOTIVATION AND RELATED WORK

1.1.1

3

Inference and Graphical Models

Many problems in NLP, computer vision, coding theory, computational biology, and otherareas require predicting output variables which are strongly interdependent. Probabilistic
graphical models (Pearl, 1988; Wainwright and Jordan, 2008; Koller and Friedman, 2009) area framework that allows the unified treatment of such problems, combining tools from graphtheory, statistics, and optimization.
Many advances were made recently by the graphical models community regarding approximate inference algorithms, both for marginal and MAP inference, based on sampling,heuristic search, or variational approximations. A line of research that has stimulated a lotof recent interest is the linear programming relaxation of Schlesinger (1976), which in thisthesis we designate by “LP-MAP inference.” Many message-passing and dual decomposition algorithms have been proposed recently to address this problem (Wainwright et al.,2005a; Kolmogorov, 2006; Komodakis et al., 2007; Globerson and Jaakkola, 2008; Rush et al.,2010). The advantage over other approximate algorithms is that the underlying optimization
problem is well-understood and the algorithms are convergent or provide certain guarantees. Moreover, there are ways of tightening the relaxation toward the exact solution (Sontag
et al., 2008).
Despite these advances, the case of constrained graphical models has barely been addressed, and it is not well understood how the approximate algorithms behave in the presence of declarative constraints. For example, it is not clear whether marginal and MAP inference are equally difficult, or one is more amenable to approximations than the other. Some
previous work applied message-passing algorithms under global constraints, but it was tailored to specific applications, such as bipartite matching (Duchi et al., 2007) and dependency
parsing (Smith and Eisner, 2008). A formal characterization of constrained graphical models
is still missing, in particular one that is compatible with the kind of declarative constraints
that can be encoded through symbolic rules. This lack of theoretical understanding contrasts
with a number of practical achievements, brought by new representation formalisms suchas probabilistic relational networks (Friedman et al., 1999), constrained conditional models(Roth and Yih, 2004), and Markov logic networks (Richardson and Domingos, 2006). It is
likely that consistent advances in the field can only be achieved through a better understanding of the underlying approximations and their geometry.
Some open questions are:
• How to ally the graphical model machinery and the use of declarative constraints, e.g.,in first-order logic?
• Can we construct faster approximate algorithms, more amenable to the presence ofconstraints?
• Can we interpret those approximate algorithms geometrically, and write down the
optimization problem that they are addressing?

1.1.2

Learning Structured Predictors

Given a graphical model for representing a structured output, a fundamental problem is thatof learning its parameters. This is the main concern of structured prediction. In this thesis,

CHAPTER 1. INTRODUCTION

4

we focus on discriminative structured linear models, introduced in a string of seminal work
starting with conditional random fields (Lafferty et al., 2001), and followed by the structured
perceptron algorithm (Collins, 2002a), and structured support vector machines (Taskar et al.,
2003; Altun et al., 2003; Tsochantaridis et al., 2004).
In the formalisms above, the learning problem corresponds to the minimization of aregularized empirical risk functional:
minimizeΩ(w) +

1 N
L(w; x n , yn )N n∑=1with respect to w,(1.1)

where (( x n , yn ))nN=1 is the training data, Ω is a regularizer and L is a loss function, parameterized by w. Despite the progress in this area, many important questions remain unanswered.
• What is the “right” loss function for each problem? Is it worth interpolating between
some of the losses that have been proposed (such as the logistic loss, hinge loss, etc.)?
• Which algorithms are more suitable for a large-scale setting (large N)? Currently,
online algorithms such as the structured passive-aggressive algorithm (Crammer et al.,2006) or stochastic gradient descent (Bottou, 1991a; Vishwanathan et al., 2006) appear
to be the most suitable, but can we design better ones?
• And what if we use an approximate inference algorithm as a subroutine—what would
be the impact in the model that is learned?
Regarding the regularizer Ω, even more is left to be done. Most work so far either does
not regularize, or uses a regularizer with the sole goal of controlling overfitting or producing
a compact model. However, as recent work in sparse modeling suggests (Zhao et al., 2009;
Jenatton et al., 2009; Bach et al., 2011), a clever design of the regularizer is an opportunity
for exploiting rich prior knowledge, or for discovering patterns in the data. Very little has
been done toward this goal in structured prediction and in NLP, where sparse modeling is
still taking its first steps. The following questions remain unanswered:• How to deal with the structure of the feature space? Can we promote structural patterns or discover patterns in the data through the regularizer?
• If so, how can we explore large feature spaces efficiently, without having to touch every
feature? How to adapt online algorithms to solve the resulting optimization problem
and exploit the sparsity?

1.1.3

Natural Language Processing

This thesis applies structured prediction models and algorithms to problems in NLP. Ourpreferred application is syntactic parsing, which has seen considerable progress in the lasttwo decades. The development of treebanks for several languages, of which the Penn Treebank is an example (Marcus et al., 1993), made possible approaching the problem with supervised learning techniques. These treebanks contain tens of thousands of sentences pairedwith the corresponding syntactic parse trees, annotated by humans.
In this thesis, we adopt dependency syntax as the grammar formalism of choice. The historical origins of dependency syntax go back to the Sanskrit grammar of Pān.ini in the 4th

1.2. PREVIOUS PUBLICATIONS

5

century BC, and its modern treatment is due to the seminal work of Tesnière (1959); Hudson(1984); Mel’čuk (1988), among others. It is a lightweight formalism which, due to its lexical motivation, relative ease of annotation, and the fact that it does not require the explicit
construction of a grammar, presents computational advantages over other formalisms, suchas phrase-structure grammars (Chomsky, 1965). Statistical dependency parsers have been
first developed by Eisner (1996); McDonald et al. (2005b), and have since then gained prominence in NLP, with applications to relation extraction (Culotta and Sorensen, 2004), machinetranslation (Ding and Palmer, 2005), and question answering (Wang et al., 2007).
Increasingly accurate models for dependency parsing have been proposed lately, some ofthem while this thesis was in preparation (McDonald and Pereira, 2006; Smith and Eisner,
2008; Koo and Collins, 2010; Koo et al., 2010; Huang and Sagae, 2010). These models become
more accurate by incorporating more global features, for example through an increase ofhorizontal or vertical Markov context. We will review this related work in detail in Chapter 2,
with pointers to the literature.
Some open questions are:
• How can we incorporate global features that promote certain properties (such as nearly
projective parse trees) when they are observed in the data?
• Are horizontal and vertical Markovian assumptions adequate for free-order languages?
How can we still do inference beyond those assumptions?
• How can we inject prior knowledge, such as declarative constraints, without having toredesign inference algorithms from scratch?

1.2 Previous Publications
To make this dissertation a coherent piece, we omit some of the research work carried out
during the doctoral studies. This includes work in information-theoretic kernels for structured inputs (Martins et al., 2008c,b, 2009a), later applied to image classification and patternrecognition problems (Bicego et al., 2010a; Martins et al., 2010a; Bicego et al., 2010b). Somework on stacked learning of dependency parsers (Martins et al., 2008a), and on document
summarization through a joint model for sentence extraction and compression (Martins andSmith, 2009) was also omitted. Finally, we only mention very briefly some of our work on
characterizing learning through approximate inference (Martins et al., 2009c).

1.3 Contributions and Thesis Statement
We next summarize the main contributions of this thesis, addressing some of the open problems mentioned in the previous section.
1. We make graphical models capable of handling declarative constraints by introducing a simple set of logic factors. Despite their simplicity, these factors constitute thebuilding blocks for expressing arbitrary logic constraints. We derive analytical expressions for their messages, marginals, entropies, and marginal polytopes. In a global
sense, we provide a formal characterization of constrained graphical models and their

6

CHAPTER 1. INTRODUCTION
geometry, extending the unconstrained case. Our results provide a unified treatmentof isolated cases studied in some of the aforementioned previous work.
2. We propose AD3 , a new dual decomposition algorithm for LP-MAP inference. AD3
is particularly suitable for models that cannot be lightly decomposed, typical when
dealing with declarative constraints. It has the same modular architecture of previous
dual decomposition algorithms, but it is faster to reach consensus, and it is suitable for
embedding in a branch-and-bound procedure toward the true MAP. We derive efficient
procedures for handling the above logic factors, turning AD3 into a powerful tool for
constrained structured prediction. We also introduce a new active set procedure for
dealing with dense, large or combinatorial factors, whose only interface with the factor
is an oracle for computing the local MAP.
3. We cast dependency parsing as a concise integer program and introduce turbo parsers.
Through a multi-commodity flow formulation, we obtain a polynomial number of constraints in the integer linear program (ILP), contrasting with previous formulations,where this number was exponential. We incorporate rich global features and constraints into the model, leading to substantial performance gains. We show how thelinear relaxation of the ILP, as well as other parsers recently proposed, are all instancesof turbo parsers: parsers that perform approximate inference in loopy graphical models,
ignoring global effects caused by the loops.2
4. We generalize the MIRA algorithm for a wide family of loss functions. This family
includes support vector machines, conditional random fields, and the structured perceptron. The resulting algorithms are similar to online and stochastic gradient descent,but have the advantage of self-adjusting their stepsizes, dispensing the specification ofa learning rate. They have an interpretation as dual coordinate ascent algorithms.
5. We propose new online proximal-gradient algorithms for kernel learning and structured sparsity. This addresses one of the open questions posed above, regarding therole of regularizers in the learning problem. We employ group-structured regularizers
for learning combinations of multiple kernels and for identifying relevant feature templates. Our regularizers are designed to take into account the structure of the feature
space and the desired sparsity patterns. Our approach can also handle overlappinggroups of features, through sequential proximal steps. The online algorithms allow
exploring large feature spaces with fast runtime and minimal memory requirements.

Thesis Statement. Our claim is that the structure of natural language can be effectively
modeled with the machinery of structured prediction and graphical models. A vanilla application of the existing methods, however, is not sufficient. First, there is a long tail of
linguistic phenomena that can only be captured by rich models with global features andconstraints. Approximate algorithms based on relaxations are a principled way for making
predictions with these rich models, liberating the model designer from the handcuffs of algorithmic concerns. Second, we argue that an effective exploration of large model spaces ispossible without sacrificing computational efficiency of learning or accuracy, by endowing
the learner with a regularizer that promotes structured sparsity.
2 This strategy finds a parallel in error-correcting coding theory, where it underlies the famous turbo-codes.

1.4. ORGANIZATION OF THE THESIS

7

1.4 Organization of the Thesis
This thesis is divided into four parts, which we describe below.
Part I: Background. The first part systematizes the previous work in structured prediction
and NLP which is relevant for this thesis, providing the necessary background for better
exposing our contributions in remaining chapters. It includes the following chapters:• Chapter 2 describes the NLP applications addressed in this thesis. All of them involve
predicting some kind of structure.
• Chapter 3 provides background on supervised learning and structured prediction. We
describe structured linear models and online algorithms for learning them from data.
• Chapter 4 describes inference in graphical models. We discuss in detail some approximate inference algorithms and the relaxed problems that they address.
The reader who is familiar with this background material might prefer to skip the aforementioned chapters in a first reading, and proceed to the second part.
Part II: Inference. This part addresses models and algorithms for inference with structured
outputs, in which we present novel contributions. It is comprised of the following chapters:• Chapter 5 extends structured predictors and graphical models to deal with constrained
outputs. It presents an expressive set of logic factors, along with closed-form expressions for computing messages, marginals, and other quantities of interest.
• Chapter 6 introduces the AD3 algorithm, analyzes its convergence properties, andshows how to deal with the logic factors introduced in Chapter 5. It also provides anactive set method for arbitrary factors, presenting experiments in benchmark datasets.
• Chapter 7 presents turbo parsers, along with a concise ILP formulation for non-projective
dependency parsing and the corresponding constrained factor graph with logic factors.
Detailed experiments are presented for 14 languages.
Part III: Learning. This part is about methods and algorithms for learning structured predictors. The following chapters are its constitution:• Chapter 8 introduces a new class of online learning algorithms that can deal with awide family of structured loss functions. The algorithms are evaluated in named entity
recognition and dependency parsing tasks.
• Chapter 9 introduces a new online algorithm for learning structure predictors withmultiple kernels, establishing its regret and convergence properties.
• Chapter 10 proposes a new structured sparsity approach for selecting feature templates
in structured prediction problems, adapting the algorithm introduced in Chapter 9. Experiments are presented in various sequence labeling tasks and in dependency parsing.
Part IV: Conclusions. This part concludes, by providing, in Chapter 11, a summary ofcontributions and drawing possible directions of future work.

8

CHAPTER 1. INTRODUCTION

Part I

Background9Chapter 2

Structure in Natural Language
This chapter provides a brief overview about the structured problems in natural language
processing that this thesis addresses.
At the heart of this thesis is the word “structure,” which derives from Latin structura, aform of the verb struere (“to build”). According to the Oxford dictionary, “structure” canrefer to either the arrangement of parts that form something complex, or the actual object
that is constructed from the several parts. We will adopt primarily the former sense, and talkabout the object with structure as a “structured object.”1 Furthermore, we will designate by“structured set” a discrete set of objects that can be formed by arranging parts in different
ways; and by “structured problem” a computational problem where the goal is to output astructured object chosen from a structured set. In Chapter 3, we will provide a more formal
definition of “structured prediction” in terms of statistical dependencies between the partsof an object.
Structured problems abound in natural language processing. For example, machinetranslation and speech recognition both involve generating text as output—a structured object formed by a sequential arrangement of words. In syntactic parsing, one is given a sentence and the goal is to find a tree-structured arrangement of words or phrases, which is also
a structured object. Semantic parsing aims to represent text through logical forms or otherstructured representations. There are several textbooks devoted to statistical approaches tothese problems: for example, Charniak (1996a); Manning and Schütze (1999); Jurafsky andMartin (2000); Koehn (2010); Smith (2011).
In this thesis, we focus on a few paradigmatic and well-defined structured problems, allinvolving some input object or observation x ∈ X, given which we want to predict a structured
output y ∈ Y. To do so, we have some model at our disposal, which has an associatedprediction rule h : X → Y. Typical prediction rules for structured problems are built of local
score functions θ p ( x, y p ) where each p is a place, and y p is an output assignment at that place;
the goal is then to predict the global assignment yb that maximizes the sum of local scores:
yb := arg max
y ∈Y∑ θ p (x, y p )

!
.

(2.1)

p

Sometimes, instead of places, it is more convenient to talk about the “parts of the structure,”
1 Named after the word “structure” are doctrines and disciplines such as structuralism or structural linguistics;

this is not the sense of the word that we will be using throughout.

11CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

12

and to see y as a collection of parts. By part we mean a pair r := ( p, y p ) comprised of a place
and an output assignment at that place. According to the above, each part r = ( p, y p ) will
have a score φr ( x ) := θ p ( x, y p ). The prediction rule (2.1) can as well be written as
yb := arg max
y ∈Y∑ φr (x)

!
.

(2.2)

r ∈y

This chapter is organized as follows: in Section 2.1, we describe tagging and segmentation problems. Sections 2.2 and 2.3 provide a background on syntax and parsing, where
the target structure is a tree. Specifically, we address phrase-structure grammars in Section 2.2, and spend Section 2.3 on dependency parsing, which is the syntactic formalism
most prominently discussed in this thesis.

2.1 Tagging and Segmentation
Many tasks involving natural language have a sequential nature. This is not surprising:languages are sets of strings, i.e., sequences of characters in an alphabet. Common needs
are that of segmenting such sequences, or tagging each character with a particular label. We
next discuss tagging, without losing generality—any segmentation task can be reduced totagging by specifying labels denoting the beginning and ending of segments.
Depending on the application, there are several possible levels of granularity regarding
what a “character” should mean: it can be a letter or digit (this is the case, for example, in
handwriting recognition, in which we want to map from image representations of handwritten text characters to their textual representation), a phoneme (as in speech recognition), or aword (for example, in part-of-speech tagging and named entity recognition).

2.1.1

Sequence Models

We describe two statistical sequence models commonly used in NLP: hidden Markov models(HMMs) and conditional random fields (CRFs).
Hidden Markov models. HMMs (Jelinek, 1997; Rabiner, 1989) are a probabilistic analogueof finite state machines. They are are generative models that regard X := ( X1 , . . . , X L ) as asequence of random variables describing observations, and Y := (Y1 , . . . , YL ) as a sequenceof random variables describing states. Each state Yi emits a symbol Xi according to anemission probability P( Xi |Yi ), and the state sequence forms a Markov chain with transition
probabilities P(Yi |Yi−1 ). We have, in addition, special “start” and “stop” states for modelingthe beginning and ending of sequences. We denote by Λ := {λ1 , . . . , λ|Λ| } the set of possible
states. Figure 2.1 shows a graphical representation. With this setup, the joint probability
distribution P( X, Y ) factorizes as follows:!
!
L

P( X, Y ) =∏ P(Xi |Yi )i =1

LP(Y1 |start)∏ P(Yi |Yi−1 )P(stop|YL ).

(2.3)

i =2

Given an HMM and a particular observation sequence x, an important problem is that ofsearching for the most probable Y given X = x. Defining edge scores θi,i+1 ( x, yi , yi+1 ) :=

2.1. TAGGING AND SEGMENTATION

13

log P(yi+1 |yi ), and node scores
 log P( x1 |y1 ) + log P(y1 |start) if i = 1
θi ( x, yi ) :=log P( x L |y L ) + log P(stop|y L ) if i = L log P( x |y )otherwise,
i i

(2.4)

this boils down to solving an optimization problem of the following form:
yb = arg max

y1 ,...,y L

L

L −1

i =1

i =1

∑ θi (x, yi ) + ∑ θi,i+1 (x, yi , yi+1 )

!
.

(2.5)

Eq. 2.5 expresses a common characteristic of structured prediction problems: computing
a prediction corresponds to solving a combinatorial optimization problem. Observe that thisproblem has the general form in Eq. 2.1, where the set of places is {1, . . . , L} ∪ {(i, i + 1)}iL=−11 .
Such problem is often called maximum a posteriori (MAP) inference or MAP decoding. Another important inference problem is that of computing the posterior marginal distributions
P{Yi | X = x } for each i = 1, . . . , L; this is called marginal inference or marginal decoding.
Both problems are tractable in HMMs thanks to dynamic programming: MAP inference
can be carried out in time O( LΛ2 ) with the Viterbi algorithm (Viterbi, 1967; Forney, 1973);
for marginal inference, the forward-backward algorithm (Baum and Petrie, 1966) has the same
runtime complexity.2
The model just exposed is a bigram sequence model—i.e., scores depend only on context
windows of size 2 (in other words, the state sequence is first-order Markov). More generally,
one can think of K-gram models with K ≥ 2. Such models can be reduced to bigram models
by redefining the state space, with a corresponding increase in the computational complexity,
which becomes O( LΛK ).
Another important problem in HMMs regards the estimation of the emission and transition probabilities; this is commonly refered as the learning or training problem. If learning is
supervised, i.e., if there is training data containing paired observation and state sequences,these probabilities can be easily fit by the maximum likelihood criterion, which have a closed
form solution, involving counting and normalizing transition and emission events. Unsupervised learning of HMMs is more involved; the most popular procedure is the ExpectationMaximization (EM) algorithm (Baum et al., 1970; Dempster et al., 1977).3
Conditional Random Fields. CRFs (Lafferty et al., 2001) are discriminative rather than generative: they model the conditional distribution P(Y | X ) instead of the joint P( X, Y ). The
distribution takes the form:
!
L −1
L
1
P(y| x ) :=exp ∑ θi ( x, yi ) + ∑ θi,i+1 ( x, yi , yi+1 ) ,(2.6)
Z (θ, x )i =1
i =1
where Z (θ, x ) is a normalization factor (Z (., x ) is called the partition function). Unlike HMMs,
the score functions in CRFs need not correspond to log-probabilities. Rather, for each place
2 We assume the reader is familiar with both these algorithms; if not, a good tutorial is Rabiner (1989). We will
discuss the subject of MAP and marginal inference in more depth in Chapter 4, in the context of probabilistic
graphical models.
3 Again, we refer to Rabiner (1989) for further details.

14

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

Figure 2.1: Typical sequence models in natural language tasks. Left: a hidden Markov
model, drawn as a directed graphical model. Middle: a conditional random field, drawn as
an undirected graphical model. Right: a skip-chain conditional random field, obtained byadding an edge modeling a long-range dependency.

p, the score function θ p is defined as the inner product of a weight vector w ∈ RD , whichparameterizes the model, with a joint feature vector f p ( x, y p ) ∈ RD , that is,θ p ( x, y p ) := w · f p ( x, y p ).

(2.7)

We describe this kind of parameterization in more depth in Section 3.2. An important advantage of CRFs over HMMs is that each score function θ p may depend globally on the observation sequence x (in HMMs, the score θi ( x, yi ) depends on x only via xi , and θi,i+1 ( x, yi ) does
not depend on x at all.) In spite of this, the MAP inference problem for CRFs is identical tothat of HMMs (both correspond to Eq. 2.5), and marginal inference can also be carried out
with the same forward-backward algorithm, which can be used as a by-product to evaluate
the partition function Z (θ, x ). On the other hand, the training of CRFs is more involved, andrequires numerical optimization algorithms. We will discuss this issue further in Section 3.4.
CRFs are represented as undirected graphical models; see Figure 2.1 (middle).
As in HMMs, increasing the context of the model to K-grams has a cost which is exponential in K. This highlights the fact that the tractability of inference is intimately linked to
the factorization assumptions made by the model. As an example, skip-chain CRFs (Suttonand McCallum, 2006; Galley, 2006)—which are CRFs that contain additional pairwise scoresfor non-consecutive states (see Figure 2.1, right) are in general intractable to decode, even if
those additional scores are spurious. This is because the dynamic programming algorithms
need to exponentially increase the number of states to accommodate such pairwise scores.
For these kinds of models, we must consider approximate inference algorithms. This subject
will play a major role in this thesis.

2.1.2

Tagging Problems in Natural Language Processing

We next enumerate the natural language processing applications addressed in this thesisthat involve tagging or segmentation.
Handwriting Recognition. In handwritten text recognition, we are given as input a sequence of images of alphanumeric characters and the goal is to determine their textual
representation (Figure 2.2). Intuitively, we want our model to be able to capture interactions
between consecutive characters: that way, even if some characters are hard to read (such as
the fifth letter in Figure 2.2, which could as well be a c or a e), we may “guess” the correct

2.2. PHRASE-STRUCTURE GRAMMAR

15

Figure 2.2: A sequence of images of alphanumeric characters, extracted from the OCR datasetof Taskar et al. (2003), publicly available at http://www.cis.upenn.edu/˜taskar/ocr.
The correct output sequence is the word “braces.”
Input:
Output:Meanwhile
RB
B-ADVP,,O

overall
JJ
B-NP

evidence
NN
I-NP

on
IN
B-PP

the
DT
B-NP

economyNN
I-NP

remains
VBZ
B-VP

fairlyRB
B-ADJP

cloudedVBN
I-ADJP.
.
O

Figure 2.3: A sentence, a predicted sequence of part-of-speech tags (both provided as input),
and a segmentation into phrase chunks (example taken from the CoNLL-2000 shared task
dataset, available at http://www.cnts.ua.ac.be/conll2000/chunking/). Words inthe same segment are represented with the same colors.

character by looking at the other possible characters appearing in the context. We address
handwriting recognition in Chapter 9.
Text Chunking. Text chunking is a sentence segmentation task. It consists of dividing
a text in syntactically correlated parts of words, called phrases; an example is provided in
Figure 2.3. Text chunking can be seen as a very shallow form of parsing (see Section 2.2),that ignores phrase-structure recursion. The typical way of transforming segmentation into asequence labeling task is by defining B-I-O tags, which stand for beginning of a segment (B-x),
inside a segment (I-x), and outside any segment (O). Once such a set of tags is defined, thesame kind of sequence models seen above can be used to produce a segmentation. It is often
useful to have part-of-speech tagging as a preprocessing step (hence providing additional
part-of-speech inputs, as shown in Figure 2.3), so that unigram and bigram scores may take
into consideration these tags. This is an example of how several structured tasks can be used
in a pipeline model. We address text chunking in Chapter 10.
Named Entity Recognition. Information extraction aims to automatically extract structured information from text in natural language. An important step is that of identifying
and classifying named entities; these are names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. Like text chunking, this canalso be framed as a segmentation task; see Figure 2.4. We address named entity recognitionin Chapters 8 and 10.

2.2 Phrase-Structure Grammar
Probabilistic sequence models, which we reviewed last section, are a stochastic version offinite-state machines, the class of automata that generates regular languages. A step higher inthe hierarchy of languages leads us to context-free grammars and push-down automata (Chomsky, 1956, 1965). The stochastic variant of these machines is more powerful than hidden

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

16
Input:
Output:Only
RB
O

France
NNP
B-LOC

and
CC
O

Britain
NNP
B-LOC

backedVBDOFischler
NNP
B-PER

’s
POS
O

proposal
NN
O

.
.
.

Figure 2.4: A sentence, a predicted sequence of part-of-speech tags (both provided as input),and extracted named entities (example taken from the CoNLL-2003 shared task dataset,available at http://www.cnts.ua.ac.be/conll2003/ner/). Words in the same segment are represented with the same colors.

Markov models, and is especially amenable for modeling the syntax of natural languages.4
Definition 2.1 (Chomsky 1965) A phrase-structure grammar or context-free grammar is atuple G = (Λ, Σ, P, S) where:
1. Λ is a finite set of non-terminal symbols. Elements of Λ are denoted by upper case letters(X, Y, Z, . . .). Each non-terminal symbol is a syntactic category: it represents a different typeof phrase or clause in the sentence.
2. Σ is a finite set of terminal symbols (disjoint from Γ). This is the alphabet of the language
defined by the grammar. Elements of Σ are denoted by lower case letters (a, b, c, . . .).
3. P is a set of production rules, i.e., a finite relation from Λ to (Σ ∪ Λ)∗ . We represent a
production rule as h X → αi, where X ∈ Λ and α ∈ (Σ ∪ Λ)∗ . G is said to be in Chomsky
normal form (CNF) if any production rule in P is either of the form X → YZ or X → a.
4. S is a start symbol, used to represent the whole sentence. It must be an element of Λ.
By starting from S and applying some of the production rules in P, a sentence can be derived
that is composed only of terminal symbols. Figure 2.5 depicts an example of a simple
grammar along with a parse tree for a particular sentence.
Any context-free grammar can be transformed to be in CNF without loosing any expressive power in terms of the language it generates. We henceforth assume that G is in CNF
without loss of generality.
A fundamental characteristic of natural languages is ambiguity; Figure 2.6 illustrates threeplausible interpretations for a sentence, yielding different parse trees. The ambiguity iscaused by the several places to which the prepositional phrase could be attached. This kindof syntactic ambiguity (PP-attachment) is very frequent in natural language.
Weighted context-free grammars. Weighted CFGs model the uncertainty in interpretationsof natural language sentences (Booth and Thompson, 1973; Baker, 1979). In early probabilistic models, one defines a conditional probability ψπ := P(α| X ) for each production ruleπ ∈ P of the form h X → αi. These probabilities can be estimated from a corpora of sentences with annotated parse trees (a treebank, see Figure 2.7) by simply counting events and4 This does not mean that natural languages are context free. There is an immense body of work on grammar

formalisms that relax the “context-free” assumption, and those formalisms have been endowed with a probabilistic framework as well. Examples are: lexical functional grammars, head-driven phrase structured grammars,
combinatorial categorial grammars, tree adjoining grammars, etc. Some of these formalisms are mildly context
sensitive, a relaxation of the “context-free” assumption which still allows polynomial parsing algorithms. Thereis also equivalence in expressive power among several of these formalisms.

2.2. PHRASE-STRUCTURE GRAMMAR

17

S

S --> NP VP
NP --> Det Adj N
VP --> V NP Adv
Adj --> minimal
Adv --> here
Det --> aN --> logic
N --> role
V --> plays

NP

VP

N
Logic

V

Adv

NP

plays

here
Det

Adj

N

a

minimalrole

Figure 2.5: Left: a simple grammar. Right: a parse tree for the sentence Logic plays a minimalrole here (extracted from the Penn Treebank), derived according to that grammar.

S

NP

S

VP

V

VP

NP

The man

The man

NP

sees
NP

PP

the boyin the parkwith a telescope

V

NP

PP

sees

the boyin the parkwith a telescope

S

NP

VP

The man
V

NP

PP

PP

sees

the boyin the parkwith a telescope

Figure 2.6: Three possible interpretations for the sentence The man sees the boy in the park witha telescope. Top left: the boy is in a park and he has a telescope. Top right: the boy is in apark, and the man sees him using a telescope as an instrument. Bottom: the man is in thepark and he has a telescope, through which he sees a boy somewhere.

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

18
( (S

(NP-SBJ (NNP BELL) (NNP INDUSTRIES) (NNP Inc.) )(VP (VBD increased)(NP (PRP$ its) (NN quarterly) )(PP-DIR (TO to)(NP (CD 10) (NNS cents) ))(PP-DIR (IN from)(NP(NP (CD seven) (NNS cents) )(NP-ADV (DT a) (NN share) ))))(. .) ))

Figure 2.7: Example of an annotated sentence in one of the most widely used treebanks, thePenn Treebank (Marcus et al., 1993). See http://www.cis.upenn.edu/˜treebank/ forfurther information.
normalizing, analogously to the case of HMMs described in the previous section. In suchmodels, the joint probability of a sentence x and parse tree y factors as
P( x, y) = ∏ ψππn ( x,y)

,

(2.8)

π ∈P

where nπ ( x, y) is the number of times production rule π is applied in the derivation. For
example, for the sentence in Figure 2.5 this probability would be
P( x, y) = P(NP VP|S) × P(N|NP) × P(Logic|N) × P(V NP Adv|VP) × P(plays|V)× P(Det Adj N|NP) × P(a|Det) × P(minimal|Adj)
× P(role|N) × P(here|Adv).

(2.9)

When a sentence is ambiguous, the most likely parse tree can be obtained by maximizing
this quantity with respect to y. This maximization can also be written in the form (2.2), by
specifying a decomposition into parts. To see this, let us first introduce the following sets: a
set of possible spans, i.e., phrase constituents anchored on the input sentence,(Rs : =h X, i, ji

X ∈ Λ,
1≤i≤j≤L),(2.10)

and a set of anchored production rules,(Rp :=h X, Y, Z, i, j, ki

h X → YZ i ∈ P,
1≤i≤k<j≤L)([h X, i i

h X → xi i ∈ P,
1≤i≤L

)
.

(2.11)

These are our sets of “parts,” the building blocks that may or may not participate in theparse tree y. In other words, any parse tree y can be represented as a subset of R := Rs ∪ Rp ,and this subset is always unique.5 If we assign a score θr ( x ) to each part r ∈ R, we can thencast the search for the most likely parse tree yb (the MAP inference problem) as the problem
5 Note, however, that the converse is not true: there may be subsets in R that do not correspond to valid parsetrees—namely, subsets for which productions and spans are inconsistent or cannot be combined to build up atree. We will see later that this kind of problem can be addressed by deriving a set of consistency equations thatwork as constraints in a linear optimization problem.

2.2. PHRASE-STRUCTURE GRAMMAR

19

of obtaining the combination of rules which maximizes the overall score:yb = arg max

y ∈Y( x )∑ θr ( x )!
,(2.12)r ∈ywhere Y( x ) denotes the set of valid parse trees for sentence x.
Given an anchored production rule r ∈ Rp , denote by π (r ) ∈ P the corresponding
production rule; the correspondent scores in Eq. 2.12 for the probabilistic context-free model
defined above (which is a generative model) are θr ( x ) := log ψπ for each r ∈ Rp , and θr ( x ) :=0 for each r ∈ Rs . Alternatives are discriminative log-linear models (the analogue of CRFs
for context-free parsing, Finkel et al. 2008), and max-margin models (Taskar et al., 2004b);
both result in a problem of the form (2.12), for the same decomposition into parts.
Inference in Weighted CFGs. Let us turn to the problem of solving Eq. 2.12. Since thenumber of possible parse trees grows exponentially with the sentence length, it is computationally prohibitive to carry out this maximization directly. However, we can make use of
dynamic programming to carry out this computation efficiently, through the Cocke-KasamiYounger (CKY) algorithm (Kasami, 1966; Younger, 1967; Cocke and Schwartz, 1970), which
has runtime O(|P| L3 ), where L is the length of the sentence, and |P| is a grammar constant,specifying the number of production rules in the grammar.
The other important inference problem is that of computing the posterior marginals, Pr{r ∈y| x }, for each constituent phrase span r ∈ Rs . This can be solved with the inside-outsidealgorithm, which is the “sum-product variant” of the CKY algorithm. We assume the reader
is familiar with both the CKY and the inside-outside algorithms; if not, good references
are Charniak (1996a) and Manning and Schütze (1999). Essentially, these algorithms can beseen as the counterparts of the Viterbi and the forward-backward algorithms of probabilistic
sequence models to context-free parsing models.
Further Refinements: Parent Annotation, Lexicalization, Latent Variables. The construction of natural language parsers from text corpora goes back to the early 1990s, throughthe seminal works of Pereira and Schabes (1992); Black et al. (1993); Magerman (1995). The
earliest attempts to use probabilistic context-free grammars in a supervised manner, estimating their parameters via a large treebank, go back to Charniak (1996b). The performance of
these models, though, is very far from the current state of the art, and the reason is thatthey make too strong independence assumptions. A number of refinements has been madesince then toward more accurate parsers. One important refinement is parent annotation, theanalogue of increasing the Markov context in sequence models. This strategy splits each
non-terminal symbol in the grammar (e.g. Z) by annotating it with all its possible parents(e.g. creates nodes Z X , ZY , . . . every time production rules like X → Z ·, X → · Z, Y → Z ·, orY → · Z exist in the original grammar). This increases the vertical Markovian length of themodel, hence weakening the independence assumptions. Parent annotation was initiated by
Johnson (1998) and carried on in the unlexicalized parsers of Klein and Manning (2003) andfollow-up works.
In a different line of research, a major improvement was achieved thanks to lexicalization(Magerman, 1995; Eisner, 1996; Charniak, 1997; Collins, 1999), which allows to exploit word

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

20
S (plays)

plays

Logic
NP (Logic)

VP (plays)

role
a

here

minimal

N (Logic)
Logic
V (plays)

Adv (here)

NP (role)

plays

here
Det (a)Adj (minimal)

N (role)aminimalrole

Figure 2.8: Left: a lexicalized parse tree for the sentence Logic plays a minimal role here. Right:
a dependency tree for the same sentence, obtained by dropping all constituents and only
keeping the lexical annotations.

context when predicting non-terminal node spans. This technique annotates each phrase
node with the lexical item (word) which governs that phrase: this is called the head word of
the phrase. Figure 2.8 shows an example of a lexicalized parse tree. To account for lexicalization, each non-terminal symbol in the grammar (e.g. Z) is split into many symbols, eachannotated with a word that may govern that phrase (e.g. Z w1 , Z w2 , . . .). This greatly increases
the size of the grammar, but it has a significant impact on performance. We will return to
the topic of lexicalization in the next section, as it is intimately related with dependency
grammars.
As in sequence models, a significant progress was achieved by moving from generative
to discriminative models, as the latter allow the inclusion of features that depend arbitrarilyon the input sentence, with an impact on performance. Discriminative parsers have beendevised by Taskar et al. (2004b) and Finkel et al. (2008).
Splitting the variables in the grammar by introducing latent variables appears as analternative to lexicalization and parent annotation. There is a string of work concerning
latent variable grammars, both for the generative and discriminative cases (Matsuzaki et al.,2005; Dreyer and Eisner, 2006; Petrov and Klein, 2007, 2008a,b). Some related work also
considers coarse-to-fine parsing, which iteratively applies more and more refined models(Charniak et al., 2006; Petrov, 2009).
Finally, there is a totally different line of work which models parsers as a sequence of
greedy shift-reduce decisions made by a push-down automaton (Ratnaparkhi, 1999; Henderson, 2003). When discriminative models are used, arbitrary conditioning can be doneon past decisions made by the automaton, allowing to include features that are difficult tohandle by the other parsers. This comes at the price of greediness in the decisions taken,
which implies suboptimality in maximizing the desired objective function.

2.3. DEPENDENCY PARSING

*

Logic

21

plays

a

minimal

role

here

Figure 2.9: A dependency tree for the sentence Logic plays a minimal role here, drawn with thedrawdeptree.pl visualization tool developed by Terry Koo. Note the additional dummyroot symbol (∗) which is included for convenience.

2.3 Dependency Parsing
We now move to a diferent syntactic formalism, dependency grammar. Consider again thesentence Logic plays a minimal role here, along with the lexicalized parse tree displayed in
Figure 2.8 (left). If we drop the phrase constituents and keep only the head words, the parsetree would become as depicted in Figure 2.8 (right). This representation is called a dependency
tree; it can be alternatively represented as shown in Figure 2.9.
Dependency trees retain the lexical relationships involved in lexicalized phrase-basedparse trees. However, they drop phrasal constituents, which render non-terminal nodes
unnecessary. This has computational advantages (the grammar constant involved in thecomplexity of the parsing algorithms becomes much smaller) as well as design advantages(no grammar is necessary, and treebank annotations are much simpler, since no internal constituents need to be annotated). It also shifts the focus from internal syntactic structures andgenerative grammars (Chomsky, 1965) to lexical and transformational grammars (Tesnière,1959; Hudson, 1984; Mel’čuk, 1988; Covington, 1990). Dependency grammars are also related with link grammars (Lafferty et al., 1992) and head-automata (Alshawi, 1996). Lately,
this formalism has been used as an alternative to phrase-based parsing for a variety of tasks,
ranging from machine translation (Ding and Palmer, 2005) to relation extraction (Culotta
and Sorensen, 2004) and question answering (Wang et al., 2007), to name just a few.
We proceed to a formal definition of a dependency tree:6
Definition 2.2 (Dependency tree.) Let x = ( x0 , x1 , . . . , x L ) be a string, where x0 := ∗ is a special
symbol. We denote the tokens in x by integers 0, 1, . . . , L. A dependency tree for x is a directed tree
y rooted at 0 that spans {0, 1, . . . , L}.
We represent y by its set of arcs, called dependency arcs or dependency links. In an arc(h, m), the source token h is called the head and the target token m is called the modifier. The
span of (h, m) is the set of tokens between h and m, span(h, m) := {k ∈ N | min{ h, m} ≤k ≤ max{ h, m}}. The tree structure induces a partial order: we write a y d if there is adirected path in y from token a to token d, in which case a is called an ancestor of d and d isa descendant of a.
Given our early depiction of a dependency tree as the outcome of a lexicalized contextfree phrase-structured tree, one might wonder if any dependency tree can be constructed thisway. The answer turns out to be negative: the ones that can be constructed like this are thesingle-rooted, projective dependency trees, in which there is only one arc whose head is 0, and6 Some authors provide an alternative definition, which corresponds to what we call a projective dependency

tree (see Definition 2.4 in the sequel).

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

22

dependency arcs are nested (cannot cross each other); this is made formal in Definition 2.4
and Propositions 2.1–2.2. We start by the definition of a projective arc,7 and then move on toprojective trees.
Definition 2.3 (Projective arc.) Given a dependency tree y, an arc (h, m) ∈ y is called projective
if all tokens in its span descend from h. It is called non-projective otherwise.
Definition 2.4 (Projective dependency tree.) A dependency tree y is projective if all its arcs areprojective.
We say that arcs (h, m) and (h0 , m0 ) cross each other if they are not nested (span(h, m) *span(h0 , m0 ) and span(h0 , m0 ) * span(h, m)) but their spans have non-empty intersection(span(h, m) ∩ span(h0 , m0 ) 6= ∅). A dependency tree is said to be single-rooted if there is aunique arc departing from 0.
Proposition 2.1 A dependency tree is projective if and only if no arcs cross each other.
Proof. See Appendix A.1.
Proposition 2.2 A dependency tree can be constructed from a lexicalized context-free phrase-structured
tree if and only if it is projective and single-rooted.
Proof. See Gaifman (1965).
One of the reasons why it is attractive to consider the general class of dependency trees
as defined in Definition 2.2 is that in many languages (e.g., those which are free-order) the
projectivity assumption is too stringent. Even in languages with fixed word order (such as
English) there are syntactic phenomena which are awkward to characterize using projective
trees arising from the context-free assumption, such as wh-movement and general fillergap constructions. Such phenomena are commonly characterized with additional linguistic
artifacts (e.g., inserting a trace symbol in the gap position, distinguishing between deep andsurface structure, etc.).8 The allowance of non-projective arcs dispenses with these artifacts.
An example is the sentence (extracted from the Penn Treebank)
We learned a lesson in 1987 about volatility.
There, the prepositional phrase in 1987 should be attached to the verb phrase headed bylearned (since this is when we learned the lesson), but the other prepositional phrase about
volatility should be attached to the noun phrase headed by lesson (since the lesson was about
volatility). To explain such phenomena, context-free grammars need to use additional machinery which allows words to be scrambled (in this case, via a movement transformationand the consequent insertion of a trace). In the dependency-based formalism, we can get ridof all those artifacts by allowing non-projective parse trees; an example for the sentence above
is shown in Figure 2.10.
7 Other definitions of “projective arc” have been proposed in the literature; this one is due to Kahane et al.
(1998).
8 The best way of analyzing wh-movement and similar phenomena is a subject of intense debate in syntax
theories (Chomsky, 1993, 1995; Sag et al., 1999; Bresnan, 2001).

2.3. DEPENDENCY PARSING*

We

learned23alesson

in

1987

about

volatility

Figure 2.10: A parse tree which is not projective. The arc lesson → about is non-projective.

In the remaining part of this section, we discuss models and algorithms for projective
and non-projective dependency parsing. Before doing so, we shall mention that dependency
trees are often presented with label annotations on their arcs, which provide more detailed
syntactic information. For example, the arc enjoys → She could be labeled as SUBJ to denotethat the modifier She has a subject function, and the arc enjoys → school could be labeled
as OBJ to denote that the modifier school has an object function. In our thesis, we consider
dependency parsing as the problem of finding an unlabeled tree, conveying the “bare bones”
of the syntactic structure.9
Arc-Factored Models. We start by considering a simple kind of models which are calledarc-factored. These models assign a score θh,m ( x ) to each possible arc (h, m) with h 6= m and
m 6= 0. In a probabilistic model, the conditional probability of a parse tree y is given by:
P(y| x ) ∝∏ exp(θh,m (x)).

(2.13)(h,m)∈y

As usual, from the point of view of the parsing algorithm, it does not matter whether thescores come from a generative or discriminative model, or which features were used tocompute the scores. The three important inference tasks are:
1. Obtain the most likely dependency tree (the MAP inference problem),yb = arg max
y∑θh,m ( x ).

(2.14)(h,m)∈yNote that, once again, this takes the form in Eq. 2.1.
2. Compute the partition function (for a log-linear model),Z (θ, x ) = ∑∏ exp(θh,m (x)).

(2.15)y (h,m)∈y

3. Compute the posterior marginals for all the possible arcs (for a log-linear model),Prθ {(h, m) ∈ y| x },for every (h, m).

(2.16)

As we will see, all these three problems can be solved efficiently if the model is arc-factored,
both for the projective and non-projective case.
9 Several possibilities exist to extend a bare bone parser to output dependency labels: one is to use a joint
model for inferring the dependencies and labels altogether (McDonald and Pereira, 2006); another is to follow a
two-stage approach that first gets the dependencies, and then the labels (McDonald et al., 2006).

24

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

Unfortunately, the arc-factored assumption is often too severe, as it fails to capture “interactions” between arcs. These interactions are usually important: for example, two tokens
m and s, when considered individually, may both be strong candidates for dependents of averb h, for example because both look like plausible subjects for that verb. In accordance,
an arc-factored model could well predict a dependency parse tree containing both (h, m)and (h, s). However, it is usually unlikely that a verb accepts more than one subject, hence
at least one of those arcs would be an incorrect prediction. Consider now a more powerful model that includes a score (say θh,m,s ) for the simultaneous inclusion of (h, m) and (h, s).
Such a model would disfavor the presence of both arcs and could well pick only one of them,eventually achieving a correct prediction where h gets only one subject.
Pairs of arcs as above, of the form (h, m) and (h, s), are usually called siblings (because
they render tokens m and s as siblings in the tree). Another important kind of arc pairs
is those of the form ( g, h) and (h, m), which are commonly called grandparents (since they
render g grandparent of m). We next discuss how inference under these more powerfulmodels can be done, both for the projective and non-projective cases.
Projective Dependency Parsing. Given the strict relationship between projective dependency trees and lexicalized phrase-structured trees, it is not surprising that the projective
case can be addressed by what are essentially variations of the CKY algorithm, albeit with alexical flavour.10
For arc-factored models, the three tasks listed above can be solved in O( L3 ) time with
Eisner’s dynamic programming algorithm (Eisner, 1996, 2000), a clever adaptation of the
CKY algorithm that makes use of incomplete spans. A max-product variant of the algoritm
is what is necessary for the first task (MAP inference), while the last two tasks (computing
the partition function and the marginals) can be solved with the sum-product variant.
Eisner’s algorithm can be extended for models that include scores for consecutive siblingarcs on the same side of the head,11 while keeping the O( L3 ) runtime (Eisner, 1996). Models
that include such scores (in addition to scores for individual arcs) are said to use second-orderhorizontal Markov context.
More recently, Carreras (2007) addressed second-order vertical Markovization by extending
Eisner’s algorithm to handle scores for a restricted kind of grandparent arcs, albeit increasing
the asymptotic runtime to O( L4 ). Even more recently, Koo and Collins (2010) introduced
third-order dependency parsers that use grand-siblings and tri-siblings, along with an algorithm which still runs in O( L4 ) time.
Like in phrase-structure parsing, there is a totally different line of work which models
parsers as a sequence of greedy shift-reduce decisions (Nivre et al., 2006; Huang and Sagae,2010)—these are called transition-based parsers. These parsers achieve a very appealing speedaccuracy trade-off: empirically, they have been reported to exhibit expected linear runtime,
10 Note, however, that making these variations efficient is not an obvious step and was only accomplished by

Eisner (1996). The process of lexicalizing a context-free grammar consists in decorating the symbols in eachproduction rule with lexical information: for example X → YZ is multiplied into several rules of the form
X [h] → Y [h] Z [m] and X [h] → Y [m] Z [h], where h and m are lexical items, i.e., terminal symbols. When parsing
a string, there are two additional indices involved in each of these rules, hence a naı̈ve adaptation of the CKY
algorithm would render O( L5 ) runtime. The trick is to use incomplete spans (Eisner and Satta, 1999; Eisner,
2000), which reduce runtime to O( L4 ) in general bilexical phrase-structured grammars, and to O( L3 ) bilexical
split grammars.
11 These are pairs of sibling arcs, ( h, m ) and ( h, s ) such that no arc ( h, r ) exists with r between m and s.

2.3. DEPENDENCY PARSING

25

and are only slightly less accurate than the state of the art.
Non-Projective Dependency Parsing. What about the case of non-projective dependency
parsing—where the trees are not constrained to be projective? For arc-factored models,
efficient algorithms are also available for solving the three problems in Eqs. 2.14–2.16. Interestingly, these algorithms are not related with dynamic programming.
The first problem, i.e., that of computing the most likely dependency tree (MAP inference), corresponds to finding a maximum weighted directed spanning tree (also called amaximum weighted arborescence) in a directed graph. This fact has first been noted by
McDonald et al. (2005b). This problem is well known in combinatorics and can be solved inO( L3 ) using Chu-Liu-Edmonds’ algorithm (Chu and Liu, 1965; Edmonds, 1967).12
The second and third problems—computing the partition function and the marginals—
can be solved by invoking another important result in combinatorics, a weighted versionof Kirchhoff’s matrix-tree theorem (Kirchhoff, 1847; Tutte, 1984). This fact has been noticed
independently by Smith and Smith (2007); Koo et al. (2007); McDonald and Satta (2007). The
cost is that of computing a determinant and inverting a matrix, which can be done in timeO( L3 ).13 The procedure is as follows. We first consider the directed weighted graph formed
by including all the possible dependency links (h, m) (including the ones departing from thestart symbol, for which h = 0 by convention), along with weights ψh,m := exp(θh,m ( x )). We
then compute its ( L + 1)-by-( L + 1) Laplacian matrix L whose entries are:(Lhm :=∑hL0 =0 ψh0 ,m if h = m,−ψh,motherwise.

(2.17)

Denote by L̂ the (0, 0)-minor of L, i.e., the matrix obtained from L by removing the first row
and column. Consider the determinant det L̂ and the inverse matrix L̂−1 . Then:• the partition function is given by
Z (θ, x ) = det L̂;(2.18)• the posterior marginals are given by(Prθ {(h, m) ∈ y| x } =ψh,m × ([L̂−1 ]mm − [L̂−1 ]mh ) if h 6= 0
ψ0,m × [L̂−1 ]mmotherwise.

(2.19)

Unfortunately, extensions beyond the arc-factored model have a much greater impact on
runtime in the non-projective case than they have in the projective case: horizontal and vertical Markovization renders MAP inference NP-hard (McDonald and Pereira, 2006; McDonald
and Satta, 2007). Yet, approximate algorithms have been proposed to handle “second-ordermodels” that seem to work well: a projective parser followed by hill-climbing (McDonald
et al., 2006), loopy belief propagation (Smith and Eisner, 2008), and a dual decomposition
12 Asymptotically faster algorithms exist, e.g., Tarjan (1977) and Gabow et al. (1986) propose algorithms thatsolve the same problem in O( L2 ) and are even faster when only some candidate arcs are considered.
13 Again, there are faster asymptotic algorithms for carrying out this computation; e.g., an adaptation of
Coppersmith-Winograd algorithm would take O( L2.376 ) (Coppersmith and Winograd, 1990). However, it islikely that the constants hidden in the O notation are too large to be useful in practice.

CHAPTER 2. STRUCTURE IN NATURAL LANGUAGE

26

Sequence labelingPhrase structure
parsing
Dependency
parsingY
L-length strings(Λ L )
Phrase structuretrees
Dependency
trees

Parts
Unigram and bigram assignments
Anchored phrase spans and production rules(maybe lexicalized and/or parent annotated)
Candidate arcs
(maybe siblings, grandparents, etc.)

Table 2.1: Summary of the sets of outputs and parts for the tasks discussed in this chapter.
method (Koo et al., 2010). In this thesis, we present an integer programming formulation
with a polynomial number of constraints, along with an efficient algorithm for solving thelinear programming relaxation (Chapter 7). Transition-based parsers have also been adapted
to handle non-projective parsing, albeit with a slower runtime than transition-based projective parsers (Nivre, 2009).

2.4 Conclusion
In this chapter, we have described several structured problems in NLP: handwriting recognition, text chunking, named entity recognition, phrase-structure parsing, and dependency
parsing. We have focused on statistical models to tackle such problems, such as hidden
Markov models, conditional random fields, probabilistic context-free grammars, and otherstatistical parsers. We have discussed some of the existing work on this area.
There are a number of common ingredients in all these problems:• A structured output set Y from which we want to pick a prediction yb;• A set of parts R which are the building blocks of the elements of Y;• Some sort of constraints that tell which combinations of parts produce valid outputs;• A statistical model that defines a global score function, which decomposes into local
scores defined on the parts.
What the parts are and how they are entangled is what characterizes each of the problems.
Table 2.1 summarizes what these are in the several tasks discussed in this chapter. The next
chapters (Chapters 3 and 4) will make things more formal, by covering the inference andlearning problems in structured prediction.

Chapter 3

Structured Prediction and Learning
Algorithms
The models and algorithms described in this thesis are applicable to general structured prediction problems. These are problems with a strong interdependence among the output
variables, often with sequential, graphical, or combinatorial structure. Problems of this kindarise in natural language processing, computer vision, robotics, and computational biology.
The latest years have witnessed a considerable progress towards a unified formalism fortackling this kind of problems, most notably due to the seminal works of Lafferty et al.
(2001); Collins (2002a); Altun et al. (2003); Taskar et al. (2003); Tsochantaridis et al. (2004).
In this chapter, we synthesize key foundational prior work in structured prediction, providing a brief overview of the tools and concepts that will be necessary for presenting themain contributions of this thesis.1 This chapter is organized as follows. Section 3.1 presents
the basics of supervised learning. Section 3.2 describes linear models, which is the kind ofmodels that we use throughout, along with a brief review of discriminative methods. In Section 3.3, we define structured prediction rather informally, as a distinctive set of problems thatdeserve special attention. The remaining pair of sections is devoted to the problem of training structured predictors. We discuss the underlying optimization problems (Section 3.4)
and focus on online learning algorithms (Section 3.5).

3.1 Supervised Learning
Humans act and make decisions on a daily basis. Part of the inference procedure that guides
these decisions is about observing the state x of some object (the input) and using our current
understanding of the world (our model, learned from past experience) to predict some hidden
property y (the output). Hopefully, we will learn to predict better and better as we get to see
more data. This intuition carries out to machines.
We are broadly interested in learning a map h : X → Y, where X is an input set and Y isan output set. Given x ∈ X, we call yb := h( x ) a prediction. In the sequel, Y will always bediscrete; we refer to its elements as labels or classes. Simple examples are binary classification,
where Y := {0, 1}, and multi-class classification, where Y is finite and |Y| ≥ 2. In structured
prediction (to be introduced in Section 3.3), Y can be very (possibly infinitely) large. A map
1 Additional background on this subject can be found in Bakır et al. (2007) and the references therein.

27

28

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

h as just defined is called a hypothesis or a classifier. We want h to yield accurate predictions
in a sense to be made precise below.
In the supervised learning paradigm, we are given access to training data D consisting of
input-output pairs (called examples or instances):
D : = ( x 1 , y1 ), . . . , ( x N , y N ) ∈ (X × Y) N .

(3.1)

We follow the common assumption that there is an underlying joint probability distribution
P( X, Y ) that governs the generation of data, and that D is a sample drawn i.i.d. from this
distribution. The machine only gets to see the sample; the true distribution P( X, Y ) is unknown. The goal of supervised machine learning is to “learn” h from the data in D. To thisend, one considers a hypothesis class H ⊆ {h : X → Y}.2 The precise choice of H is usually
driven by our prior knowledge about the problem at hand, as well as eventual computational
limitations. This involves one or more of the following decisions: choosing the representionof inputs and outputs, designing features or a kernel function, deciding on the number ofhidden layers in a neural network, choosing the number of latent variables in a probabilistic
model, making factorization assumptions, etc. We are then led to the following problem:• Given the hypothesis class H, how to pick h ∈ H?
Obviously, one wants h to generalize well to unseen data: given a new instance ( x, y) ∼P( X, Y ) from which only x is observed (a test example), the prediction yb = h( x ) shouldbe “close” (in expectation) to the true output y. This can be formalized as follows: letρ : Y × Y → R+ be a non-negative cost function, which vanishes on the diagonal (i.e., suchthat ρ(y, y) = 0 for every y ∈ Y). The value ρ(yb, y) represents the cost incurred when thetrue output is y and the machine predicts yb. An example is the 0/1 cost:(ρ0/1 (yb, y) :=1 if yb 6= y0 otherwise,(3.2)

which can be written compactly as ρ0/1 (yb, y) = [[yb 6= y]], where [[π ]] is 1 if predicate π is true,and 0 otherwise. One wants a classifier h that incurs as little cost as possible on unseen data.
This is captured through the following notion:
Definition 3.1 (Expected and empirical risk.) Let EX,Y denote the expectation operator underP( X, Y ). The expected risk of a classifier h (with respect to a cost function ρ) is the quantity
risk(h; ρ) := EX,Y [ρ(h( X ), Y )].

(3.3)

Given training data D, the empirical risk of h isriskD (h; ρ) := ED [ρ(h( X ), Y )] =1 N
ρ ( h ( x n ), y n ).
N n∑
=1

(3.4)

2 There is a variety of learning algorithms which are characterized by the different kinds of hypothesis classes

they operate on; examples are nearest neighbor classifiers, decision trees, multilayer neural networks, or linear
models. In this thesis we will focus on the last—cf. Section 3.2. There are numerous textbooks that provide
a background on several of the learning formalisms mentioned above, e.g., Mitchell (1997); Duda et al. (2001);
Schölkopf and Smola (2002); Bishop (2006).

3.2. LINEAR MODELS

29

Figure 3.1: A regression problem with N training examples, drawn as blue circles. Test
examples are drawn as white circles. Our hypothesis classes are polynomials up to somedegree D. As we increase the complexity of the hypothesis class (i.e., as we increase D),
we obtain classifiers that fit the training data better, thus reducing the empirical risk—inparticular, with polynomials up to degree D = N − 1 zero empirical risk can always beachieved. However, this increases the danger of overfitting the training data, leading topoor generalization (right). This is in contrast with a too small D, which would lead tounderfitting (left). A general machine learning problem is to find an equilibrium between
these two extremes (middle).
When the cost function is clear from the context, we omit the dependency of the expected
and empirical risk on ρ, writing these quantities simply as risk(h) and riskD (h). An optimal
decision would be to pick a classifier h∗ ∈ H which minimizes the expected risk. However,since the true distribution P( X, Y ) is unknown, this risk cannot be assessed. The best we
have is the empirical risk; yet, optimizing this risk directly is usually problematic for a coupleof reasons:
1. For many cost functions (e.g., the 0/1 cost) empirical risk minimization leads to a difficult combinatorial problem, which becomes intractable even for “simple” hypothesis
classes (such as linear models).
2. If the training set D is too small and/or the hypothesis class H has too large a “complexity,” then the classifier minimizing the empirical risk may overfit the data. Thismeans that the gap between the empirical and expected risks can be too large, so thatthe former is no longer a good indicator of the quality of the classifier. Typical bounds√
on this gap grow as fast as O( D/N ), where D is a measure of complexity3 of H and
N = |D|. Figure 3.1 illustrates this point.
The first problem is commonly sidestepped by replacing riskD (h) by a surrogate loss
function more amenable to optimization. The second problem is avoided by introducing a
regularizer that penalizes the “complexity” of a hypothesis. This will be made clear in thenext section, where we discuss linear models.

3.2 Linear Models
When one is faced with the choice of a model for supervised learning, the class of linear
models4 appears at the front line. Linear models have several virtues: simplicity, ease of
3 There are many ways we can assess the complexity of a hypothesis class from the point of view of statisticallearning theory. Examples are the Vapnik-Chervonenkis (VC) dimension (Vapnik, 1998) and the Rademacher
complexity (Anthony and Bartlett, 2009). See Schölkopf and Smola (2002) for further details on VC-theory andstatistical learning.
4 These models are also commonly referred to as generalized linear models (Hastie et al., 2001). The term
generalized comes from the fact that, in regression, they generalize standard linear regression models through theuse of a link function.

30

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

interpretation, and the ability of being rendered non-linear through the use of a kernel
function. We consider a joint feature map f : X × Y → RD that represents input-output pairs
as vectors of features in a D-dimensional Euclidean space. The actual choice of features isusually driven by practical considerations and it is where the practicioner should exploit her
knowledge about the problem at hand. Features can take multiple forms:• integer values associated with counts of events observed in ( x, y) ∈ X × Y;
• binary-valued indicators that some predicate π holds in ( x, y);• categorical features, usually converted to an array of binary values (as many as thenumber of categories);• real-valued statistics, measurements, or confidence values (such as log-probability estimates) output by another model run at a previous stage.
In linear models, the hypothesis class H := {hw : X → Y | w ∈ RD } is comprised of linear
discriminant functions of the form:5hw ( x ) := arg max w · f ( x, y),
y ∈Y

(3.5)

where w ∈ RD is a parameter vector and · denotes the standard inner product in RD , w ·f ( x, y) := ∑dD=1 wd f d ( x, y). Intuitively, each parameter wd is a weight for feature f d ( x, y) (andtherefore w is also commonly called a weight vector).6
The problem in Eq. 3.5 is called inference or decoding. In binary and multi-class classification with few classes, this problem is easily solved by enumerating all the scoresθ (y) := w · f ( x, y) for each y ∈ Y, and picking the class with the largest score. The caseis different in structured prediction, however, due to the hardness-of-enumeration assumption,
to be described in Section 3.3.
We now turn to the problem of learning the model parameters w. Following up on thelast section, this can be formulated as an optimization problem of the form7
minimize Ω(w) + N1 ∑nN=1 L(w; x n , yn )w.r.t.
w ∈ RD ,(3.6)

where Ω : RD → R is a regularizer and L : RD × X × Y → R is a loss function. Problem 3.6
is called learning or training. One of the nice aspects of linear models is that it is simple toformulate learning as a convex optimization problem, once we make a suitable choice of Ωand L (namely, both need to be convex functions). This brings two great benefits: first, we
can use the machinery of convex optimization algorithms to solve Eq. 3.6 efficiently. Second,
we do not need to worry about local optima (since every local minimum will be a global
minimum). This is an important advantage over more complex models, such as multilayer
neural networks, or some probabilistic models with latent variables.
5 This notation is sloppy because there may be multiple arguments maximizing the objective. We assume thatties are broken arbitrarily using some predefined fixed rule.
6 These models can be “kernelized” by letting H be a reproducing kernel Hilbert space with kernel K :(X × Y)2 → R. We will see instances in Chapter 9.
7 This is commonly referred in the literature as regularized empirical risk minimization. We note however thatthe term “empirical risk” enclosed in this expression does not usually refer to the original cost function (e.g., the0/1 cost) but to a surrogate function which is tractable to optimize.

3.2. LINEAR MODELS

3.2.1

31

Regularization

A natural choice for the regularizer Ω is a norm, possibly raised to a power q ≥ 1: this function is always guaranteed to be convex (see, e.g., Boyd and Vandenberghe 2004, pp. 73, 86),
and penalizes the magnitude of the weight vector, which can be interpreted as a penalizationof the complexity of the classifier. The most typical choices are:8• L2 -regularization: ΩλL2 (w) := λ2 kwk22 ;• L1 -regularization: ΩτL1 (w) := τ kwk1 ,where λ and τ are non-negative scalars—called regularization coefficients—that control theintensity of regularization. These coefficients are typically tuned with held-out data or crossvalidation. Both regularizers have first been studied in the context of ill-posed problems. The
use of L2 -regularization goes back to Tikhonov (1943) and, in the context of regression, iscommonly referred as ridge regularization. In NLP, it has been used in classification tasks by
Chen and Rosenfeld (1999, 2000), among others, for regularizing maximum entropy models.
While L2 -regularization is often robust to the inclusion of irrelevant features, such features
typically get nonzero weights (leading to a dense weight vector w as a solution of Eq. 3.6).
L1 -regularization was popularized by Tibshirani (1996) in the context of sparse regression,
being called Lasso regularization.9 It has been used by Kazama and Tsujii (2003) and Goodman(2004) as an alternative to L2 -regularization in maximum entropy models. Unlike the L2 case,this usually leads to a sparse weight vector w as a solution of Eq. 3.6, where some weights
become exactly zero—consequently, the corresponding features can be discarded from themodel. There are a couple of reasons why this can be desirable:• It yields a more compact model, with less memory requirements;• It allows pinpointing the relevant features, yielding model interpretability.
An empirical comparison between these two regularization strategies has been presented by
Gao et al. (2007). Other related regularizers have been proposed, such as elastic nets (Zou andHastie, 2005), which are an interpolation of L1 and L2 . These have been used by Lavergne
et al. (2010) to regularize conditional random fields. In Chapter 10, we will introduce otherregularizers, that are able to promote structured sparsity.

3.2.2

Loss Functions for Discriminative Learning

Let us now discuss the loss function L in Eq. 3.6. In linear models, it is assumed that
L(w; x, y) depends on w only via the inner product w · f ( x, y).
Most learning methods are either generative or discriminative. The ones of the first kinddesign the loss function through a “generative story” about how data is generated; for example, they may model the joint probability P( X, Y ) and use that to define the loss function
8 Both regularizers have a Bayesian interpretation. In probabilistic models, the loss term L usually corresponds
to the negative log-likelihood of a (joint or conditional) probability distribution parameterized by the weightvector w. Bayesians would rather see w as another random variable instead of a “parameter vector,” and do
so by defining a prior belief P(w) on the weights. In that case, the regularizer Ω can be interpreted as thenegative logarithm of this prior distribution, and the solution of Eq. 3.6 as the maximum a posteriori estimate ofw. Under this lens, ΩλL2 corresponds to using independent zero mean Gaussian priors for each feature weightwd ∼ N(0, λ−1 ), and ΩτL1 corresponds to zero-mean Laplacian priors, p(wd ) ∝ exp(τ |wd |).
9 For least absolute shrinkage and selection operator.

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

32(this is the procedure that underlies, e.g., the naı̈ve Bayes method). In contrast, discriminative
methods shift the focus from data generation to output prediction. Comparisons between
generative and discriminative methods for different problems and regimes have appeared
several times in the literature (Ng and Jordan, 2002; Liang and Jordan, 2008), and fusions
between the two kinds of methods have also been proposed (Jaakkola and Haussler, 1998;
Jebara et al., 2004; Martins et al., 2009a). Since the focus of this thesis is on discriminative
methods, we describe only those.
Logistic Regression. A log-linear model is a probabilistic model that takes the form:
Pw (y| x ) :=1
exp(w · f ( x, y)),Z (w, x )(3.7)

where Z (w, x ) := ∑y0 ∈Y exp (w · f ( x, y0 )) is a normalization factor. This model can be fit to
the data by maximizing the conditional log-likelihood. This procedure leads to the following
logistic loss function, which is plugged in Eq. 3.6:

LLR (w; x, y) := −w · f ( x, y) + log ∑ exp w · f ( x, y0 ) .

(3.8)

y 0 ∈Y

These are also called maximum entropy models, since the dual optimization problem of Eq. 3.6
with L = LLR becomes that of picking a distribution P(Y | X ) with maximal entropy subject
to first-order moment matching constraints (i.e., constraints of the kind EY | x ( f ( x, Y )) =N
1n n
N ∑n=1 f ( x , y )), up to some slack which depends on the regularizer. This is a manifestationof the conjugate duality relation that we revisit in Section 4.3.1.
Support Vector Machines. If all we want is a model for prediction, probabilistic models are
not a real necessity, and we may well go without them.10 Support vector machines are purely
discriminative classifiers based on the principle of maximizing the margin of separation. We
describe here the multi-class formulation put forth by Crammer and Singer (2002), which
addresses the following learning problem:
minimizew.r.t.
s.t.

λ1 N
k w k2 +
ξn
2
N n∑
=1

(3.9)

N
w ∈ R D , ξ ∈ R+

w · f ( xn , yn ) ≥ w · f ( xn , y0n ) + ρ(y0n , yn ) − ξ n ,∀n ∈ {1, . . . , N }, ∀y0n ∈ Y.

To better understand what Eq. 3.9 is optimizing, let us regard the variables ξ n as slack variables. The quantity 1/kwk corresponds to the separation margin, which we seek to maximize.
With ξ n = 0, the constraints require that the parameters w are such that the score of the true
output yn exceeds that of any competitor y0n 6= yn by an amount of at least the value of the
10 A good reason for doing so is that, from the viewpoint of statistical learning theory, estimating a probability

distribution is a more complex problem than that of training a classifier with good generalization capability. It
may not be wise to embrace such a more complex problem as an intermediate step (Vapnik, 2000). However,there might be reasons for considering probabilistic models, since they have a very appealing semantics, welldefined procedures for dealing with hidden variables, and are able to complement predictions with well-foundedmeasures of confidence (such as a posterior distribution over outputs).

3.3. WHAT IS STRUCTURED PREDICTION?

33

cost ρ(y0n , yn ). When ξ n > 0, some violations of these constraints are permitted, but they pay
a linear penalty (cf. the term ∑nN=1 ξ n in the objective).
We can rewrite Eq. 3.9 in the form of Eq. 3.6 by letting Ω be a L2 -regularizer, and defining
the following hinge loss function:00
LSVM (w; x, y) := −w · f ( x, y) + max
w
·
f
(
x,
y
)
+
ρ
(
y
,
y
)
.
0
y ∈Y

(3.10)

Comparing Eqs. 3.8 and 3.10, we observe that the logistic and hinge losses have resemblances: where the former uses a soft-max, the latter uses a max; and the hinge loss explicitly
includes the cost function in the scope of this max. We will revisit this comparison in Chapter 8. Finally, it is worth mentioning that, even though support vector machines are mostcommonly used with L2 -regularization, that is not a strict necessity.11 We may arbitrarily
cross the loss functions described in the last and current sections with the regularizers seenin Section 3.2.1.

3.3 What is Structured Prediction?
In Chapter 2, we described several instances of structured prediction problems. The term
is usually employed to denote classification problems where Y is a very large set, endowedwith some sort of structure. But what does “structure” mean in this context? Why does structured prediction deserve special treatment? What is there that precludes a straightforward
adaptation of the tools and techniques that already exist for multi-class classification?
We address these questions by explicitly formulating the assumptions that, in the scopeof this thesis, characterize structured prediction problems.
• First assumption: input-specific outputs. When there are just a few classes, we predict y by searching from the entire output set Y; this is commonly inappropriate instructured prediction. For example, in sequence labeling problems (Section 2.1), some
outputs are structurally impossible: if x is a sequence of length L, then the corresponding y must be of the same size. In parsing tasks (Sections 2.2–2.3), we know that theyield of a parse tree must be the input sentence.
To take this into consideration, we denote by Y( x ) the set of admissible outputs for aSgiven x ∈ X. We define Y = x∈X Y( x ). We assume that the true distribution P( X, Y )vanishes on non-admissible pairs: that is, we must have P( x, y) = 0 if y ∈/ Y( x ).
• Second assumption: hardness of enumeration. We assume that the output set Y( x )is so large that it is intractable to enumerate its elements. In sequence labeling, |Y( x )|grows exponentially with the length of the input string x; in dependency parsing,
the number of valid parses for a sentence with L words is ( L + 1) L−1 , which is evenworse.12
11 See Bradley and Mangasarian (1998) for L -regularized variants of support vector machines. However, note1that those variants cannot be kernelized: the Representer Theorem of Kimeldorf and Wahba (1971) hinges on
L2 -regularization. For more details, see Schölkopf and Smola (2002).
12 This is a consequence of Cayley’s formula (Borchardt, 1860; Cayley, 1889), which states that the number of
undirected spanning trees of a complete graph on N vertices is N N −2 . The result follows from observing that a
non-projective dependency tree spans L + 1 words (including the dummy root word) and the orientation of theedges is automatically determined given this root word.

34

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS
We assume that |Y( x )| grows superpolynomially with some meaningful measure of“length” for the input x (for example, the number of bits necessary to encode x in amachine). For every x ∈ X, we assume that there is an integer L( x ) such that Y( x )can be represented as a subset of a product set Y1 × . . . × Y L(x) , where each Yi is finite.
L( x )

Hence, outputs y ∈ Y( x ) can be seen as tuples of atomic components, y := (yi )i=1 .
• Third assumption: global coupling. There are strong statistical dependencies among theatomic components. In other words, we assume that the true joint distribution cannot
L( x )
be accurately approximated as P( X, Y ) ≈ P( X ) ∏i=1 P(Yi | X )—otherwise, the global
problem could be trivially split as a sequence of local atomic ones. Besides this, insome tasks (such as parsing) there may be global structural constraints regarding whichtuples are valid representations of elements of Y( x ).
To sum up, we assume that the atomic output variables Y1 , . . . , YL(x) are “globally”
coupled through structural constraints, statistical interdependencies, or both.
Putting together all aspects above, we arrive at the following informal definition:
Definition 3.2 (Structured prediction.) Structured prediction refers to a classification problem inwhich the three assumptions above are met.
According to this definition, the “structure” emanates from the global coupling assumption; and the inadequacy of standard multi-class classification methods stems from the hardness of enumeration assumption, which, among other things, implies that the inference
problem in Eq. 3.5 cannot be solved efficiently by brute force.
We distinguish between two kinds of approaches that have been taken to address structured prediction problems: greedy methods and global optimization methods.
Greedy methods. These methods aim to approximate the inference problem in Eq. 3.5 byperforming a sequence of greedy decisions. Typically, atomic outputs are predicted one
at the time, and predictions made in the past are included as features to help make later
decisions—these are commonly called history-based features. These approaches have been
pursued in sequence labeling (Giménez and Marquez, 2004), phrase-structure parsing (Blacket al., 1993; Magerman, 1995; Ratnaparkhi, 1999), and transition-based dependency parsing(Yamada and Matsumoto, 2003; Nivre et al., 2006; Huang and Sagae, 2010; Goldberg andElhadad, 2010). The main advantages of greedy methods are: (i) their ability to incorporate
non-local contextual features, and (ii) their computational efficiency. If we ignore the timenecessary to evaluate features and compute scores, the prediction runtimes of greedy methods are essentially the same as if the global coupling assumption were ignored. They also
permit efficient learning, once the data instances are mapped into a sequence of actions.13
It all boils down to training one or several binary of multi-class classifiers. The great bottleneck of greedy methods is error propagation: if a mistake is made early on, then all future
decisions will be severely affected—this leads to burst errors, which are often undesirable.
Recently, search-based methods (e.g., Pal et al. 2006; Daumé et al. 2009) were proposed that
integrate learning and search, while reducing greediness by using beam search, A∗ search,
13 It is worth noting, however, that this mapping is usually one-to-many, i.e., different actions may lead to thesame instance. Heuristics are commonly used to pick a canonical sequence of actions.

3.4. DISCRIMINATIVE LEARNING OF STRUCTURED PREDICTORS

35

or other combinatorial search algorithms that are able to backtrack. While these techniques
can alleviate error propagation and they can “learn” to do better inference, this comes at thecost of increasing runtime, and there is usually a trade-off between these two.

Global optimization methods. These methods attack directly the optimization problem in
Eq. 3.5. Solving this problem exactly (exact decoding) usually requires designing the model ina clever way, by making certain factorization assumptions that limit the interactions between
atomic outputs. In sequence labeling and parsing tasks, well-known examples are hidden
Markov models (Rabiner, 1989), conditional random fields (Lafferty et al., 2001), stochastic context-free grammars (Magerman, 1995; Eisner, 1996; Charniak, 1997; Johnson, 1998;
Collins, 1999), and arc-factored dependency parsers (McDonald et al., 2005b); all these werereviewed in Chapter 2 and correspond to some decomposition of the formhw ( x ) = arg max∑ w · f p (x, y p ),y ∈Y( x ) p ∈P

(3.11)

where P is a set of places. A suitable way of representing the model factorization is via
graphical models, as will be seen in detail in Chapter 4; there, the places correspond to thefactors of a factor graph. The main advantage of global optimization methods over greedy
methods is that they do not suffer from error propagation. The price to pay is that the model
factorizations that permit tractable inference are often too stringent. Typically, designingand extending the model are tasks that have to obey algorithmic considerations. It is often
worthwhile to embrace approximate inference techniques (non-greedy ones) to tackle those
problems. All this will be discussed in Sections 4.5–4.6.
The approaches investigated in this thesis are based on global optimization. In particular,
we devise approximate inference algorithms that can handle the sort of non-local features thatare typically incorporated in greedy methods, hence alleviating what is seen as the maindrawback of global optimization methods for structured prediction.

3.4 Discriminative Learning of Structured Predictors
We now adapt the supervised learning framework described in Sections 3.1–3.2 for learning
structured predictors. We assume a linear model where the features decompose asf ( x, y) := ∑ f p ( x, y p ),(3.12)

p ∈P

where each p ∈ P is called a place, and each pair ( p, y p ) is called a part. The partial output
y p takes values in a set Y p , which is typically a product set of atomic outputs (for example,
in sequence models, places refer to unigrams and bigrams). We assume that the set ofplaces P includes the atomic places i ∈ {1, . . . , L( x )} mentioned above. This will be mademore formal in Chapter 4, when we talk about factor graphs. With this decomposition, theinference problem becomes that of Eq. 3.11.

36

3.4.1

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

Structured Cost Functions

Let us now discuss the cost functions that are suitable for structured prediction problems.
One thing to note is that the 0/1 cost function, which is the typical choice in multi-classclassification problems, is in general inadequate in structured prediction, because it does
not give partial credit for getting most of the structure right. Toward this end, we consider
decomposable cost functions.14
Definition 3.3 (Decomposable cost.) A cost function ρ : Y × Y → R+ is decomposable if it canbe written asρ(yb, y) = ∑ ρ p (ybp , y p ),(3.13)p ∈P

for some choice of “local” cost functions ρ p : Y p × Y p → R+ .
L( x )L( x )

Example 3.1 (Hamming cost.) Let yb = (ybi )i=1 and y = (yi )i=1 . The Hamming cost is definedas:L( x )

ρ H (yb, y) = ∑ [[ybi 6= yi ]](3.14)

i =1

In words, ρ H (yb, y) returns the number of mismatched atomic outputs. Assuming, as stated above,that the set of places include the atomic places, then we have that the Hamming cost is decomposable,with ρ p ≡ 0 for every non-atomic place p, and ρ{i} ≡ ρ0/1 for every i ∈ {1, . . . , L( x )}.
We point out, however, that decomposable cost functions are not always the most suitable
for some problems—while the Hamming cost gives a measure of error rate regarding how
much of the structure was predicted correctly, for some problems metrics based on precision,
recall, or Fβ -measure are the most suitable. While precision and recall can both be captured
with decomposable costs, that is not the case with Fβ .

3.4.2

Conditional Random Fields

A conditional random field or CRF (Lafferty et al., 2001) is a generalization of a logistic regression model for structured prediction. One considers a conditional probabilistic model thattakes the form in Eq. 3.7, but where f ( x, y) has a decomposition as in Eq. 3.12.15 This yields:
Pw (y| x ) :=
1
exp  ∑ w · f p ( x, y p ) ,Z (w, x )p ∈P( x )(3.15)

where Z (w, x ) is the partition function.
14 To accommodate the fact that sets of admissible outputs may be input dependent, we assume that any costfunction ρ : Y × Y is infinity-valued for pairs (yb, y) for which there is no x such that both yb and y belong to Y( x );in what follows we abbreviate and define ρ as if its domain was Y( x ) × Y( x ).
15 Strictly speaking, in a CRF the decomposition is with respect to the cliques of a Markov network (which will
be defined in Chapter 4), hence P(Y | X ) forms a Markov random field (hence the name). However, it is common
to say “conditional random field” even when the underlying structure is not described by a Markov network,
as soon as it decomposes as in Eq. 3.12. An example is “CRF parsing” (Finkel et al., 2008), where the structure
comes from a context-free grammar.

3.4. DISCRIMINATIVE LEARNING OF STRUCTURED PREDICTORS

37

The loss function of conditional random fields (called structured logistic loss) corresponds
to the negative conditional log-likelihood, which can be written as the soft-max (log-sumexp) of a term which decomposes into parts:
LCRF (w; x, y) := −w · f ( x, y) + log

exp w · f ( x, y0 )∑0y ∈Y( x )=log

exp ∑ w · δ f p ( x, y p , y0p )∑0y ∈Y( x )!
(3.16)

p ∈P

where δ f p ( x, y p , y0p ) := f p ( x, y0p ) − f p ( x, y p ). Given a model trained by minimizing the(regularized) structured logistic loss, there has been two decoding strategies proposed in theliterature: MAP decoding and minimum risk decoding.
MAP decoding. MAP decoding (also called Viterbi decoding) seeks the configuration withmaximal probability according to the model:
yb := arg max Prw {Y = y| x }.

(3.17)y ∈Y( x )Minimum risk decoding. Minimum risk decoding is a decoding rule that can be usedin any probabilistic model, and which is commonly used with CRFs. It corresponds topredicting the output yb ∈ Y( x ) which minimizes the decision risk Ew (ρ(yb, Y )). Under a 0/1
cost, it reverts to the MAP decoding rule. When a Hamming cost is used, the objective tominimize becomes:!
L( x )

Ew (ρ H (y, Y )) = Ew

∑ ρ0/1 (yi , Yi )i =1

L( x )= ∑ Ew (ρ0/1 (yi , Yi ))i =1

L( x )=∑ (1 − Prw {Yi = yi |x}),(3.18)

i =1

hence under a Hamming cost, the minimum risk decoding rule becomes:L( x )

yb := arg max ∑ Prw {Yi = yi | x },y ∈Y( x ) i =1

(3.19)

in other words, one needs to compute the posterior marginals Prw {Yi = yi | x } and thenobtain the configuration that maximizes the sum of marginals. When the outputs are unL( x )constrained, i.e., Y( x ) ∼= ∏i=1 Yi , this rule amounts to maximizing the posterior marginals
componentwise, ybi = arg maxyi ∈Yi Prw {Yi = yi | x }.

3.4.3

Structured Support Vector Machines

Support vector machines can be also generalized for structured prediction, as shown by
Taskar et al. (2003); Altun et al. (2003); Tsochantaridis et al. (2004). We present here the formulation of Taskar et al. (2003), which takes the name max-margin Markov networks. Rather
than the likelihood of a probabilistic model, the goal is now to maximize the separation

38

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

margin, with respect to a cost function ρ, which we assume to be decomposable. The corresponding loss function is called structured hinge loss and is written as:
LSSVM (w; x, y) := −w · f ( x, y) + max w · f ( x, y0 ) + ρ(y0 , y)0y ∈Y( x )= max  ∑ w · δ f p ( x, y p , y0p ) + ρ p (y0p , y p ) 
y 0 ∈Y( x )(3.20)p ∈P( x )where δ f p ( x, y p , y0p ) is defined similarly as in Eq. 3.16. Comparing Eq. 3.16 with Eq. 3.20, we
observe only two differences:• The soft-max operator in Eq. 3.16 is replaced by a max operator in Eq. 3.20;• Eq. 3.20 includes the cost function in the scope of the maximization.
We will come back to this issue in Chapter 8, where we will devise a family of loss functions
that interpolate between conditional random fields and structured support vector machines.

3.5 Online and Stochastic Learning Algorithms
Now that we have discussed different learning formalisms for structured prediction, expressing them as optimization problems, it is time to describe learning algorithms for tacklingthese problems. Recall the general optimization problem of Eq. 3.6:
minimize Ω(w) + N1 ∑nN=1 L(w; x n , yn )w.r.t.
w ∈ RD .

(3.21)

An important requirement is that algorithms are scalable, i.e., they must have reasonable
memory consumption needs and manageable runtime when N and D are large. This requirement rules out some of the fast optimizers, such as interior-point methods, which havesuperlinear convergence guarantees. On the other hand, algorithms with slower convergence rates, such as the gradient method or its stochastic/online counterparts, look suitable
for large-scale machine learning: even though they are usually slow to converge asymptotically, they are fast to reach a region close to the optimum, hence can be stopped early andretrieve a model with good generalization properties. Bottou and Bousquet (2007) describe
in more detail this tradeoff between approximation, estimation, and optimization error;16 theinclusion of the latter is distinctive of large-scale settings.
The development of algorithms for learning structured predictors is a very active areaof research. We limit our discussion to the algorithms that will be used in our thesis; weomit others, such as the exponentiated gradient (Collins et al., 2008) and the primal and dual
extragradient (Taskar et al., 2006a,b), which are amongst to the most competitive for this task.
We review in Appendix B some basic concepts of convex analysis that are used throughout the remaining sections.
16 Approximation error is the difference between the minimum expected risk (for some oracle h : X → Y notnecessarily in the hypothesis class H) and the best achievable one in H; estimation error is the difference between
the best achievable expected risk and that of the hypothesis minimizing Eq. 3.21; optimization error is the difference between the latter term and the actual model which is returned by an algorithm optimizing Eq. 3.21 up toa tolerance.

3.5. ONLINE AND STOCHASTIC LEARNING ALGORITHMS

3.5.1

39

Stochastic Gradient Descent

For each instance n, define Fn (w) := Ω(w) + L(w; x n , yn ), and F (w) := N1 ∑nN=1 Fn (w). The
learning problem in Eq. 3.21 corresponds to an unconstrained minimization of F (w). Let us
assume for a moment that F : RD → R is differentiable, and denote by ∇ F (w) the gradientof F at w, which is given by ∇ F (w) = N1 ∑nN=1 ∇ Fn (w). A gradient method for solving thisproblem would iteratively pick a stepsize ηt and apply an update rule of the formw t +1 ← w t − η t ∇ F ( w t ).

(3.22)A proper choice of stepsizes ensures a convergence rate of O(1/e), in the sense that at
most T = O(1/e) iterations are necessary to achieve an objective value which differs no
more than e from the optimum (this difference is the optimization error). With accelerated
gradient techniques (Nesterov, 1983) it is even possible to achieve a faster convergence rate√of O(1/ e). However, a single iteration of these methods requires processing an entire batchof data for computing the gradient ∇ F (w). This makes the gradient method unsatisfying ina large-scale setting, where N is large. One possibility to sidestep this issue is to parallelize
the computation; another, which we describe next, is to use stochastic gradients.
Stochastic gradient methods (Robbins and Monro, 1951; Bottou, 1991a, 2004) replace each
gradient evaluation with a “noisy” estimate built from a single instance. In other words, ateach iteration t (called round) they pick an instance n(t) uniformly at random, with replacement; then the following update is made based only on that instance:
wt+1 ← wt − ηt ∇ Fn(t) (wt ),(3.23)

When this choice is not done at random but instead cycles over the data through n(t) =t − N bt/N c, the method is called online gradient (or incremental gradient, in the optimization
literature).17 In stochastic and online gradient methods, one can obtain convergence for asuitable choice of the stepsize sequence (ηt )t∈N , which in this context is commonly called the√learning rate. In general, by choosing ηt = O(1/ t), one can guarantee O(1/e2 ) convergence(in a PAC18 setting), for arbitrary convex loss functions and regularizers. This result can beestablished following the analysis of Zinkevich (2003) for online convex programming, andadopting a simple online-to-batch conversion that averages the iterates (Cesa-Bianchi et al.,2004). That convergence rate can be improved if the objective function is λ-strongly convex(which always happens if we use L2 -regularization, Ω := ΩλL2 , provided the loss function L isconvex), in which case we have O(1/(λe)) convergence with ηt := O(1/(λt)). The analysis
is due to Hazan et al. (2007) and is essentially the same that underlies the Pegasos algorithm(Shalev-Shwartz et al., 2007, 2010).
Stochastic gradient descent algorithms have been proposed as a way of training L2 regularized CRFs in large-scale settings (Vishwanathan et al., 2006), achieving in practice
faster progress than more sophisticated batch optimization methods that were previously
used, such as conjugate gradient and L-BFGS (Sha and Pereira, 2003). With Ω := ΩλL2 , the17 There are also methods that use mini-batches instead of a single instance. The theoretical analysis of suchmethods is very similar to that of stochastic/online gradient descent.
18 “Probably approximately correct” (Valiant, 1984).

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

40

Algorithm 1 Stochastic Gradient Descent for L2 -regularized CRFs/SVMs
1: input: D, λ, loss function L, number of rounds T, learning rate sequence (ηt )t=1,...,T
2: initialize w1 : = 0
3: for t = 1 to T do
4:choose n = n(t) randomly and take training pair ( x n , yn )e w L(wt , x n , yn ) (Eqs. 3.25 and 3.27)
5:computing the gradient or a subgradient g := ∇6:
update the model:
wt+1 ← (1 − ληt )wt − ηt g
7: end forT

8: output: the last weight vector w
b ← w T +1 , or the average w
b ← T1 ∑t=1 wt+1

gradient of Fn at w becomes ∇ Fn (w) = λw − ∇w L(w, x n , yn ), hence the updates become
wt+1 ← (1 − ληt )wt + ηt ∇w L(w, x n , yn ).

(3.24)

This yields Algorithm 1. When applied to CRF training, one needs to compute the gradientof the structured logistic loss, which is19

∇ LCRF (w, x, y) = − f ( x, y) + ∇w log Z (w, x )= − f ( x, y) + Ew ( f ( x, Y )),(3.25)

where Ew ( f ( x, Y )) = ∑ p∈P ∑y p ∈Y p (x) µ p (y p ) f p ( x, y p ) is the vector of feature expectations; tocompute this vector, all that is necessary is the set of posterior marginals:µ p (y p ) := Prw {Yp = y p | x }.

(3.26)

Hence, each round of the Algorithm 1 requires performing a marginal inference step in order
to compute the gradient. This is the step that usually requires the most computational effort.
In Chapter 8, we will consider problems for which marginal inference is not tractable andhas to be approximated. We will also introduce an algorithm which does not require thespecification of a learning rate.

3.5.2

Stochastic Subgradient Descent

An analogous procedure can be applied for learning structured support vector machines.
There is a small twist though, which is the fact that the structured hinge loss is not differentiable. However, the same algorithm works with subgradients, as shown by Ratliff et al. (2006)
and Shalev-Shwartz et al. (2007, 2010). Let ∂F (w0 ) be the subdifferential of F at a point w0 ,e F (w0 ) ∈ ∂F (w0 ) a particular subgradient; define analogously ∂LSSVM (w0 )and denote by ∇eand ∇ LSSVM (w0 ) for the loss function LSSVM . From Danskin’s theorem (see Appendix B), we
have thatf ( x, yb) − f ( x, y) ∈ ∂LSSVM (w, x, y),(3.27)

19 We use a result to be derived in Chapter 4 that states that the gradient of the log-partition function equals

the vector of feature expectations (cf. Eq. 4.30).

3.5. ONLINE AND STOCHASTIC LEARNING ALGORITHMS

41

where yb is a solution of the cost-augmented inference problem:
yb = arg max

y 0 ∈Y( x )w · f ( x, y0 ) + ρ(y0 , y) .

(3.28)The update equations become:
wt+1 ← (1 − ηt λ)wt + ηt f ( x n(t) , yn(t) ) − f ( x n(t) , ybn(t) ) .

(3.29)√The same rule as above applies for the learning rates: by choosing ηt = O(1/ t), one
can guarantee O(1/e2 ) convergence; since we are considering L2 -regularization, a better
asymptotic convergence of O(1/(λe)) can be achieved with a faster decaying scheduleηt = O(1/(λt)). The latter is essentially the Pegasos algorithm (Shalev-Shwartz et al., 2007,2010).20 The similarity with the CRF case is striking: at each round, instead of computing
the feature vector expectation using the current model, we compute the feature vector associated with the cost-augmented prediction using the current model. If the cost function isdecomposable, then this step can be solved with an inference algorithm for computing the
MAP assignment: all that is necessary is to redefine the scores for each part by adding to
w · f p ( x, y0p ) the quantity ρ p (y0p , y p ), and then call a MAP decoder. Throughout this thesis,
we will see multiple problems for which this step is not tractable, and approximate MAP
inference is necessary. We defer details for Chapters 5, 7 and 8. This is also the step thatusually requires the most computational effort.

3.5.3

The Perceptron Algorithm

Perhaps the oldest algorithm for training a linear classifier is the perceptron algorithm (Rosenblatt, 1958), depicted as Algorithm 2. The perceptron was first applied to structured prediction by Collins (2002a). The algorithm works as follows: at each round, it takes an input
datum, and uses the current model to make a prediction. If the prediction is correct, nothing
happens (the update in line 6 will be vacuous in that case). Otherwise (i.e., if the model madea mistake), a correction will take place by adding to w the feature vector with respect to thecorrect output and subtracting the feature vector with respect to the predicted (wrong) output. Algorithm 2 is remarkably simple; yet it often reaches a very good performance, usually
not much worse than that obtained with CRFs or SSVMs. The most important property ofthe perceptron algorithm is its ability to find a separating hyperplane, as we will see next.
Definition 3.4 (Separability.) We say that the training data D is separable with margin γ > 0 (orsimply γ-separable) if there is a weight vector w ∈ RD with unit norm (kwk = 1) such thatw · f ( x n , yn ) > w · f ( x n , y0 ) + γ,∀n ∈ {1, . . . , N }, ∀y0 ∈ Y( x n ) \ {yn },(3.30)

in which case any vector of the form cw with c > 0 is said to separate D. We say that D is separable
if the set {γ > 0 | D is γ-separable} is non-empty, in which case its supremum is called the marginof separation of D.
20 The original version of Pegasos includes a projection onto the L -ball, but convergence is not affected by2omitting that projection, as established by Shalev-Shwartz et al. (2010).

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

42

Algorithm 2 Structured Perceptron
1: input: data D, number of rounds T
2: initialize w1 : = 0
3: for t = 1 to T do
4:choose n = n(t) randomly
5:take training pair ( x n , yn ) and predict using the current model wt :yb := arg max wt · f ( x n , y0 )y 0 ∈Y( x n )update the model:

6:

wt+1 ← wt + f ( x n , yn ) − f ( x n , yb)
7: end forT

8: output: the last weight vector w
b ← w T +1 , or the average w
b ← T1 ∑t=1 wt+1

Proposition 3.1 (Rosenblatt 1958; Collins 2002a) Let D be γ-separable, and let R be such thatk f ( x n , yn ) − f ( x n , y0 )k ≤ R for any n ∈ {1, . . . , N } and y0 ∈ Y( x n ). Then the perceptron algorithm
converges to a separating weight vector, after committing at most b R2 /γ2 c mistakes.

We can regard the structured perceptron as a subgradient method for solving a learning
problem of the form in Eq. 3.21 (albeit without regularization), with L := LSP defined as:
w · f ( x, y0 )y 0 ∈Y( x )!

LSP (w; x, y) := −w · f ( x, y) + max=

max

y 0 ∈Y( x )∑ w · δ f r (x, yr , yr0 )

(3.31)r ∈R( x )where δ f r ( x, yr , yr0 ) is defined similarly as in Eq. 3.16. We call this the structured perceptron
loss; note the resemblance with the structured hinge loss (Eq. 3.20), the difference being thelack of the cost term. To guarantee convergence, though, we would need to add a suitablestepsize for the updates in Algorithm 2.

3.5.4

Online Passive-Aggressive Algorithms

Online passive-aggressive algorithms (Crammer and Singer, 2003; Crammer et al., 2006) are arefinement of the perceptron algorithm, outperforming the latter in a number of tasks. They
are related with the previously proposed Margin Infused Relaxed Algorithm (MIRA), in
particular they correspond to what has been called 1-best MIRA in McDonald et al. (2005a).
Here, we talk solely about the so-called max-loss variant of PA-I (in the terminology of
Crammer et al. 2006), which very much resembles the stochastic subgradient algorithm for
SVMs, but with an automatic mechanism for setting the stepsize.
At each round t, the passive aggressive algorithm updates the weight vector by setting it

3.5. ONLINE AND STOCHASTIC LEARNING ALGORITHMS

43

Algorithm 3 Online Passive Aggressive (max-loss, PA-I)
1: input: data D, parameter λ, number of rounds T
2: initialize w1 : = 0
3: for t = 1 to T do
4:choose n = n(t) randomly
5:take training pair ( x n , yn ) and compute a cost-augmented prediction using the current
model wt :yb := arg max wt · f ( x n , y0 ) + ρ(y0 , yn )y 0 ∈Y( x n )

6:
7:
8:
9:

if yb = yn thenkeep the model unchanged, wt+1 ← wt
else
compute loss: `t = wt · f ( x nn
, yb) − wt · f ( x n , yn ) + oρ(yb, yn )t

10:compute stepsize: ηt = min λ−1 , k f (xn ,yn −` f (xn ,by)k2

update the model: wt+1 ← wt + ηt ( f ( x n , yn ) − f ( x n , yb))12:
end if
13: end for
14: output: the last weight vector w
b ← w T +1 , or the average w
b ← T1 ∑tT=1 wt+1
11:

to the solution of the following optimization problem, involving the n := n(t)th instance:
minimize λ2 kw − wt k2 + ξ
w.r.t.
w ∈ RD , ξ ≥ 0s.t.
w · f ( x n , yn ) ≥ w · f ( x n , yb) + ρ(yb, yn ) − ξ,

(3.32)

where
yb := arg max wt · f ( x n , y0 ) + ρ(y0 , yn )(3.33)

y 0 ∈Y( x )is the “cost-augmented prediction.” By inspecting Eq. 3.32 we see that the passive-aggressive
scheme attempts to achieve a tradeoff between conservativeness—by penalizing large changes
from the previous weight vector via the term λ2 kw − wt k2 —and correctness—by requiring,
through the constraints, that the new model wt+1 “separates” the true output from theprediction with a margin (although slack ξ ≥ 0 is allowed). The intuition for this large
margin separation is the same for support vector machines. Note that if the cost-augmented
prediction matches the true output (by = ym ), which means that the current model alreadyachieves a large margin on this example, then the solution of the problem Eq. 3.32 leaves theweight vector unchanged (wt+1 = wt ). Otherwise, this quadratic programming problem has
a closed form solution:
wt+1 ← wt + ηt ( f ( x m , ym ) − f ( x m , yb)),withηt = min λ−1

wt · f ( x m , yb) − wt · f ( x m , ym ) + ρ(yb, ym ),k f ( x m , ym ) − f ( x m , yb)k2(3.34)
.

(3.35)The resulting algorithm is depicted as Algorithm 3. For other variants of the passiveaggressive scheme, see Crammer et al. (2006).

44

CHAPTER 3. STRUCTURED PREDICTION AND LEARNING ALGORITHMS

Passive-aggressive algorithms can be interpreted as dual coordinate ascent algorithms
for learning support vector machines in an online setting (Shalev-Shwartz and Singer, 2006;
Kakade and Shalev-Shwartz, 2008). We will elaborate on this issue in Chapter 8, where we
propose new variants of coordinate ascent algorithms to tackle other loss functions besides
the structured hinge loss.

Chapter 4

Graphical Models and Inference
Algorithms
In this chapter, we provide background on probabilistic graphical models, describing their rolein modeling inference problems within the scope of structured prediction.
Graphical models enable compact representations of probability distributions and arewidely used in computer vision, natural language processing, and computational biology(Pearl, 1988; Lauritzen, 1996; Koller and Friedman, 2009). Two kinds of models have been
proposed in the literature: directed and undirected. Directed models express causal relationships, being called Bayesian networks. Their undirected counterparts—Markov networks—
model dependencies among variables, rather than causality. There are ways of converting
Bayes networks into Markov networks and vice-versa. In this thesis, undirected graphical models are our formalism of choice; in particular, we use intermediate representations
called factor graphs (Tanner, 1981; Kschischang et al., 2001), which are more informative thanMarkov networks since they explicitly represent the factors of a distribution.
This chapter summarizes prior work in graphical models, with a focus on models and algorithms for approximate inference. We restrict discussion to unconstrained models. Later, in
Chapters 5 and 6, we address the issue of hard constraints and present original contributions
for constrained structured prediction.

4.1 Undirected Graphical Models
In this section, we describe Markov networks and factor graphs, two undirected graphicalmodel formalisms. Before doing so, we briefly review some basic terminology of graphtheory that will be used throughout.
An undirected graph is a tuple G := (V, E), where V is a set of nodes, or vertices, andE ⊆ (V2 ) is a set of edges.1 For convenience, we write uv to denote an edge {u, v}. Two nodesu and v are said to be adjacent or neighbors if uv ∈ E. We denote by N(u) the neighborhoodof a node u, N(u) := {v ∈ V | uv ∈ E}. The cardinality of this set is called the degree of u,
deg(u) := |N(u)|. A clique in G is a subset of nodes α ⊆ V which are pairwise adjacent, i.e.,
for any u, v ∈ α with u 6= v we have uv ∈ E. By definition, any singleton v is a clique. A
clique is maximal if it is not contained in any other clique. A path in G is a sequence of nodes
1 (V) denotes the set of all unordered pairs of distinct vertices.
2

45

46

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS(v0 , . . . , vn ) such that vi−1 vi ∈ E for every i ∈ {1, . . . , n} and v1 , . . . , vn are distinct. Such apath is said to connect v0 and vn . If v0 = vn the path is called a cycle. A graph is connected
if any two nodes are connected by a path. A tree is a connected graph that does not containany cycles, and a forest is a graph whose connected components are trees.

4.1.1

Markov Random Fields

A Markov network is an undirected graph H := (V, E) whose nodes index a structured
random variable Y := (Yi )i∈V and whose cliques correspond to the factors of the distribution
P(Y ). Formally, we have
Pr{Y = y} ∝ ∏ ψα (yα ),(4.1)
α ∈C(H )where C(H ) is a set of cliques of H —not necessarily maximal—and ψα : Yα → R+ are nonnegative potential functions. A structured random variable Y which factors this way is calleda Markov random field (MRF). In Eq. 4.1 and throughout, we use Greek subscripts (α, β, γ, . . .)to denote a subset of indices. For example, Y α := (Yi )i∈α , with α ⊆ V, denotes the tuple ofvariables that are indexed by the elements of α. As usual, uppercase letters denote randomvariables and lowercase letters denote values of those variables. We consider only discrete
fields and assume that each Yi takes values in a finite set Yi .2 Each Y α takes values on the
Cartesian product set Yα := ∏i∈α Yi , and Y takes values on the full product set Y := ∏i∈V Yi .
The reason why Markov networks are appealing representations is that many conditional
independence properties can be immediately read from the graph H . For example, every
variable is conditionally independent from of all other variables given its neighbours (thelocal Markov property); and two subsets of variables are conditionally independent given aseparating subset (the global Markov property).3 We refer to Koller and Friedman (2009) for athorough exposition of these and other properties of Markov networks.
Pairwise Markov networks. A pairwise MRF is one whose potentials only express up to
pairwise interactions, i.e., in which the cliques in C(H ) are composed of nodes and edges.
In this case, Eq. 4.1 becomesP(y) ∝ ∏ ψi (yi ) ∏ ψij (yi , y j ).
i ∈V

(4.2)

ij∈E

Many models can be represented as pairwise Markov networks, most noticeably sequencemodels for the tasks described in Section 2.1, grid models for image segmentation, or graphs
for relational learning. Some of these are illustrated in Figure 4.1. In fact, any Markov
network can be converted to an equivalent one which is pairwise; see Wainwright and Jordan (2008,
Appendix E) for a construction. While of great conceptual interest—as it allows to transfer
2 Many formalisms discussed in this section extend naturally to discrete or continuous domains where Y areiinfinite sets; however these are out of the scope of this thesis.
3 The converse statement—that a distribution satisfying these Markov properties must factorize according to
Eq. 4.1—does not hold in general unless one makes additional assumptions, e.g., that P(Y ) has full support. This
important result is known as the Hammersley-Clifford theorem (Hammersley and Clifford, 1971; Besag, 1974).
In this thesis we generally work with distributions that do not have full support (due to hard constraints), hence
the theorem will not apply. This is not a reason for concern, however, as this converse statement is not neededin any place.

4.1. UNDIRECTED GRAPHICAL MODELS

47

Figure 4.1: Top: several pairwise Markov random fields (a sequence, a grid, and an arbitrary
graph). Bottom: factor graph representations for the same models.

properties of pairwise Markov networks to general networks—its practical consequences arelimited by the fact that the construction may lead to a blow-up in the number of states.

4.1.2

Factor Graphs

An alternative representation of probability distributions is through factor graphs (Kschischang et al., 2001), which explicitly represent the factors of the distribution (i.e., the actual
cliques that are used in Eq. 4.1). A factor graph is a bipartite graph G := (V ∪ F, E), where
V and F are disjoint sets of nodes (respectively called variable nodes and factor nodes) andE ⊆ V × F is a set of edges, each connecting a variable node to a factor node. We say that
P(Y ) factors according to G if it has the form
P(y) ∝ ∏ ψi (yi ) ∏ ψα (yα ),
i ∈V

(4.3)

α ∈F

where each ψi : Yi → R+ and ψα : Yα → R+ are non-negative potential functions, respectively
defined on the variable nodes and on the factor nodes.4
We use Latin letters i, j, . . . to denote variable nodes and Greek letters α, β, . . . for factor
nodes. For notational convenience, we often represent the latter by the set of indices ofvariable nodes that are linked to it (without loss of generality, we assume that there are no
distinct factor nodes with the same neighborhood set). The bottom part of Figure 4.1 shows
examples of factor graphs.
Given a factor graph G along with all the potential functions—which give a full specification of the distribution P(Y )—we want to solve inference problems in such a graph. The
three tasks that will be our main concern throughout are:
1. MAP inference. Computing an assignment with maximal probability (called a maxi4 Some authors omit the variable potential functions, writing the factorization above as P ( y ) ∝∏α∈F ψα (yα ).
That is not less general, as one can always define unary factors αi such that N(αi ) = {i } whose potentials play thesame role as our variable potentials (ψαi ≡ ψi ). We choose to explicitly use variable potentials for two reasons:(i) in most cases unary potentials are used anyways, hence we just avoid having to define unary factors all thetime; (ii) in the sequel we will see that there is a certain “symmetry” between potentials and posterior marginals
that is preserved if we consider variable potentials.

48

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS
mum a posteriori or MAP assignment):yb := arg max P(y).

(4.4)

y

2. Partition function evaluation. Computing the normalizing factor of Eq. 4.3, whichcorresponds to evaluating the partition function Z:
Z (ψ) := ∑y ∈Y

∏ ψi (yi ) ∏ ψα (yα )
i ∈V

!
(4.5)

α ∈F

3. Marginal inference. Computing posterior marginals for “local” variable assignments(by “local” we mean “at the variable and factor nodes of G ”):µi (yi ) := P(Yi = yi )andµ α ( y α ) : = P (Y α = y α ) .

(4.6)

4.2 Message-Passing Algorithms
If G has no cycles (i.e., if it is a tree or a forest), the three inference problems stated above
can all be solved with dynamic programming. The MAP assignment can be obtained via
the max-product algorithm, a generalization of the Viterbi algorithm for trees (Viterbi, 1967;
Forney, 1973); the normalization factor and the marginals can be computed through thesum-product algorithm, which generalizes the forward-backward algorithm (Baum and Petrie,1966).5 In both cases, the runtime is linear in |E| and (in general) exponential in the largest
factor degree.6 These algorithms work by passing information through the graph, and areequivalent to the belief propagation algorithm of Pearl (1988)—which we next describe—for aparticular scheduling of the messages.
The belief propagation (BP) algorithm iteratively passes messages between variables andfactors reflecting their local “beliefs.” In sum-product BP, the messages take the form:
Variable to factor:∏ M β →i ( yi )(4.7)

∑ ψα (yα ) ∏ Mj→α (y j ).

(4.8)

Mi→α (yi ) ∝ ψi (yi )β ∈N ( i )β6=αFactor to variable:

Mα → i ( y i ) ∝y α ∼ yij ∈N ( α )j 6 =iIn Eq. 4.8, we employ the standard ∼ notation, where a summation/maximization indexedby yα ∼ yi means that it is over all yα with the i-th component held fixed and set to yi . The
order by which the messages are applied is called its schedule. In the sequel, we assume
a flooding schedule, according to which one iterates between computing all messages of thekind in Eq. 4.7, and then all messages of the kind in Eq. 4.8.
5 Note that query-oriented algorithms such as the variable elimination algorithm (Zhang and Poole, 1994;

Dechter, 1999) are not suitable for computing all marginals, since they are tailored to answer a single “query” at
the time.
6 In some sense, the two algorithms (max-product and sum-product) are isomorphic and are both special casesof the generalized sum-product algorithm for semirings (Shafer and Shenoy, 1990; Dawid, 1992; Goodman, 1999;
Aji and McEliece, 2000; Mohri, 2002).

4.2. MESSAGE-PASSING ALGORITHMS

49

Upon convergence, variable and factor beliefs are computed as:bi (yi ) ∝ ψi (yi )Variable beliefs:∏ Mα → i ( y i ) ,(4.9)∏ Mi → α ( y i ) .

(4.10)α ∈N ( i )bα (yα ) ∝ ψα (yα )Factor beliefs:

i ∈N ( α )If the BP algorithm attains a fixed point, i.e., if all messages are left unchanged after completing a full round of the iterations (4.7–4.8), then the resulting beliefs must satisfy thefollowing calibration equations, for every i ∈ V and yi ∈ Yi :7
bi ( y i ) =∑ bα ( y α ) .

(4.11)y α ∼ yiIf G has no cycles, then BP always converges and these beliefs are precisely the posterior
marginals in Eq. 4.6 (i.e., we have µi (yi ) = bi (yi ) and µα (yα ) = bα (yα )). Given the marginals,
we have the following reparametrization of the distribution P(Y ):P(y) =∏ α ∈F µ α ( y α ).
∏i∈V µi (yi )deg(i)−1(4.12)A distribution in the form (4.12) is said to be calibrated. Given a calibrated distribution, one
can evaluate probabilities, and by choosing a particular assignment, it is straighforward toevaluate the partition function (Eq. 4.3).
In max-product BP, the summations in Eqs. 4.8 and 4.11 are replaced by maximizations,!

Mα → i ( y i )∝max

y α ∼ yiψα (yα ) ∏ M j→α (y j ) ,(4.13)

j 6 =ibi (yi ) = max bα (yα );(4.14)y α ∼ yiand upon convergence, one can compute the exact MAP variable-wise from the beliefs(Eqs. 4.9–4.10) through:ybi ∝ arg max bi (yi ),y i ∈ Yi∀i ∈ V.

(4.15)As mentioned above, the runtime of the belief propagation algorithm is exponential in
the largest factor degree; this follows from observing that the computation of the factor-tovariable messages (Eqs. 4.8 and 4.13) involves (in general) enumerating all |Yα | configurations. Linearity in |E| comes from the fact that convergence is achieved after at most |E|
message passing iterations.

4.2.1

Graphs with Cycles and Approximate Inference

The correctness of the BP algorithm falls apart when G has cycles, which leads to nuisanceeffects due to some messages being double counted during the updates. Exact inference is
7 The proof is immediate, by observing that the messages and the beliefs are related through M

Mα→i (yi )−1 bi (yi ) and Mα→i (yi ) ∝ Mi→α (yi )−1 ∑yα ∼yi bα (yα ).

i →α ( yi ) ∝CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

50

still possible via the junction tree algorithm, which constructs a new graph from G (calleda junction tree) by collapsing some of the nodes into supernodes, and then runs the belief
propagation algorithm on the junction tree (see Lauritzen 1996; Koller and Friedman 2009 fordetails). This however leads to a runtime which grows exponentially with a parameter called
the treewidth of G ,8 rendering the procedure intractable for graphs with a large treewidth.
This should come as no surprise: it can be shown that exact MAP inference in a general graph(even one with only pairwise interactions) is NP-complete, and computing the normalization
factor is #P-hard. For details and pointers to the literature, see, e.g., Sontag (2010, Chapter 2).
A variety of approximate inference methods have been proposed: some based on sampling (Geman and Geman, 1984; Marroquin et al., 1987; Gelfand and Smith, 1990; Casella
and Robert, 1999; Finkel et al., 2005), others on approximate search (Zhang and Clark, 2008;
Daumé et al., 2009), yet others on variational inference (see Wainwright and Jordan 2008 andreferences therein). In this thesis, we focus on the last kind. One instance is the loopy belief
propagation algorithm (loopy BP), which is exactly what we have described, but applied to a
general graph G with cycles. In other words, loopy BP “pretends” that G is a tree by ignoring
the global effects caused by the cycles. Of course, as would be expected, it is not guaranteed
to compute the exact quantities, or even to converge.9 Nevertheless, it often works well inpractice for some graph topologies.

4.3 The Geometry of Graphical Models
Inference in graphical models is better understood by characterizing their geometry. We
next highlight a few crucial geometric aspects, some of which inspire approximate inference
algorithms in loopy graphs, as we shall see. The two main ingredients are:
• The marginal polytope, which characterizes the structure of the graphical model;• A duality relationship of exponential families (Legendre-Fenchel duality) which allows
expressing marginal inference as a variational problem.
In this section, we briefly review the main concepts. Additional details can be found in themonograph by Wainwright and Jordan (2008).

4.3.1

Graphical Models as Exponential Families

An exponential family10 E := { Pθ (.) | θ ∈ RR } is a family of distributions, each of the following form:
1Pθ (y) =exp(θ · φ(y)) Q(y),(4.16)Z (θ)
where θ ∈ RR is a parameter vector, φ(y) ∈ RR is a vector of sufficient statistics (or features),
Z : RR → R+ is the partition function, and Q is a base measure on Y. In this section, we
8 The treewidth is the size of the largest clique of the triangulated Markov network representation of G , minus

one. See Lauritzen (1996) or Koller and Friedman (2009) for details.
9 Although it is provably correct or at least convergent for some special classes of graphs which are not trees:
see Weiss and Freeman (2001b,a); Bayati et al. (2005); Huang and Jebara (2007).
10 Also called a family of Gibbs distributions in statistical physics, and a family of log-linear distributions in natural
language processing and machine learning.

4.3. THE GEOMETRY OF GRAPHICAL MODELS

51

assume that Q is a constant-valued function (Q ≡ 1 without loss of generality), whence it
can be dropped from Eq. 4.16. The partition function takes the form:
Z (θ) := ∑ exp(θ · φ(y)).

(4.17)y ∈YExponential families are extremely general; their asymptotic and geometric properties havebeen widely studied in statistics (Darmois, 1935; Pitman, 1936; Koopman, 1936) and information geometry (Murray and Rice, 1993; Amari and Nagaoka, 2001).
Factor graphs with strictly positive potentials can be regarded as exponential families.
Indeed, we can let the parameters θ be the log-potentials,
θα (yα ) := log ψα (yα ),θi (yi ) := log ψi (yi ),(4.18)and define the sufficient statistics φ(y) := χ(y) as the indicator vector of local configurations(on variable and factor nodes) induced by the output y ∈ Y:
χi,yi0 (y) := [[yi = yi0 ]],χα,y0α (y) := [[yα = y0α ]].

(4.19)The dimension of the vectors θ and φ(y) is R := ∑i∈V |Yi | + ∑α∈F |Yα |. We write Eq. 4.16 as
1Pθ (y) =exp
Z (θ)∑ θi ( yi ) + ∑ θ α ( y α )!
,(4.20)α ∈Fi ∈V

and Eq. 4.17 as
Z (θ) := ∑ expy ∈Y∑ θi ( yi ) + ∑ θ α ( y α )

i ∈V

!
.

(4.21)α ∈FThis is equivalent to Eq. 4.3 when all potentials are strictly positive. This parametrization ofa factor graph, through θ, is called the canonical overcomplete parametrization (Wainwright andJordan, 2008).11

Example 4.1 (Pairwise MRFs: Boltzmann, Ising, and Potts models.) The canonical overcomplete parametrization of a pairwise MRF isPθ (y) ∝ exp∑ θi (yi ) + ∑ θij (yi , y j )

i ∈V

!
.

(4.22)ij∈E

A binary pairwise MRF is one in which all variables are binary, Yi := {0, 1}. Such models are also
called Ising models.12 In that case, there are 2|V| + 4|E| parameters in Eq. 4.22. The same family
11 The name “overcomplete” comes from the fact that the map θ 7 → P is not injective: in fact, if we define θ0θsuch that θi0 (yi ) = θi (yi ) + ci and θα0 (yα ) = θα (yα ) + cα , where ci and cα are arbitrary real constants, we have that0

θ and θ parameterize the same distribution.
12 Named after Ernst Ising (1900–1998), who first proposed these models in statistical physics to model interaction between particles (Ising, 1925). These models are also called Boltzmann machine models in the artificial neuralnetworks literature (Ackley et al., 1985).

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

52

can be parameterized with only |V| + |E| parameters through:∑ si yi + ∑ sij yi y j

Ps (y) ∝ exp

i ∈V

!
.

(4.23)

ij∈E

This reparametrizarion does not lose expressive power. Given s, we can define θ as
θ i (1) : = s i ,θi (0) := 0,

θij (1, 1) := sij ,θij (0, 0) = θij (0, 1) = θij (1, 0) := 0,(4.24)and vice-versa:
s i = θ i (1) − θ i (0) +∑ (θij (1, 0) − θij (0, 0)),sij = θij (1, 1) − θij (0, 1) − θij (1, 0) + θij (0, 0).

j ∈ N (i )(4.25)
Of particular interest are ferromagnetic Ising models, in which all edge couplings are “attractive”(i.e., sij ≥ 0 for all ij ∈ E).13 In such models, neighboring nodes “prefer” to be in the same state.
Ferromagnetic Ising models permit tractable MAP inference through graph cut algorithms (Greiget al., 1989); in general Ising models, MAP inference is NP-hard. A Potts model is a generalizationof an Ising model for K > 2 states, i.e., Yi := {1, . . . , K } for every i ∈ V. Potts models are extensively
used in computer vision (for image segmentation and stereo vision problems) and relational networkmodeling. It is common to define attractive log-potentials as θij (yi , y j ) := αij × [[yi = y j ]], whereαij ≥ 0 is a constant, which corresponds to the assumption that neighbors prefer to be in the samestate. Markov networks with these property are called associative (Taskar et al., 2004a).
Example 4.2 (Conditional random fields.) We have described CRFs in Section 3.4.2, defined via
a set of places P, local feature vectors f p ( x, y p ) ∈ RD and a weight vector w ∈ RD :1
Pw (y| x ) :=exp
Z (w, x )∑ w · f p (x, y p )!
.

(4.26)p ∈P

We can regard the “places” as the variable and factor nodes in a factor graph G := (V, F ), in whichcase we may rewrite Eq. 4.26 as
1
Pw (y| x ) :=exp
Z (w, x )∑ w · f i (x, yi ) + ∑ w · f α (x, yα )!
.

(4.27)α ∈F

i ∈VThis is a discriminative linear model in which the joint feature map decomposes asf ( x, y) := ∑ f i ( x, yi ) + ∑ f α ( x, yα ).
i ∈V(4.28)α ∈F

Note that Eq. 4.28 allows features to be shared across variables and factors. For each x ∈ X, suchmodels form an exponential family E ( x ) parameterized by w ∈ RD . This family is a subset of thecanonically parameterized family in Eq. 4.20: there is an affine transformation that transforms thefeature weights w ∈ RD into canonical overcomplete parameters θ ∈ RR :θi (yi ) = w · f i ( x, yi ),θα (yα ) = w · f α ( x, yα ).

(4.29)

13 Equivalently, the potentials are “submodular,” as commonly refered in the computer vision literature.

4.3. THE GEOMETRY OF GRAPHICAL MODELS

53

We can write this compactly as θ = F( x )> w, where F( x ) is a D-by-R feature matrix, whose columns
are the local feature vectors f i ( x, yi ) and f α ( x, yα ).

4.3.2

Mean Parametrization and Marginal Polytope

Consider a particular distribution Pθ ∈ E , and its vector of expected sufficient statistics Eθ [φ(Y )].14
A fundamental property of exponential families is that the gradient of the log-partition function equals that quantity:∇θ log Z (θ) =∑y∈Y exp(θ · φ(y))φ(y)∇θ Z (θ)== Eθ [φ(Y )].
Z (θ)
Z (θ)

(4.30)

Recall that with a canonical overcomplete parametrization, the sufficient statistics are simply
indicator functions of variable and factor configurations, i.e., we have φ(y) := χ(y) (cf.
Eq. 4.19); hence the expected sufficient statistics are precisely the posterior marginals on local
configurations (cf. 4.6), i.e., we haveµi (yi ) = Eθ [χi,yi (Y )],µα (yα ) = Eθ [χα,yα (Y )].

(4.31)

We stack these marginals together into a marginal vector µ := Eθ [χ(Y )] ∈ RR . This yields analternative parametrization of the exponential family E , called the mean parameterization. To
make this formal, we need to specify the range of these parameters, i.e., the set of marginals
which are realizable by some arbitrary distribution P(Y ):MARG(G ) : =nµ ∈ RR | ∃ P(Y ) s.t. µ = EY ∼ P(Y ) [χ(Y )]= conv{χ(y) | y ∈ Y}.

o(4.32)The second equality is by definition of convex hull (cf. Appendix B), since a well-defined
distribution P(Y ) is one which satisfies ∑y P(y) = 1 and P(y) ≥ 0 for every y ∈ Y. Being
the convex hull of a finite set, MARG(G ) is a polytope—it is called the marginal polytope of
G (Wainwright and Jordan, 2008). The marginal polytope has two important properties:
1. Each vertex of MARG(G ) corresponds to an output y ∈ Y;
2. Each point in MARG(G ) corresponds to a vector of marginal probabilities that is realizable by some distribution.
This is represented schematically in Figure 4.2. By definition of marginal vector, we havethat the map θ 7→ µ is well defined (i.e., for every θ ∈ RR there is one and only oneµ(θ) ∈ MARG(G )); from Eq. 4.30, we have that it is precisely the gradient map ∇ log Z :R M → MARG(G ). Note however that this map is not injective, since the θ-parametrization
is overcomplete. Yet, there is a bijection between E and the relative interior15 of the marginal
polytope, as the next proposition asserts.16
14 We denote Eθ ≡ EY ∼ Pθ (Y ) to make explicit the dependency on the parameters θ.
15 See Appendix B for the definition of relative interior.
16 Theorem 3.3 in Wainwright and Jordan (2008) is stated for a minimal parametrization; however they discuss

its extension to the overcomplete case in their Appendix B.1.

54

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

Figure 4.2: Representation of the marginal polytope for a simple factor graph connecting two
nodes and one factor. Each point in the polytope is a realizable marginal vector, and eachvertex correspond to a possible assignment; we illustrate, on the right, one of those vertices,
corresponding to the configuration Y1 = 3 and Y2 = 1. The gradient map ∇ log Z maps the
canonical overcomplete parameters to the mean parameters.

Proposition 4.1 (Wainwright and Jordan 2008, Th. 3.3) Let µ ∈ relint(MARG(G )). Then there
exists some θ ∈ RR such that µ = Eθ [χ(Y )]. Furthermore, all such θ are equivalent in the sensethat they all parameterize the same distribution Pθ (Y ).

4.3.3

MAP Inference as a Linear Program

Given a distribution Pθ ∈ E , MAP inference (Eq. 4.4) is a combinatorial optimization problem
which can be written as:
maximize ∑i∈V θi (yi ) + ∑α∈F θα (yα )w.r.t.
y ∈ Y.

(4.33)

Let θ · µ = ∑i∈V ∑yi ∈Yi θi (yi )µi (yi ) + ∑α∈F ∑yα ∈Yα θα (yα )µα (yα ). Given that there is a oneto-one correspondence between the vertices of MARG(G ) and the elements in Y, we cantransform Eq. 4.33 into the following linear program:
MAP: maximize θ · µ
w.r.t.
µ ∈ MARG(G ).

(4.34)The equivalence between the two formulations is due to the fact that any linear program
whose constraint set is non-empty and bounded—which is the case of MARG(G )—has a solution which is attained at a vertex. Unfortunately, for a general graph G , reformulating thenon-convex combinatorial problem in Eq. 4.33 as the continuous and convex one in Eq. 4.34
does not make it easier. Although a fundamental result in convex analysis, the MinkowskyWeyl theorem (Rockafellar, 1970), guarantees that the marginal polytope MARG(G ) has arepresentation in terms of a finite number of linear inequalities, it is widely believed that
this number grows superpolynomially with the size of the graph G .17
17 If this statement turns out to be false, it will rule out important conjectures in complexity theory, such as
P 6= NP. To see this, note that the well-known problem of finding a maximum cut in a graph is equivalent to
MAP inference in a Ising model, and hence can be reduced to the linear program in Eq. 4.34. If there was a4.3. THE GEOMETRY OF GRAPHICAL MODELS

55

Yet, the linear programming formulation in Eq. 4.34 is very useful, since it sheds light on
the MAP inference problem, and opens up ways of devising approximate inference algorithms
by considering approximations of MARG(G ). We will see specific examples later on.

4.3.4

Marginal Inference and Conjugate Duality

We now turn to the problem of computing the marginals and evaluating the partition function. The relationship between the two had appeared in Eq. 4.30, which expressed themarginals as the gradient of the log-partition function. This is part of a bigger picture thatinvolves an important form of duality (called Legendre-Fenchel duality). To display the whole
picture, we need to introduce another ingredient, the Shannon entropy of a distribution:
H ( P(Y )) := EY [− log P(Y )].

(4.35)

From Proposition 4.1, we have a bijection P(Y ) 7→ µ, and therefore we may express the entropy in terms of the µ-parameters, H : MARG(G ) → R, where we write (abusing notation)
H (µ) to denote the entropy of the distribution P(Y ) which is parameterized by µ.
Proposition 4.2 (Wainwright and Jordan 2008, Th. 3.4) The log-partition function, log Z : RR →R, and the negative entropy, − H : MARG(G ) → R, are conjugate dual of each other. As a consequence, the following variational representation for the log-partition function holds:
log Z (θ) =maxµ∈MARG(G )θ · µ + H ( µ ).

(4.36)

Furthermore, the solution of Eq. 4.36 is attained at µ(θ), i.e., the marginal vector corresponding to
the distribution Pθ. Conversely, the entropy function also has a variational representation:− H (µ) = max θ · µ − log Z (θ),θ∈RR

(4.37)

with a solution being attained for any θ such that µ = µ(θ). Therefore:
H ( Pθ ) = −θ · µ(θ) + log Z (θ),(4.38)

for any θ that parameterizes Pθ.
The duality expressed in Proposition 4.2 is intimately related with the well-known equivalence between conditional log-likelihood maximization in logistic regression models andentropy maximization subject to first moment constraints. Note also that by applying Danskin’s theorem to Eq. 4.36, we recover the fact that ∇ log Z (θ) = µ(θ).
Like in the case of MAP inference, Eq. 4.36 allows us to formulate marginal inference asa convex optimization problem over the marginal polytope:
Marginals: maximize θ · µ + H (µ)
w.r.t.
µ ∈ MARG(G ).

(4.39)

polynomial number of constraints in Eq. 4.34, then the linear program (and hence max-cut) would be solvablein polynomial time.

56

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

Note that the only difference with respect to the MAP inference case (Eq. 4.34) is the inclusion of the entropic term H (µ) in the objective, which breaks the linearity. Albeit non-linear,
this term is convex, hence Eq. 4.34 is still a convex program. But the entropic term introduces another complication, which adds to the already known difficulty in characterizing
the marginal polytope MARG(G ): when expressed in terms of µ, the entropy lacks a closed
form (in general) and is difficult to characterize.

4.4 Local Polytope Approximation
We mentioned in Section 4.3.3 that, for a general graph G with cycles, the marginal polytope
MARG(G ) is difficult to characterize as a set of linear inequalities. The reason is that thecycles induce global consistency constraints which are hard to specify. There are, however,
weaker constraints that all realizable marginals must satisfy to ensure local consistency:
1. Non-negativity. All local marginals µi (yi ) and µα (yα ) must be non-negative.
2. Normalization. We must have ∑yi ∈Yi µi (yi ) = 1 for every i ∈ V, and ∑yα ∈Yα µα (yα ) = 1
for every α ∈ F, to ensure that at a local level all marginals normalize properly.
3. Marginalization. A variable participating in a factor must have a marginal which isconsistent with the factor marginals, i.e., we must have µi (yi ) = ∑yi ∼yα µα (yα ) for everyi ∈ V, yi ∈ Yi and α ∈ N(i ).
Putting everything together, we form the so-called local polytope:∀i ∈ V∑yi ∈Yi µi (yi ) = 1,R
LOCAL(G ) = µ ∈ R.
µi (yi ) = ∑yα ∼yi µα (yα ), ∀i ∈ V, yi ∈ Yi , α ∈ N(i )µα (yα ) ≥ 0,∀α ∈ F, yα ∈ Yα

(4.40)The number of constraints that define LOCAL(G ) is linear in R, rather than superpolynomial.The elements of LOCAL(G ) are called pseudo-marginals. The “local” character of
LOCAL(G ) may be emphasized by noting that it can be equivalently expressed as the intersection of smaller marginal polytopes, “lifted”18 to the ambient space RR :n
LOCAL(G ) := µ ∈ RRµ|α ∈ MARG(G |α )ofor every α ∈ F ,(4.41)

where we denote by G |α the subgraph formed by a single factor α and their neighbors, andby µ|α the restriction of µ to the entries µα (.) and µi (.) for every i ∈ N(α). Since any true
marginal vector must satisfy the constraints above, LOCAL(G ) is an outer approximation:
MARG(G ) ⊆ LOCAL(G ).
The next proposition states that the approximation is tight for tree-structured graphs.
Proposition 4.3 If G does not have cycles, then MARG(G ) = LOCAL(G ).
18 See Appendix B for a definition of lifting.

(4.42)

4.5. BETHE’S VARIATIONAL PRINCIPLE AND LOOPY BELIEF PROPAGATION

57

Figure 4.3: Marginal polytope (in green) and its outer aproximation, the local polytope (inblue). Each element of the marginal polytope corresponds to a distribution P(Y ), and eachvertex corresponds to a configuration y ∈ Y, having coordinates in {0, 1}. The local polytope
may have additional fractional vertices, with coordinates in [0, 1].

Proof sketch. It suffices to show that LOCAL(G ) ⊆ MARG(G ). Given µ ∈ LOCAL(G ),define P(Y ) through Eq. 4.12; then show that marginalizing P(Y ) over variables and factors
recovers µ, i.e., that ∑y∼yi P(y) = µi (yi ) and ∑y∼yα P(y) = µα (yα ). This follows from themarginalization constraints that define LOCAL(G ), along with the fact that for any acyclic
G , no two distinct factors α and β share more than one variable (which would cause a cycle).
Unfortunately, the inclusion MARG(G ) ⊆ LOCAL(G ) is proper even for very small
graphs with cycles (see e.g. Wainwright and Jordan 2008, Example 4.1). All we have is:
Proposition 4.4 The integral points of LOCAL(G ) are the vertices of MARG(G ). Hence:
MARG(G ) = conv(LOCAL(G ) ∩ ZR ).

(4.43)

Proof. Since LOCAL(G ) is contained in the unit cube [0, 1] R , its integral points are binaryvalued. From the normalization constraints, each integral point µ ∈ LOCAL(G ) must indicate a single assignment at each factor and variable node, and the marginalization constraints force these to agree. Hence µ = χ(y) for some y ∈ Y, and therefore we have{χ(y) | y ∈ Y} = LOCAL(G ) ∩ ZR . The result follows from taking the convex hull on bothsides.
We depict in Figure 4.3 a schematic representation of the local polytope. We will see in
the next two sections that this approximation plays a crucial role in approximate inference
algorithms, both for marginal inference (Section 4.5) and for MAP inference (Section 4.6).

4.5 Bethe’s Variational Principle and Loopy Belief Propagation
We now describe how the loopy BP algorithm is related to an approximation of the variational expression of Eq. 4.39. For convenience, we will use in the sequel the following vector
notation for the local marginals,µi := (µi (yi ))yi ∈Yi ,µα := (µα (yα ))yα ∈Yα .

(4.44)As mentioned above, for a general graph G , the entropy H : MARG(G ) → R lacks a closed
form. A popular approximation is the so-called Bethe entropy (Bethe, 1935),19 which we
19 Named after the physicist Hans Bethe (1906–2005).

58

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

denote by HBethe : LOCAL(G ) → R:
HBethe (µ) := ∑ (1 − deg(i )) Hi (µi ) + ∑ Hα (µα ),(4.45)α ∈Fi ∈V

where Hi (µi ) := − ∑yi ∈Yi µi (yi ) log µi (yi ) and Hα (µα ) := − ∑yα ∈Yα µα (yα ) log µα (yα ) are local
entropies (at the variable and the factor nodes, respectively).20 The next proposition statesthat this approximation is exact for tree-structured graphs.
Proposition 4.5 If G has no cycles, then H = HBethe .
Proof. From Proposition 4.3, we have that MARG(G ) = LOCAL(G ), hence H and HBethe
have the same domain. That H (µ) equals the right hand side of Eq. 4.45 is an immediate
consequence of the reparametrization of the distribution in Eq. 4.12.
For a general graph G , Bethe’s entropy approximation is inexact and neither lower orupper bounds the true entropy. Furthermore, HBethe is not concave in general (unlike H). The
quality of the approximation strongly depends on the topology of the graph. While in many
practical problems the two functions are “close” enough, we will see some disappointing
examples in Chapter 5 where HBethe is a very poor approximation.
To summarize, we have discussed two approximations so far; both ignore global effects
caused by cycles and both are tight when G is acyclic:• The marginal polytope MARG(G ) is outer approximated by LOCAL(G );• The entropy H is approximated by the Bethe “entropy” HBethe .
Putting these two pieces together, we obtain the following variational approximation for the
log-partition function (cf. Eq. 4.36):
log Z (θ) ≈maxµ∈LOCAL(G )θ · µ + HBethe (µ).

(4.47)The maximizer of Eq. 4.47 (call it µ̃(θ)) can be regarded as a surrogate for the true marginals,
and hopefully µ̃(θ) ≈ µ(θ). The following important result, established by Yedidia et al.
(2001), relates the loopy BP algorithm with the Bethe variational problem.
Theorem 4.6 (Yedidia et al. (2001)) Any stationary point resulting from the updates in the sumproduct loopy BP algorithm corresponds to a local optimum of the problem in Eq. 4.47.
We point to Yedidia et al. (2001) for a formal proof. Essentially, the messages in loopy BP
correspond to Lagrange multipliers in a dual formulation of the problem in Eq. 4.47, and thevector of beliefs locally optimizes Eq. 4.47. However, the sum-product loopy BP algorithm
may not converge, and the local optimum may not be a global one—this is because the Bethe
variational problem is in general nonconvex, due to the nonconcavity of HBethe .
20 To better understand this approximation, observe that for a pairwise MRF, H
Bethe can be rewritten as:

HBethe (µ) = ∑ Hi (µi ) − ∑ Iij (µij ),i ∈V(4.46)

ij∈Fµwhere Iij (µij ) := ∑yi ∈Yi ∑y j ∈Yj µij (yi , y j ) log µ (y )µij (y ) is the mutual information between each pair of variables.
i

i

j

j

In words, the Bethe entropy approximation equals the sum of the node entropies (as if the variables were all
independent), discounted by the pairwise mutual informations.

4.6. LINEAR PROGRAMMING RELAXATION FOR MAP INFERENCE

59

Generalized message-passing algorithms. Finding good entropy approximations and mapping them to message-passing algorithms is a very active area of research. Yedidia et al.
(2001, 2005) proposed a generalized BP algorithm that works with the more refined Kikuchi
entropy approximation, at the expense of additional computational burden. There is alsomuch work building approximations of the form
H (µ) ≈ ∑ ci Hi (µi ) + ∑ cα Hα (µα ),i ∈V(4.48)α ∈F

for several choices of scalars ci an cα , called entropy counting numbers. Particularly appealingis the case where those scalars are chosen so that the approximate entropy is concave, making
the variational approximation convex. An example (for pairwise Markov networks) is the
sum-product tree-reweighted BP algorithm (Wainwright et al., 2005b), guaranteed to produce
an upper bound of H, which in turn yields an upper bound of the log-partition function.
Other approximations in the form (4.48) have been proposed by Wiegerinck and Heskes(2003); Weiss et al. (2007); Hazan and Shashua (2010).

4.6 Linear Programming Relaxation for MAP Inference
Approximate MAP inference is in some sense easier than marginal inference, since we do notneed to worry about approximating the entropy; i.e., only the marginal polytope matters. We
describe approximate algorithms related with the local polytope approximation described in
Section 4.4; tighter polyhedral and semi-definite approximations exist, which result in more
expensive algorithms (Sontag, 2010; Wainwright and Jordan, 2008).
From Proposition 4.4, we have that the MAP inference problem (Eq. 4.34) is equivalent
to the following integer linear program (ILP):
maximize θ · µ
w.r.t.
µ ∈ LOCAL(G ),µ integer.

(4.49)

Ignoring the integer constraint of Eq. 4.49 yields the so-called Schlesinger’s linear relaxation(Schlesinger, 1976), which we call the LP-MAP inference problem:
LP-MAP: maximize θ · µ
w.r.t.
µ ∈ LOCAL(G ).

(4.50)

Since it is a relaxation, the solution of LP-MAP provides an upper bound of the optimal
objective value in Eq. 4.49. While any off-the-shelf LP solver can be employed for this task,
specialized algorithms have been designed that exploit the structure of graph, achieving
superior performance on several benchmarks (Yanover et al., 2006). However, Theorem 4.6
cannot be extended to the max-product case (described in Section 4.2); i.e., even if maxproduct loopy BP attains a stationary point, that point is not necessarily a solution of the
LP-MAP problem (Wainwright and Jordan, 2008). A number of message-passing algorithms
have been proposed that slightly change the message updates, fixing this issue (Wainwrightet al., 2005a; Kolmogorov, 2006; Werner, 2007; Globerson and Jaakkola, 2008; Ravikumar
et al., 2010). Some of these algorithms give certificates that they have found the true MAP

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

60

Algorithm 4 MPLP Algorithm for LP-MAP (Globerson and Jaakkola, 2008)
1: input: factor graph G , parameters θ
2: initialize γ = 0
3: repeat4:
for each factor α ∈ F do
5:pass messages from variables in N(α) to factor α:
δi→α (yi ) := θi (yi ) +∑γ β →i ( y i ).

β∈N(i )\{α}

6:

7:
8:

pass messages from factor α to the variables in N(α):
1γα→i (yi ) := −δi→α (yi ) +max θα (yα ) + ∑ δj→α (y j ) .
deg(α) yα ∼yij ∈N ( α )end forcompute current value of the dual objective:!
d := ∑ max θi (yi ) +i ∈Vy i ∈ Yi

∑ γα → i ( y i )α ∈N ( i )+ ∑ maxα ∈F y α ∈Yαθα (yα ) −∑ γα → i ( y i )!
.

i ∈N ( α )convergence or small enough progress in the dual.
10: output: approximate MAP solution yb:
9: until

ybi = arg max θi (yi ) +y i ∈ Yi

∑ γα → i ( y i )!
.

α ∈N ( i )assignment when the relaxation is tight (e.g., the “weak tree agreement” condition in treereweighted methods). We terminate this chapter by briefly describing two classes of LPMAP inference algorithms proposed in the literature: message-passing algorithms based ondual coordinate descent, and dual decomposition algorithms based on the subgradient method.
We refer to Wainwright and Jordan (2008); Sontag et al. (2011) for thorough descriptions.

4.6.1

Block Coordinate Descent Algorithms

Many coordinate descent methods have been proposed in the literature to solve the LP-MAP
problem. These methods are characterized by different reformulations of the linear program,leading to different duals, and by the choice of blocks of coordinates they optimize over.
Werner (2007) proposed a max-sum diffusion algorithm reminiscent of a technique due to
Kovalevsky and Koval (1975); Globerson and Jaakkola (2008) proposed the MPLP algorithm
as a way of “fixing” max-product belief propagation, along with several variants; Sontag
and Jaakkola (2009) proposed a more efficient choice of blocks. Some refinements of themethods above, along with an empirical comparison, are presented in the recent monographby Sontag et al. (2011). MPLP is illustrated as Algorithm 4; in Appendix A.2, we show that
MPLP is a block coordinate descent algorithm. We will use MPLP in Chapter 6 as a baseline.
Note that, unlike the loopy BP algorithm, the messages in the MPLP algorithm have a4.6. LINEAR PROGRAMMING RELAXATION FOR MAP INFERENCE

61

sequential order, i.e., one must loop through each factor, compute the incoming and outgoingmessages involving all the variables linked to that factor, and move to the next factor. Eachstep improves the dual objective by moving along a block of coordinates. Under certain
conditions, if the relaxation is tight, one might obtain a certificate of optimality. Whenthe relaxation is not tight, it is sometimes possible to reduce the size of the problem or touse a cutting-plane method (Sontag et al., 2008) to make progress toward the true MAP.
One advantage of block coordinate descent algorithms is that they are provably convergent(unlike the loopy BP algorithm). However, they may get stuck at a corner, which is a generaldisadvantage of coordinate descent algorithms in non-smooth optimization (see Bertsekas
et al. 1999, Sect. 6.3.4 for details). We refer to Globerson and Jaakkola (2008) and Sontag
et al. (2011) for additional properties of this and other coordinate descent algorithms.

4.6.2

Dual Decomposition with the Projected Subgradient Algorithm

A different class of algorithms for LP-MAP is based on dual decomposition, a classical optimization technique (Dantzig and Wolfe, 1960; Everett III, 1963; Shor, 1985). This method has
been proposed in the context of graphical models by Komodakis et al. (2007); Johnson et al.
(2007), and has been shown quite effective in many NLP problems (Koo et al., 2010; Rushet al., 2010; Auli and Lopez, 2011; Rush and Collins, 2011; Chang and Collins, 2011). Like the
MPLP algorithm (and other message passing algorithms), the dual decomposition method
is derived via a reformulation and dualization of Eq. 4.50. The reformulation consists of:
1. Adding new variables µiα (yi ), for each factor α, which are “replicas” of the pseudomarginals µi (yi ). To avoid confusion, we rename the original variables µi (yi ) to ζ i (yi ).
2. Enforcing agreement among those variables—this is done by adding equality constraints of the form µiα (yi ) = ζ i (yi ), for every α ∈ N(i ).
For convenience, we also introduce “split” log-potentials θiα (yi ); any choice that satisfies∑ θiα (yi ) = θi (yi )(4.51)α ∈N ( i )can be used (a simple choice would be to set θiα (yi ) := deg(i )−1 θi (yi )). Given the characterization of the local polytope as the intersection of “smaller” marginal polytopes (cf. Eq. 4.41),
we have that the LP-MAP problem in Eq. 4.50 can be expressed as:
maximize∑α ∈F

w.r.t.

∑ ∑i ∈ N ( α ) y i ∈ Yiµ|α ∈ MARG(G |α ),ζ i (yi ) ∈ R,s.t.

θiα (yi )µiα (yi ) +∑ θα (yα )µα (yα )!
(4.52)y α ∈Yα∀α ∈ F,∀i ∈ V, yi ∈ Yi ,µiα (yi ) = ζ i (yi ),∀i ∈ V, α ∈ N(i ), yi ∈ Yi ,where MARG(G |α ) is the marginal polytope of the α-subgraph of G , and we denote byµ|α :=

(µiα (.))i∈N(α) , µα (.)

(4.53)

62

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

a marginal vector in that α-subgraph. By introducing a Lagrange multiplier λiα (yi ) for eachagreement constraint, we can write the Lagrangian function as:
L(µ, ζ, λ) =∑ ∑ (θiα (yi ) + λiα (yi ))µiα (yi ) + ∑ θα (yα )µα (yα )

∑

α ∈F

i ∈ N ( α ) y i ∈ Yi

−∑

∑ ∑

α ∈ F i ∈ N ( α ) y i ∈ Yi!

y α ∈Yαλiα (yi )ζ i (yi ).

(4.54)

This function is to be maximized with respect to µ and ζ, and minimized with respect to λ.
Since the ζ-variables are unconstrained, we have(∑α∈F gα (λ|α ) if λ ∈ Λ,
max L(µ, ζ, λ) =(4.55)+∞
otherwise,
µ,ζ
where

(
Λ :=

∑

λ

α ∈N ( i ))λiα (yi ) = 0,∀i ∈ V, yi ∈ Yi,

(4.56)and gα (λ|α ) is the solution of the following local problem (called the α-subproblem):gα ( λ | α ) : =maxµ|α ∈MARG(G |α )∑(θiα (yi ) + λiα (yi ))µiα (yi ) + ∑ θα (yα )µα (yα )!
.

(4.57)yαi,yiThe dual problem becomesminimize ∑α∈F gα (λ|α )w.r.t.
λ ∈ Λ.

(4.58)In the literature, the optimization problem in Eq. 4.58 is commonly referred to as the master,and each α-subproblem in Eq. 4.57 as a slave. Note that each of these α-subproblems is itself
a MAP assignment problem in the α-subgraph G |α (cf. 4.34). As a consequence, the solutionµb|α will be integer and will correspond to a particular configuration ybα .
The dual problem (4.58) can be solved with a projected subgradient algorithm.21 By Danskin’s rule, a subgradient of gα (λ|α ) is readily given by∂gα (λ|α )= µbiα (yi ),∂λiα (yi )∀i, α ∈ N(i );(4.59)and the projection onto Λ amounts to a centering operation. The α-subproblems (Eq. 4.57)
can be handled in parallel and then have their solutions gathered for computing this projection and update the Lagrange variables. Putting these pieces together yields Algorithm 5.
For compactness, we use vector notation: we write θα := (θα (yα ))yα ∈Yα , θi := (θi (yi ))yi ∈Yi ,ζ i , ωiα and λiα . We assume a black-box procedure ComputeMap, whichand similarly for µbiα , breceives log-potentials as input and outputs the local MAP (as a degenerate marginal vector).

21 The same algorithm can be derived by applying Lagrangian relaxation to the original MAP. A slightly dif-ferent formulation is presented by Sontag et al. (2011) which yields a subgradient algorithm with no projection.

4.6. LINEAR PROGRAMMING RELAXATION FOR MAP INFERENCE

63

Algorithm 5 Dual Decomposition with Projected Subgradient (Komodakis et al., 2007)
1: input: graph G , parameters θ, maximum number of iterations T, stepsizes (ηt )tT=1
2: initialize λ = 0
3: for t = 1 to T dofor each factor α ∈ F do
set unary log-potentials ωiα := deg(i )−1 θi + λiα , for i ∈ N(α)
6:set log-potential vector ω|α := ((ωiα )i∈N(α) , θα )7:
set µb|α := ComputeMap(ω|α )8:
end forbiα
9:compute average ζ i := deg(i )−1 ∑α∈N(i) µ10:update λiα := λiα − ηt (µbiα − ζ i )11: end for
12: output: dual variable λ4:

5:Algorithm 5 inherits the properties of subgradient algorithms, hence it converges to the
optimum of (4.58) if the stepsize sequence (ηt )t∈T is diminishing and nonsummable: ηt ≥0, ∀t; lim ηt = 0; and ∑∞t=1 ηt = ∞ (Bertsekas et al., 1999). In practice, convergence can bequite slow if the number of slaves is large, as we will see in Chapter 7. In Chapter 6, we willpropose a new LP-MAP inference algorithm that is more suitable for that kind of problem.

64

CHAPTER 4. GRAPHICAL MODELS AND INFERENCE ALGORITHMS

Part II

Inference

65Chapter 5

Constrained Structured Prediction
In Chapters 3 and 4, we have described models and algorithms for structured prediction. We
have seen how graphical models are a useful representational framework, but we restrictedattention to strictly positive potential functions. In this chapter, we present novel contributions for constrained structured prediction models, which incorporate hard constraints as a wayof ruling out “forbidden” configurations. The contributions presented in this chapter are:
• We provide a unified representation for constrained structured prediction problems byformalizing the concept of a constrained factor graph. This representation puts together
several models previously proposed in the literature, such as constrained conditionalmodels and grounded Markov logic networks, providing a unified treatment.
• We characterize the geometry of constrained factor graphs. This is done by extending
the machinery of Chapter 4 to our constrained case. In particular, we extend the notions
the marginal and local polytopes of constrained graphs, establish Fenchel-Legendre
duality, and derive variational representations.
• We introduce an inventory of hard constraint factors that are able to express arbitrary
first-order logic constraints. We also provide a concise representation of their marginal
polytopes, which will be useful in later chapters.
• We provide the necessary algorithmic tools to deal with these factors. Specifically,
we describe how to compute sum-product (and max-product) messages, how to compute marginals (and max-marginals), as well as the partition function and the factor
entropies. We also provide more general results concerning how all these quantities depend solely on a black-box algorithm for computing the marginals (or max-marginals)
and evaluating the partition function associated with each factor, independently.
Some of the results presented in this chapter were originally introduced in Martins et al.
(2010f), namely the inventory of hard constraint logic factors, as well as the message update
equations, the marginals, etc. That paper, however, was focused on a particular application,
dependency parsing, so many details and proofs were omitted from the manuscript. This
chapter presents those results more leisurely, and adopts a general framework perspective
rather than being application-oriented. It also extends the paper by presenting the geometriccharacterizations listed above.
67

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

68

5.1 Motivation and Related WorkIn Chapter 4, we have described graphical models with strictly positive potentials, which results in a distribution P(Y ) with full support on the product set ∏i∈V Yi . Constrained models,on the other hand, assume that the set of admissible outputs may be properly contained in that
product set, i.e., Y( x ) ⊂ ∏i∈V Yi . There are several motivations for considering constraints:• Sometimes the set of admissible outputs Y( x ) is inherently constrained. This is
so in problems arising in error-correcting coding theory (Richardson and Urbanke,2008), bipartite matching (Duchi et al., 2007), computer vision (Nowozin and Lampert,2009), and also natural language processing (Sutton, 2004; Smith and Eisner, 2008).
For those problems, only certain arrangements of parts yield valid outputs (words ina codebook, matchings, connected regions, and trees, respectively). While constraints
can in practice be enforced by employing log-potentials with large absolute values,that strategy precludes exploiting the structure and sparsity of the constraints andoften leads to inefficient algorithms.
• We may want to inject prior knowledge in the form of declarative constraints. These
could be, e.g., first order logic expressions written by experts. The inclusion of such
constraints yields more accurate models, and is especially useful when annotated datais scarce (Roth and Yih, 2004; Punyakanok et al., 2005; Riedel and Clarke, 2006; Richardson and Domingos, 2006; Chang et al., 2008; Poon and Domingos, 2009; Meza-Ruiz andRiedel, 2009).
• We may want to enrich our model by adding new parts that are logical functions ofexisting parts. This can be accomplished by adding those parts as variable nodes in
the graphical model, and expressing the logical functions as hard constraint factors, inorder to enforce consistency with the already existing variables. This turns out to playa very important role in the multi-commodity flow models for dependency parsingthat we will introduce in Chapter 7.

5.1.1

Related Work

The importance of modeling constraints in structured problems was realized a long timeback. Early attempts to fuse the semantics of probability theory and first-order logic appear
in Gaifman (1964); Halpern (1990); more recently, this line of research has gained prominence through probabilistic programming systems, such as BLOG (Milch et al., 2007) andMarkov logic networks (Richardson and Domingos, 2006). Related frameworks are probabilistic Horn abduction (Poole, 1993), probabilistic constraint logic programming (Riezler,
1998), probabilistic relational models (Friedman et al., 1999), and relational Markov networks(Taskar et al., 2002). Many of these are formalisms for probabilistic reasoning in knowledge
databases with soft and hard first-order logic constraints. Once grounded, those networks
become constrained graphical models.
A related line of research is that of constrained conditional models (Roth and Yih, 2004;
Punyakanok et al., 2005; Chang et al., 2008). In that framework, one starts from a vanillastructured conditional model (such as a CRF), in which inference and learning are tractable;
then, a list of domain-dependent side constraints is specified, allowing us to inject rich prior5.2. CONSTRAINED GRAPHICAL MODELS AND THEIR GEOMETRY

69

knowledge. These are usually declarative constraints written in first-order logic. In general,
exact inference becomes intractable; most of the previous work resorts to small and medium
scale problems, employing off-the-shelf ILP solvers for MAP decoding. A common strategy
is to ignore the constraints at training time—this strategy is called “L+I” (“Learning Plus Inference”) in Punyakanok et al. (2005). We will interpret this strategy later as a relaxation of
the marginal polytope, which we show to be looser than LP-MAP. Constrained conditionalmodels have been applied in semantic role labeling (Roth and Yih, 2005), coreference resolution (Denis and Baldridge, 2007), sentence compression (Clarke and Lapata, 2008), anddependency parsing (Riedel and Clarke, 2006). They have also been used in the context ofsemi-supervised constraint-driven learning (Chang et al., 2007).
Apart from the work mentioned above, there have been ad-hoc adaptations of message
passing algorithms to solve particular constrained problems in graphical models, exploiting
the structure of the constraints. Examples are the bipartite matching model of Duchi et al.
(2007), the loopy BP dependency parser of Smith and Eisner (2008), and the sparse message
passing algorithm for weighted SAT of Culotta et al. (2007).
This chapter is organized as follows: we study the geometry of constrained models in
Section 5.2. In Section 5.3 we provide an inventory of hard constraint factors that can express logic constraints. We derive expressions for computing messages, marginals, and maxmarginals in Section 5.4. Section 5.5 characterizes the LP-MAP and Bethe entropy approximations. We suggest paths for future work in section 5.6, and we conclude in Section 5.7.

5.2 Constrained Graphical Models and Their Geometry
We have seen in Chapter 4 that graphical models with strictly positive potentials can be seen
as exponential families (cf. Eq. 4.16):
Pθ (y) =1
exp(θ · φ(y)) Q(y),
Z (θ)

(5.1)

where θ ∈ RR is a vector of factor and variable log-potentials, and φ(y) := χ(y) ∈ RR is anindicator vector of local configurations. In that section—and in all Chapter 4—we restrictedattention to the case where Q was constant and hence non-informative. There might be goodreasons, however, for a less trivial choice of Q. For example, we may let Q vanish on someconfigurations (those which are not in the set of admissible outputs Y( x ) ⊆ ∏i∈V Yi ):(Q(y) =

1, if y ∈ Y( x )0, otherwise.

(5.2)

This choice ensures that any distribution in the exponential family assigns null probability
to any y ∈/ Y( x ). To allow specifying a list of constraints with different scopes, we assume afactorization Q(y) := ∏ β Q β (y β ), where Q β (y β ) := [[y β ∈ Sβ ]], with each β ⊆ V and Sβ being
an acceptance set of partial configurations on β.
Definition 5.1 A constrained factor graph is a bipartite undirected graph G = (V ∪ F ∪ H, E),
where each edge in E has one endpoint in V and another in F ∪ H, and the sets V, F, and H are all
disjoint. Elements of V are called variable nodes, elements of F are soft factor nodes, and those of

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

70

Figure 5.1: Constrained factor graphs. We represent soft factors in green above the variable nodes, and hard constraints in black below the variable nodes. Left: a global constraintthat confines the set of admissible outputs to a particular codebook. Right: Typical hardconstraints in declarative constraint programming. One of them is a factor connecting existing variables to a new extra variable. This allows introducing scores that depend on theevaluation of a logical function of the former.

H are hard constraint factor nodes. In addition:• Each variable i ∈ V has a strictly positive potential function ψi : Yi → R++ .
• Each soft factor α ∈ F has a strictly positive potential function ψα : Yα → R++ .
• Each hard factor β ∈ H has a binary potential function ψβ : Yβ → {0, 1} defined as ψβ (y β ) =[[y β ∈ Sβ ]], where Sβ ⊆ Yβ is an acceptance set.
Figure 5.1 shows examples of constrained factor graphs, where the hard constraint factors
have different usages. In the sequel, we will use constrained factor graphs to represent conditional probability distributions Pw (Y | X = x ) defined as structured linear models, in which
potentials on variables and soft factors are parameterized via ψi (yi ) := exp(w · f i ( x, yi )) andψα (yα ) := exp(w · f α ( x, yα )). Hard factors, in turn, induce a function
Q( x, y) :=∏ [[yβ ∈ Sβ ]].

(5.3)

β ∈H

Hence, Pw (Y | X = x ) can be written as

Pw (y| x ) :=







1
exp
Z (w, x )∑ w · f p (x, y p )p ∈P!
, if y β ∈ Sβ , ∀ β ∈ H,

0,

(5.4)

otherwise,

where P := V ∪ F is the set of places (cf. Eq. 4.26). The set of admissible outputs is(Y( x ) =y ∈ ∏ Yii ∈V)

∀ β ∈ H : y β ∈ Sβ ( x) ,(5.5)

and the partition function isZ (w, x ) =∑ exp ∑ w · f p (x, y p )y ∈Y( x )

p ∈P

!
.

(5.6)

5.2. CONSTRAINED GRAPHICAL MODELS AND THEIR GEOMETRY

71

Binary Constrained Factor Graphs. As mentioned in Section 5.1.1, many problems involving first-order logic probabilistic models have been considered (Friedman et al., 1999;
Richardson and Domingos, 2006). Given some evidence, those models can be “compiled”
into a factor graph whose variable nodes represent truth values, i.e., we have Yi = {0, 1} forevery i ∈ V. We call such a graph a binary constrained factor graph. When all variables arebinary, it is convenient to use a more compact representation instead of the overcomplete
parametrization. For constrained pairwise MRFs, an appealing choice is the representation
described in Example 4.1 for Ising models:Ps (y| x ) =







1
exp
Z (s, x )∑ si yi + ∑ sij yi y j
i!
, if y β ∈ Sβ , ∀ β ∈ H,

ij

0,

(5.7)

otherwise.

Any canonical overcomplete parameter vector θ can be converted to a parameter vector s
through Eq. 4.25. Without loss of generality, we may assume that s has the form si :=w · f i ( x ) and sij := w · f ij ( x ). We will come back to this parametrization in several places.

5.2.1

Marginal and Local Polytopes

In Section 4.3.2, we have defined the marginal polytope of an unconstrained factor graph, as
the set of realizable marginal vectors. We now extend that definition to constrained factor
graphs G = (V ∪ F ∪ H, E). We let P = V ∪ F be the set of places of G (note that this set
contains the variable nodes and the soft factor nodes, but does not contain the hard factor
nodes). Recall that a part is a pair ( p, y p ), where p ∈ P is a place and y p ∈ Y p is a local
configuration. We denote by R := {( p, y p ) | p ∈ P, y p ∈ Y p } the set of parts of G . We
consider marginal vectors µ := (µr )r∈R indexed by the set of parts, such that µ( p,y p ) ≡ µ p (y p )denotes the posterior probability Pr{Y p = y p | X = x }. Given yb ∈ Y( x ), we define its indicator
vector χ(yb) ∈ R|R| as the vector whose entries [χ(yb)]( p,y p ) are 1 if yb p = y p , and 0 otherwise.

Definition 5.2 (Marginal Polytope) The marginal polytope of a constrained graph G is the setof marginals which are realizable for distributions that vanish outside Y( x ):(MARG(G ) : =µ ∈ R|R|∃ P(Y ) s.t. P(y) = 0, ∀y ∈/ Y( x ) ∧µ = EY ∼ P(Y ) [χ(Y )]= conv{χ(y) | y ∈ Y( x )}.

)

(5.8)

As before, the vertices of the marginal polytope are in one-to-one correspondence with theelements of Y( x ). Note that each component of a marginal vector µ ∈ MARG(G ) stands fora marginal probability of some variable or soft factor configuration—there are no components
for hard factor configurations. Hard constraint factors play an important role, however: they“eliminate” the vertices corresponding to forbidden configurations, chopping off pieces ofthe polytope.
In Section 4.4, we have described an outer approximation of the marginal polytope, called
the local polytope, which can be seen as the intersection of “smaller” marginal polytopes,
one per factor, lifted to the ambient space R|R| (cf. Eq. 4.41). To generalize that notion to
constrained graphs, we need first to define what is the “smaller” marginal polytope of a

72

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

hard constraint factor. Let β ∈ H be a hard factor connected to a set of variables N( β), withan acceptance set Sβ . For each yb β ∈ Sβ , define χ β (yb β ) as the vector in R∑i∈N(β) |Yi | whose(i, yi )-th component (with i ∈ N( β) and yi ∈ Yi ) is 1 if ybi = yi , and 0 otherwise. We define
MARG(G | β ) = conv{χ β (y β ) | y β ∈ Sβ }.

(5.9)

We then define the local polytope as:n
LOCAL(G ) := µ ∈ R|R|µ|α ∈ MARG(G |α )ofor every α ∈ F ∪ H ,(5.10)

where, for soft factors α ∈ F, we denote by µ|α the restriction of µ to the entries µα (.) andµi (.) for every i ∈ N(α); and for hard factors β ∈ H, we denote by µ| β the restriction of µ to
the entries µi (.) for every i ∈ N( β).
Proposition 5.1 Let G be a constrained factor graph, and denote by Gu the unconstrained graph thatis obtained from G by removing the hard factors. The following chain of inclusions hold:
MARG(G ) ⊆ LOCAL(G ) ⊆ LOCAL(Gu ).

(5.11)

Moreover, if Gu is free of cycles, then MARG(G ) ⊆ LOCAL(G ) ⊆ MARG(Gu ).
Proof.
Any marginal vector µ ∈ MARG(G ) much satisfy µ|α ∈ MARG(G |α ) for everyα ∈ F ∪ H, hence we have MARG(G ) ⊆ LOCAL(G ). We can express LOCAL(G ) as:
LOCAL(G ) = LOCAL(Gu ) ∩\β ∈Hconv{χ(y) | y β ∈ Sβ },(5.12)

which implies LOCAL(G ) ⊆ LOCAL(Gu ), proving the first chain of inclusions. For thesecond one, note that, if Gu is free of cycles, Proposition 4.3 states that LOCAL(Gu ) =MARG(Gu ), from which the result follows.
Proposition 5.2 The integral points of LOCAL(G ) are the vertices of MARG(G ). Hence:
MARG(G ) = conv(LOCAL(G ) ∩ Z|R| ).

(5.13)

Proof. Analogous to Proposition 4.4.
Propositions 5.1–5.2 allow us to understand geometrically some important approximations that are made in practice. For a constrained graph G which is constructed from acycle-free unconstrained graph Gu by adding hard constraints, we have the two following
outer bounds for the (generally intractable) marginal polytope MARG(G ):• the local polytope LOCAL(G ), whose integer vertices are precisely the vertices of MARG(G ),but which may contain additional fractional vertices (cf. Proposition 5.2);• the marginal polytope of the unconstrained graph, MARG(Gu ), a looser bound which may
contain additional integer vertices (namely, the configurations which are ruled out by
the constraints).

5.2. CONSTRAINED GRAPHICAL MODELS AND THEIR GEOMETRY73

This will be next illustrated with a concrete example.1
Example 5.1 Consider the factor graph G in Figure 5.2, with two binary variables (Y1 = Y2 ={0, 1}), one pairwise factor connecting the two variables, and one hard constraint factor β imposingthat at least one of the two variables takes the value 1 (i.e., with an acceptance set Sβ = {01, 10, 11}).
With an overcomplete parametrization there are 8 parameters: 2 for each variable, and 4 for eachpossible configuration in the pairwise factor. The marginal polytope MARG(G ) is thus an object
embedded in R8 . However, there are only three degrees of freedom, and in fact MARG(G ) can be seen
as a lifted version of a 3-dimensional polytope. Let us consider a minimal parametrization throughthe marginal probabilities z1 := µ1 (1), z2 := µ2 (1) and z12 := µ12 (1, 1). As can be easily verified
by writing a probability table, the remaining marginals can be obtained from these as:µ1 (0) = 1 − z1 ,µ12 (1, 0) = z1 − z12 ,µ2 (0) = 1 − z2 ,µ12 (0, 1) = z2 − z12 ,µ12 (0, 0) = 1 − z1 − z2 + z12 .

(5.14)

We can thus parameterize a marginal vector as z = (z1 , z2 , z12 ). Noting that the set of admissible
outputs is Y = {01, 10, 11} and that y12 = y1 ∧ y2 , we can regard MARG(G ) as a lifted version of
the polytope2
Z = conv{(0, 1, 0), (1, 0, 0), (1, 1, 1)}.
(5.15)
Consider now the graph Gu that is obtained from G by removing the hard constraint. The polytope
MARG(Gu ) is an outer bound of MARG(G ) and is a lifted version of the polytope
Zu = conv{(0, 0, 0), (0, 1, 0), (1, 0, 0), (1, 1, 1)}
(
)
z
≤
z
,
z
≤
z
,
2
12
1
12
=
(z1 , z2 , z12 ) ∈ [0, 1]3.
z12 ≥ z1 + z2 − 1(5.16)

Finally, let us see how the local polytope LOCAL(G ) looks. Since the unconstrained part of the graphis cycle-free, we have that LOCAL(Gu ) = MARG(Gu ). It will be shown in Section 5.3.2 that themarginal polytope of the β-subgraph G | β is a lifted version ofZβ = {(z1 , z2 ) ∈ [0, 1]2 | z1 + z2 ≥ 1}.
From Eq. 5.12, we then have that LOCAL(G ) is a lifted version ofz12 ≤ z1 , z12 ≤ z2 , Z̄ =(z1 , z2 , z12 ) ∈ [0, 1]3 z12 ≥ z1 + z2 − 1,
z1 + z2 ≥ 1



1 1 1
, ,
.
= conv (0, 1, 0), (1, 0, 0), (1, 1, 1),2 2 2(5.17)(5.18)

Hence we have a chain Z ⊂ Z̄ ⊂ Zu ; the two outer bounds of the marginal polytope are the local
1 These two outer bounds are commonly used in practice, as we shall see in the sequel. The first one isused in turbo-training (Chapters 6–7), the second one is precisely what is used in the L+I training of constrained
conditional models (Roth and Yih, 2004; Punyakanok et al., 2005; Chang et al., 2008).
2 See Appendix B for a definition of lifting.

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

74(1,1,1)(0,1,0) (1,0,0)(0.5, 0.5, 0.5)

(0,0,0)Figure 5.2: Left: A constrained factor graph G with two binary variables (Y1 = Y2 = {0, 1}),
one pairwise factor, and one hard constraint factor imposing that at least one of the variables has the value 1. Right: The marginal polytope MARG(G ) (in red), and its twoouter bounds: the local polytope LOCAL(G ) (in blue), and the polytope MARG(Gu ) (inorange). Each of these outer bounds introduce one extra vertex, and we have a chain
MARG(G ) ⊆ LOCAL(G ) ⊆ MARG(Gu ). See text for details.

polytope Z̄, which has an additional fractional vertex ( 12 , 12 , 12 ), and a looser polytope Zu (the marginal
polytope of the unconstrained graph), with an additional integer vertex (0, 0, 0). Figure 5.2 shows thethree nested polytopes.

5.2.2

Duality and Variational Representations

Most theoretical results seen in Chapter 4 still hold for constrained factor graphs, including
Propositions 4.1–4.2, and seamlessly Theorem 4.6 (see Yedidia et al. 2004, Conjectures 1–2). We state here the adaptation of Propositions 4.1–4.2 for constrained linear conditionalmodels. In doing so, we also depart from a “canonical overcomplete parameterization.”
Rather, we extend those propositions to the more general scenario where arbitrary features
are allowed and some weights are tied (i.e., shared accross factors). This is the most common
scenario in NLP problems.
Denote by Ex the exponential family of distributions of the form in Eq. 5.4, for a fixedx ∈ X. As before (Section 4.3), we let H ( P(Y | X = x )) := EY [− log P(Y | x )] denote the entropyof P(Y | X = x ). Recall that the feature vectors f ( x, y) ∈ RD decompose as in Eq. 3.12:
f ( x, y) := ∑ f p ( x, y p ),(5.19)

p ∈P

where P = V ∪ F is the set of places. For convenience, we introduce the D-by-|R| matrix
F( x ), which “stacks” the local feature vectors f p ( x, y p ) for each part ( p, y p ) ∈ R.
Proposition 5.3 There is a map coupling each distribution Pw (.| x ) ∈ Ex to a unique µ ∈ MARG(G )such that Ew [χ(Y )] = µ. Define H (µ) := H ( Pw (Y | X = x )) if some Pw (.| x ) is coupled to µ, andH (µ) := −∞ if no such Pw (.| x ) exists. Then:
1. The log-partition function has the following variational representation:log Z (w, x ) =maxµ∈MARG(G )w > F ( x ) µ + H ( µ ).

(5.20)

2. The problem in Eq. 5.20 is convex and its solution is attained at µb := µ(w), i.e., the marginal
vector corresponding to the distribution Pw (.| x ).

5.2. CONSTRAINED GRAPHICAL MODELS AND THEIR GEOMETRY

75

Figure 5.3: Dual parametrization of the distributions in Ex . Our parameter space (left) is
linearly mapped to the space of canonical overcomplete parameters (middle). The latter ismapped to the marginal polytope MARG(G ) (right). In general only a subset of MARG(G ) isreachable from our parameter space. Any distribution can be parametrized by a vector w ∈RD or by a point µ ∈ MARG(G ). Note that different parameter vectors w may parametrize
the same distribution Pw (.| x ); each equivalence class [w] is an affine subspace of RD .

3. The gradient of the log-partition function is ∇w log Z (w, x ) = F( x )µb.
4. The entropy of Pw (.| x ) can be expressed as:
H ( Pw (Y | X = x ) = −w> F( x )µb + log Z (w, x ).

(5.21)

5. The MAP configuration yb := arg maxy∈Y(x) Pw (y| x ) can be obtained by solving the LP:
χ(yb) = argmaxµ∈MARG(G )w> F( x )µ.

(5.22)

Proof. Wainwright and Jordan (2008, Theorem 3.4) provide a proof for a minimal representation, and extend it in their Appendix B.1 to the canonical overcomplete case (as stated in our
Proposition 4.2). Those results are derived for general exponential families with arbitrarybase measures, hence hold for any Q in Eq. 5.1, which includes constrained factor graphs.
The canonical overcomplete representation corresponds to the case where F( x ) is the identity matrix. In that case, the map from the parameter space to the relative interior of themarginal polytope is surjective. In the general case, we have a linear map w 7→ θ = F( x )> w
that “places” our parameters w ∈ RD onto a linear subspace of the canonical overcomplete
parameter space; therefore, our map w 7→ µ is not necessarily onto relint MARG(G ), unlike
in Wainwright and Jordan (2008), and our H (µ) is defined slightly differently: it can take thevalue −∞ if no w maps to µ. This does not affect the expression in (4.36), since the solutionof this optimization problem with our H (µ) replaced by theirs is also the feature expectation
under Pw (.| x ) and the associated µ, by definition, always yields a finite H (µ).
Figure 5.3 provides an illustration of the dual parametrization implied by Proposition 5.3.
When the graph has cycles, it is often intractable to solve any of the optimization problems
in Eqs. 5.20 and 5.22. In Chapter 4, we have discussed two approximations:• Replacing MARG(G ) by the local polytope LOCAL(G );• Replacing H ( Pw (Y | X = x )) by Bethe’s entropy approximation HBethe ( Pw (Y | X = x )).

76

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

Most of the story told in Chapter 4 connecting these surrogates to approximate inference
algorithms still applies to constrained factor graphs—this includes the loopy BP algorithm,
as well as the other message passing and dual decomposition algorithms. There are somequestions that need to be answered though, to which we devote the remaining sections:• How to compute messages or solve subproblems involving hard constraint factors?
In particular, how to deal with hard factors with a large degree, sidestepping the exponential runtime with respect to that number? Intuitively, it looks like we should be
able to exploit the sparsity of the potential functions for those graphs. We will see
in Section 5.4 that this intuition is correct, and we will provide a general recipe forcomputing messages and solving subproblems for a wide range of factors.
• When hard constraints come into play, how good are the approximate inference algorithms? In particular, how accurate is the Bethe entropy approximation, and how
does that affect the sum-product loopy BP algorithm? As expected, this strongly depends on the topology of the graph and on the actual hard constraints. We will seethat in general, the story is more intricate in marginal inference (since there is need ofapproximating the entropy), than in MAP inference (where only the marginal polytope
is approximated).
• How tractable are the marginal and local polytopes? As suggested in Example 5.1, themarginal polytope of a constrained factor graph G is in general intractable, even when
the factor graph without the constraints Gu is tractable. However, as the same example
shows, the local polytope of G is a tighter approximation than the one that results fromignoring the constraints. For dealing with this local polytope, all we need is to be able
to express the “smaller” marginal polytopes for the hard factors (cf. Eq. 5.9). We will
see in Section 5.3 that those polytopes have often a compact representation.
• How can we express constraints in first-order logic in a modular way, and run approximate inference algorithms that handle those constraints? This will also be addressed in Section 5.3, where we introduce a small set of hard constraint factors thathas great expressive power. In particular, we will be able to run approximate inference
for any problem involving constraints in first-order logic in linear time, with respect tothe size of the longest clause.

5.3 An Inventory of Hard Constraint Factors
In this section, we present hard constraint factors that work as building blocks for writing
constraints in first-order logic. Each of the factors discussed here performs a logical function,and hence we represent them graphically through logic gates. We start by enumerating thesefactors, and later in Section 5.4 we will show how to compute messages and perform othercalculations that are specific to each of them, but which will serve the same purpose—
learning and inference in constrained models.

5.3. AN INVENTORY OF HARD CONSTRAINT FACTORS

XOR

OR

77

OR-OUT

Figure 5.4: Logic factors introduced in this thesis and their marginal polytopes. Left: themarginal polytope of the XOR factor is the probability simplex. Middle: the OR factor has as
marginal polytope a faulty hypercube, with one vertex removed. Right: the OR-with-output
factor has a more intricate marginal polytope.

5.3.1

One-hot XOR and Uniqueness Quantification

The one-hot XOR factor is linked to K ≥ 1 binary variables and is defined through thefollowing potential function:(
ψXOR (y1 , . . . , yK ) :=1 if ∃!k ∈ {1, . . . , K } s.t. yk = 1
0 otherwise,(5.23)

where the symbol ∃! means “there is one and only one.” In words, the potential is zerovalued unless exactly one of the inputs (among y1 , . . . , yK ) takes the value 1. The name XOR
stems from the fact that, with K = 2, the potential function behaves like a logic ExclusiveOR. Hence ψXOR can be seen as a generalization of Exclusive-OR for K ≥ 2. The prefix“one-hot” serves to emphasize that this generalization evaluates to 1 if there is precisely one“active” input (where “active” means having a value of 1), avoiding confusion with anothercommonly used generalization related with parity checks.3
The XOR factor appeared in Smith and Eisner (2008) under the name “Exactly1,” where
it was employed in the context of dependency parsing to constrain each word to have asingle head. Since the need of imposing uniqueness constraints is frequent in NLP, it is avery useful factor in practice, as it can express a statement in first-order logic of the form4∃!y : R(y).

(5.24)The marginal polytope associated with the XOR factor is, by definition, the convex hull

3 To be more precise, that alternative definition of Exclusive-OR for K ≥ 2 evaluates to 1 if there is an odd

number of active inputs. It is also a widely used factor in models for error-correcting decoding, such as lowdensity parity-check codes (Gallager, 1962; Feldman et al., 2005; Richardson and Urbanke, 2008).
4 It can also be employed for binarizing a categorical variable—in that case, one needs to constrain the variable
to take a unique value, and that is done by connecting all possible values to an XOR factor.

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

78

NULL

XOR

Figure 5.5: A logical constraint “AtMost1” constructed with a XOR factor with a slack variable. Either one of the first two variables is active, or none is, turning on the slack variable.

of the acceptance set, which is a lifted version of the set:no
ZXOR = conv y ∈ {0, 1}K | ∃!k ∈ {1, . . . , K } s.t. yk = 1()=

z ∈ [0, 1]K(5.25)

K∑ zk = 1;(5.26)

k =1

in other words, the marginal polytope of the XOR factor is the probability simplex. This is
depicted graphically in Figure 5.4.
An XOR factor can also be used to define the function “AtMost1” introduced by Smithand Eisner (2008) (which evaluates to 1 if there is at most one active input), by adding one
extra input yK+1 (which represents a null value). This is illustrated in Figure 5.5.

5.3.2

OR and Existential Quantification

The OR factor represents a disjunction of K ≥ 1 binary variables. It is defined through thefollowing potential function:(
ψOR (y1 , . . . , yK ) :=1 if ∃k ∈ {1, . . . , K } s.t. yk = 1
0 otherwise,(5.27)

where the symbol ∃ has the usual meaning “there is at least one.” In other words, thepotential is one-valued in all cases except when all the inputs are inactive, in which case it
is zero-valued— hence the potential function behaves like a logic OR.
The difference with respect to the XOR factor is that OR imposes existence but notuniqueness. It can be used to represent a statement in first-order logic of the form

∃ y : R ( y ).

(5.28)The marginal polytope associated with the OR factor is a lifted version of the set:noZOR = conv y ∈ {0, 1}K | ∃k ∈ {1, . . . , K } s.t. yk = 1()=

z ∈ [0, 1]K(5.29)

K∑ zk ≥ 1;(5.30)

k =1

geometrically, it is a “faulty” hypercube, i.e., a hypercube which was carved by removing

5.3. AN INVENTORY OF HARD CONSTRAINT FACTORS

79

one vertex (in this case, the origin). This is depicted in Figure 5.4.
Since it behaves like a disjunction function, it is appealing to use an OR factor as acomponent of a larger network that encodes a more complex logical statement, e.g., involving
disjunctions of predicates. On the other hand, it is also useful in isolation: there are many
problems in natural language processing and relational learning that make use of categoricalvariables that can take one or more values. An OR factor can be employed to impose thisconstraint that they take at least one value.

5.3.3

Negations and De Morgan’s law

We extend the two factors above—as well as the ones we will present in the sequel—toaccommodate negated inputs. Later, we will see that all computations that involve the original
factors can be easily extended to allow negated inputs while reusing the same black box thatsolves the original problems. The ability to handle negated variables adds a great degree of
flexibility. For example, it allows us to handle negated conjunctions (NAND; discussed in
the case K = 2 by Smith and Eisner 2008):(
ψNAND (y1 , . . . , yK ) :==0 if yk = 1, ∀k ∈ {1, . . . , K }1 otherwise,ψOR (¬y1 , . . . , ¬yK )(5.31)

as well as implications (IMPLY):(ψIMPLY (y1 , . . . , yK , yK+1 ) :=

=

1 if

V

K
k =1 y k

k =1 ¬ Qk ( x ), and that

VK

⇒ y K +1

0 otherwise,ψOR (¬y1 , . . . , ¬yK , yK+1 ).

This is so because, from De Morgan’s laws, we have that ¬WKi =1 Qk ( x ) ⇒ R ( x ) is equivalent toVKk =1 Q k ( x )(5.32)
is equivalent toWn

k =1 ¬ Q k ( x ) ∨ R ( x ).

Let α be a binary constrained factor with marginal polytope Zα , and β be a factor obtained
from α by negating the kth variable. Then, the marginal polytope associated with the factor
β, which we denote by Zβ , is a simple symmetric transformation of Zα :Zβ =z ∈ [0, 1]

K( z 1 , . . . , z k −1 , 1 − z k , z k +1 , . . . , z K ) ∈ Zα .

(5.33)

By induction, this can be generalized to an arbitrary number of negated variables.

5.3.4

Logical Variable Assignments: XOR-with-output and OR-with-output

All cases seen above involve taking a group of existing variables and defining a constraint.
Alternatively, we may want to define a new variable (say, yK+1 ) which is the result of an
operation involving other variables (say, y1 , . . . , yK ), as in Figure 5.1 (right). Among otherthings, this will allow us to deal with “soft constraints,” i.e., constraints that can be violated
but whose violation will decrease the score by some penalty.

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION80

We start with the XOR-with-output factor, which we define as follows:
 1 if yK+1 = 1 ∧ ∃!k ∈ {1, . . . , K } : yk = 1
ψXOR−out (y1 , . . . , yK , yK+1 ) :=1 if yK+1 = 0 ∧ ∀k ∈ {1, . . . , K } : yk = 0 0 otherwise.

(5.34)

In words, this factor constrains at most one of the variables y1 , . . . , yK to be active; if one isactive, it constrains yK+1 = 1; if all are inactive, then it constrains yK+1 = 0. Interestingly,
this factor can be expressed using a regular XOR factor where the last variable is negated:ψXOR−out (y1 , . . . , yK , yK+1 ) = ψXOR (y1 , . . . , yK , ¬yK+1 ).

(5.35)

Using Eq. 5.33, we then have that the marginal polytope associated with the XOR-withoutput factor is a lifted version of the set:(
ZXOR−out =z ∈ [0, 1]

K +1

K

∑ z k = z K +1).

(5.36)

k =1

Another important logical assignment factor is OR-with-output:(ψOR−out (y1 , . . . , yK , yK+1 ) :=W

1 if yK+1 = k∈{1,...,K} yk
0 otherwise.

(5.37)

This factor constrains the variable yK+1 to indicate the existence (not necessarily uniqueness)of an active variable among y1 , . . . , yK . It can be used to impose the following statement infirst-order logic:
T ( x ) := ∃z : R( x, z).
(5.38)
Unlike the XOR-with-output case, the OR-with-output factor cannot be built by reusing an
OR or XOR factor with some inputs negated.5
The marginal polytope associated with the OR factor is a lifted version of the set:




_
K +1
(5.39)
ZOR−out = conv y ∈ {0, 1}
y K +1 =ykk ∈{1,...,K })(=z ∈ [0, 1]K+1

K∑ zk ≥ zK+1 , zk ≤ zK+1 , ∀k ∈ {1, . . . , K}.

(5.40)

k =1

This is also depicted graphically in Figure 5.4.
5 It can however be equivalently expressed as the product of K + 1 OR factors, since we haveψOR−out (y1 , . . . , yK , yK +1 )=K^

( y k ⇒ y K +1 ) ∧ y K +1 ⇒

k =1
K

=

∏ ψOR (¬yk , yK+1 )

K
_

!
yk

k =1

!
ψOR (y1 , . . . , yK , ¬yK +1 ),k =1

The reason we consider ψOR−out is that it may be beneficial to have a larger factor instead of many small ones,
as far as we can carry out all the necessary computations using the larger factors.

5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS

81

Soft constraints in first order logic. As mentioned above, factors that represent logical
functions with output variables allow us to deal with soft constraints. For example, suppose
we want a soft OR constraint involving the variables Y1 , . . . , YK , whose violation will affectthe score by a penalty q. One needs to do the following:
1. Introduce a new “slack” variable (call it YK+1 ) for representing the event that the constraint is violated; assign a log-potential of −q to this variable, which will be theamount of penalty for the violation.
2. Introduce an OR-with-output factor connecting inputs Y1 , . . . , YK and output YK+1 .

5.3.5

AND-with-output and Expressions with Universal Quantifiers

All the factors described so far will be placed in a binary factor graph. Interpreting the values of the variables as memberships in a subset, the corresponding MAP inference problem
can be seen an instance of set-valued (or pseudo-Boolean) optimization (Boros and Hammer,
2002) with side constraints. We may want to boost the expressive power of the model byintroducing score terms that depend on the conjunction of several variables (say y1 , . . . , yK ),i.e., of the form s1...K × ∏kK=1 yk . For K > 1, such scores are non-linear. An important technique in pseudo-Boolean optimization is linearization: one first creates an additional variableV(say yK+1 ) which is constrained to evaluate to the conjunction kK=1 yk , and then replace thenon-linear term by s1...K × yK+1 . The definition of the new variable can be made throughanother important logical assignment factor, AND-with-output:(ψAND−out (y1 , . . . , yK , yK+1 ) :=1 if yK+1 = kK=1 yk
0 otherwise.
V(5.41)The AND-with-output factor can be used to impose the following statement in first-orderlogic:T ( x ) := ∀z : R( x, z).
(5.42)By De Morgan’s laws, an AND-with-output factor can be constructed from an OR-withoutput by negating all the variables that are linked to it:ψAND−out (y1 , . . . , yK , yK+1 ) = ψOR−out (¬y1 , . . . , ¬yK , ¬yK+1 ).

(5.43)

5.4 Computing Messages and Solving Local Subproblems
Now that we have defined constrained factor graphs, characterized their geometry, andintroduced logic factors, we turn our attention to inference algorithms. We are going to see
how message passing and dual decomposition algorithms can also operate in constrained
factor graphs. Before discussing the global behaviour of those algorithms (which we will doin Section 5.5), we will describe how they behave locally—namely, how they can computemessages or solve local subproblems efficiently.
Recall from Section 4.2 that the cost of computing the factor-to-variable messages for ageneral factor α is O(exp(deg(α))), since it requires a summation/maximization over exponentially many configurations. At first sight, this fact seems to preclude the usage of the hard

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

82

constraint factors presented in Section 5.3, for any but a very small value of K := deg(α).
It turns out, however, that all those factors have potential functions which are extremely
sparse, and that this sparsity can be exploited algorithmically. We are going to show that,for all these cases, the factor-to-variable messages can be computed in time O(deg(α))—i.e.,the dependency in the factor degree is linear (rather than exponential). This extends previous findings of Duchi et al. (2007) and Smith and Eisner (2008), among others, who haveobtained similar results for other combinatorial factors.
We go farther by providing a general recipe for the computation of messages in arbitrary
factors (Algorithms 6–7), which depend only on oracles for the following tasks:• An oracle that, given the potential function of the factor, computes the local partition
function and marginals (for the sum-product case);• An oracle that, given the potential function of the factor, computes the mode and themax-marginals (for the max-product case).

5.4.1

Sum-Product Messages

Recall that, in the sum-product variant of the BP algorithm, the factor beliefs and the factorto-variable messages take the form (cf. Eqs. 4.8 and 4.10):
Mα → i ( y i ) ∝∑ ψα (yα ) ∏ Mj→α (y j )(5.44)∏ Mi → α ( y i ) .

(5.45)y α ∼ yij ∈N ( α )j 6 =ibα (yα ) ∝ ψα (yα )i ∈N ( α )A close look into Eq. 5.45 reveals that the variable-to-factor messages Mi→α (yi ) participate
in that expression as potential functions participate in a factored distribution; the case of
Eq. 5.44 is similar, but the probability function is integrated over the value assignments consistent with yi , resembling the definition of a marginal probability. We make this observation
formal through Proposition 5.4, a key result which provides a way of computing messages,
beliefs and entropies given the ability of computing marginals and the partition function.
Proposition 5.4 Let Gα be the α-subgraph of G , and consider a probability distribution factoring
according to Gα :Pω (yα ) ∝ ψα (yα ) ∏ exp(ωi (yi )).
(5.46)i ∈N ( α )Let Zα (ω ) be the corresponding partition function, and let µi (yi ; ω ) := Pω (yi ), for every i ∈ N(α),
denote the variable marginals.
Now consider an execution of the sum-product (loopy) BP algorithm run on the original graph G ,and define ω as given by the variable-to-factor messages:
ωi (yi ) := log Mi→α (yi ).

(5.47)

Then we have the following:
1. At any point of the algorithm, we have the following expression for the factor beliefs (Eq. 4.10):bα (yα ) = Pω (yα );5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS

83

2. Upon convergence, the variable beliefs (Eq. 4.9) satisfy bi (yi ) = µi (yi ; ω );3. At any point of the algorithm, the factor-to-variable messages can be computed as:
Mα → i ( y i ) ∝µi ( yi ; ω )= µi (yi ; ω ) exp(−ωi (yi )).
Mi → α ( y i )(5.48)

4. Upon convergence, the entropy of the factor, Hα (bα (.)), can be computed as:
Hα (bα (.)) = log Zα (ω ) −∑ ∑ µi (yi ; ω ) log Mi→α (yi )i ∈ N ( α ) y i ∈ Yi

= log Zα (ω ) −∑ ∑ µ i ( y i ; ω ) ωi ( y i ).

(5.49)i ∈ N ( α ) y i ∈ Yi

Proof. The first statement is immediate from the definition of factor beliefs (see Eq. 4.10)
along with Eq. 5.47. The second statement is a consequence of the calibration equation of
the factor and marginal beliefs (Eq. 4.11): given that bα (yα ) = Pω (yα ), we have that bi (yi )corresponds to the marginals of that distribution. The third statement can be obtained byrewriting Eq. 5.44 as
Mα→i (yi ) ∝ Mi→α (yi )−1 ∑ ψα (yα )y α ∼ yi

∏ M j → α ( y j ),(5.50)j ∈N ( α )and observing that the right hand side equals Mi→α (yi )−1 µi (yi ). Finally, the fourth statement
is a consequence of the first one and the conjugate duality relation between the log-partition
function and the negative entropy, expressed in Proposition 5.3.
Proposition 5.4 tells us that all the necessary computations in the sum-product (loopy)
BP algorithm which involve factor α (namely, computing the outgoing messages, the beliefs,
and the contribution of this factor to the Bethe entropy approximation) can be performed by
invoking a black box routine that just computes the partition function and the marginals forthe α-subgraph. Hence, we need only to be concerned with this black-box routine, which is,naturally, a property of the factor α.
We next revisit the hard constraint factors introduced in Section 5.3 and, in each case,obtain closed-form expressions for the factor-to-variable message ratios in terms of theirvariable-to-factor counterparts. We also derive closed-form expressions for the marginals,
partition function, and entropies. Since in all these factors the variables are binary-valued,
and the formulae for computing the messages are up to a normalization constant, only thefollowing message ratios matter:mα→i := Mα→i (1)/Mα→i (0),mi→α := Mi→α (1)/Mi→α (0).

(5.51)

The formulae for computing messages for a particular factor α can thus be seen as a transfer
function that maps from messages mi→α , to messages mα→i , ∀i ∈ N(α).
Since some of the derivations are a bit tedious, we leave all the details to Appendix C.1
and just summarize the results below.

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION84

The One-hot XOR Factor. To simplify notation and make things clearer, we assume thefactor is linked to variables 1 to K, and replace the subscript α by XOR. We refer to the full
assignment yα simply as y. Recall that we have ωi (yi ) := log Mi→XOR (yi ); since only ratios
matter, we will deal with mi→XOR := exp(ωi (1) − ωi (0)). Note that we have Mi→XOR (0) =(1 + mi→XOR )−1 and Mi→XOR (1) = mi→XOR (1 + mi→XOR )−1 .
In Appendix C.1, we derive the following expressions for the partition function and themarginals:
ZXOR (ω ) =µi (1; ω ) =µi (0; ω ) =

K

K

i =1

i =1

∑ mi→XOR × ∏(1 + mi→XOR )−1(5.52)

mi→XORK∑k=1 mk→XOR∑kK=1,k6=i mk→XOR.
∑kK=1 mk→XOR(5.53)(5.54)

From Proposition 5.4, this leads to the following message updates:! −1

K

mXOR→i =∑mk→XOR.

(5.55)

k =1,k6=i

The message updates in Eq. 5.55 have an intuitive interpretation in terms of competition
between variables: if all variables (apart from the ith) have sent messages that express alarge belief in an inactive state (i.e., low value of mk→XOR ), then mXOR→i will be set to a large
value, pushing the ith value to “take the lead” in being the active variable of the one-hot
XOR. On the other hand, if some other variable k “thinks” she should be active (through alarge value of mk→XOR ), then it will “shut off” the competing variables by making each ith
variable (i 6= k) receive a low value of mXOR→i .
The entropy can expressed in terms of the beliefs as follows (see Appendix C.1):
K

HXOR (bXOR (.)) = − ∑ bi (1) log bi (1).

(5.56)

i =1

This, again, has an intuitive interpretation. From the marginal polytope equations (Eq. 5.25),
we know that for the XOR factor we must have ∑iK=1 bi (1) = 1, hence the vector (bi (1))iK=1
represents a probability distribution of a random variable with K possible outcomes—thesame as the number of allowed configurations of the XOR factor. Then, Eq. 5.56 tells us thatthe entropy HXOR equals the entropy of that distribution.
In sum, we can compute all the relevant quantities (factor-to-variable messages, logpartition function, marginals and factor entropy) in time linear in K (rather than exponential).
The OR Factor. We next turn to the OR factor. In Appendix C.1, we derive expressions forthe log-partition function and ratio of marginals:
K

ZOR (ω ) = 1 − ∏(1 + mi→OR )−1 ;i =1µi (1; ω )= mi→ORµi (0; ω )K1−∏ (1 + mk→OR )−1

k =1,k 6=i! −1.(5.57)

5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS

85

From Proposition 5.4, we obtain the message updates:
K

1−

mOR→i =∏ (1 + mk→OR )

! −1
−1

.

(5.58)

k =1,k6=i

As in the XOR case, the message updates in Eq. 5.58 have an interpretation in terms ofcompetition between variables. However, competition is less sharp. As before, if all variables(apart from the ith) have sent messages that give large likelihood to an inactive state (i.e., lowvalue of mk→OR ), then mOR→i will be set to a large value, pushing the ith value to be active.
However, if some other variable k 6= i believes she should be active (through a large valueof mk→OR ), then the product in Eq. 5.58 will pushed to zero, which will yield mOR→i ≈ 1, a
non-informative message. In other words, if there is evidence that one of the variables (saythe kth) connected to a OR-factor is active, this will have a neutral effect on mOR→i for every
i 6= k, since the factor does not “care” anymore about the state of the other variables.
The entropy is given by:K

HOR (bOR (.)) = log 1 − ∏(1 + mi→OR )−1

!

i =1

K

−∑

K

+ ∑ log(1 + mi→OR )K

1 + mi→OR −(5.59)i =1∏ (1 + mk→OR )−1! −1

× mi→OR log mi→OR .

k =1,k 6=i

i =1

Unlike the XOR case, it does not seem straightforward to write this expression as a functionof the marginals only; yet, the entropy can still be computed from the incoming messages intime linear in K. In sum, all the relevant quantities (factor-to-variable messages, log-partition
function, marginals and factor entropy) can again be computed in linear time.

The OR-with-output Factor. In Appendix C.1, we derive expressions for the log-partition
function associated with the OR-with-output factor:
K

ZOR-out (ω ) = MK+1→OR-out (1) 1 − ∏ Mk→OR-out (0)
k =1

!

K +1

+ ∏ Mk→OR-out (0). (5.60)
k =1

where Mi→OR−out (1) = mi→OR−out (1 + mi→OR−out )−1 , by definition of mi→OR−out . The ratioof marginals is:µi (1; ω )µi (0; ω )

=






 mi→OR-out

1
1 − (1 − m −K +1→OR-out ) mK+1→OR-out ×KK∏ (1 + mk→OR-out )−1! −1
if 1 ≤ i ≤ K

k =1,k6=!
i∏ (1 + mk→OR-out ) − 1

if i = K + 1.

k =1

(5.61)

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

86

This leads to the following message updates:

mOR-out→i =1
1 − (1 − m −K +1→OR-out )K∏ (1 + mk→OR-out )

! −1
−1

,

(5.62)

k =1,k 6=i

for i ≤ K, andK

mOR-out→K+1 = ∏ (1 + mk→OR-out ) − 1.

(5.63)

k =1

The message updates in Eqs. 5.62–5.63 have again an intuitive interpretation. In Eq. 5.62, if
there is evidence that the output variable is likely to be active (large mK+1→OR-out ), then theexpression becomes close to the message passing in the OR factor. On the contrary, if it islikely that yK+1 is zero (mK+1→OR-out close to zero), then the message sent from the factor
to each variable i ≤ K is going to be close to zero as well. Eq. 5.63 is also very intuitive:
if at least one of the messages mk→OR-out is large, then mOR-out→K+1 will also become large,
pushing the output variable to be active; only if all the input variables are likely to be inactive(every mk→OR-out close to zero) will the message mOR-out→K+1 be a small value.
Like in the previous factors, the entropy can be computed via:
K +1

HOR-out (bOR-out (.)) = log ZOR-out (ω ) − ∑ µi (1; ω ) log mi→OR-out
i =1

K +1

+ ∑ log(1 + mi→OR-out ).

(5.64)

i =1

Again, all the relevant quantities (factor-to-variable messages, log-partition function, marginals
and factor entropy) can be computed in linear time.

5.4.2

Max-Product Messages

We now turn to the max-product variant of the BP algorithm, for which we derive results
similar to those obtained in the sum-product variant. In max-product BP, the factor-tovariable messages take the form (cf. Eq. 4.13):!

Mα→i (yi ) ∝ max
y α ∼ yiψα (yα ) ∏ M j→α (y j ) ;(5.65)

j 6 =ithat is, they are analogous to the sum-product case, but where the summation is replacedwith a maximization. The next proposition is the analogous of Proposition 5.4 for the maxproduct variant of the BP algorithm, and provides a way of computing messages given theability of computing max-marginals.6
Proposition 5.5 Let Gα be the α-subgraph of G , and consider a probability distribution factoring
6 Most of the results in this section may be obtained from the ones in the previous section by taking the “zerotemperature limit” of the sum-product case.

5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS
according to Gα :Pω (yα ) ∝ Peω (yα ) := ψα (yα )∏ exp(ωi (yi )).

87(5.66)i ∈N ( α )Above, we have denoted by Peω (yα ) the unnormalized distribution. Let
Pα∗ (ω ) := max Peω (yα )yα(5.67)

be the “mode” of this unnormalized distribution, and let
νi (yi ; ω ) := max Peω (yα ),y α ∼ yi(5.68)

for every i ∈ N(α) and yi ∈ Yi , denote the variable max-marginals.
Now consider an execution of the max-product (loopy) BP algorithm run on the original graph G ,and define ω as given by the variable-to-factor messages:
ωi (yi ) := log Mi→α (yi ).

(5.69)

Then we have the following:

1. At any point of the algorithm, we have the following expression for the factor beliefs (Eq. 4.10):bα (yα ) ∝ Peω (yα );2. Upon convergence, the variable beliefs (Eq. 4.9) satisfy bi (yi ) ∝ νi (yi ; ω );3. At any point of the algorithm, the factor-to-variable messages can be computed as:
Mα → i ( y i ) ∝νi (yi ; ω )= νi (yi ; ω ) exp(−ωi (yi )).
Mi → α ( y i )(5.70)

Proof. Analogous to the proof of Proposition 5.4.

Similarly to the sum-product case, Proposition 5.5 is of great significance as it statesthat all the necessary computations in the max-product (loopy) BP algorithm which involve
factor α can be performed by invoking a black box routine that computes the MAP and themax-marginals for the α-subgraph.
We next revisit the hard constraint factors defined in Section 5.3 and, in each case, obtainclosed-form expressions for the factor-to-variable max-product message ratios in terms of
their variable-to-factor counterparts (see Eq. 5.51). We use throughout the compact notation[K ] to denote the set of integers between 1 and K, i.e., [K ] := {1, . . . , K }. Again, we justsummarize the results below, leaving the details of the derivations to Appendix C.2.

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION88

The One-hot XOR Factor. In Appendix C.2, we obtained expressions for the mode andmax-marginals associated with the one-hot XOR factor:
K

∗
PXOR
(ω ) = max mi→XOR × ∏(1 + mi→XOR )−1
i ∈[K ]!
,

(5.71)i =1

K

νi (1; ω ) = mi→XOR ∏ (1 + mk→XOR )−1 ,(5.72)k =1

K

νi (0; ω ) =m j→XOR × ∏ (1 + mi→XOR )−1

maxj∈[K ]\{i }!
.

(5.73)

k =1

From Proposition 5.5, we have that the message updates are given by: −1mXOR→i =max mk→XOR.

k 6 =i(5.74)The message updates in Eq. 5.74 have an intuitive interpretation in terms of competition
between variables, just like in the sum-product case (cf. Eq. 5.55). The only difference is thatthe summation in Eq. 5.55 became a maximization in Eq. 5.74.

The OR Factor. In Appendix C.2, we derived expressions for the mode and ratio of maxmarginals associated with the OR factor:
∗
POR
(ω )

νi (1; ω )νi (0; ω )

!

K



M
(1)
=
Mk→OR (yk ) × min 1, max k→OR∏ maxykk∈[K ] Mk→OR (0)
k =1


1
= mi→OR × max 1, min m−
k →OR .

,(5.75)(5.76)k ∈[K ]\{i }As a consequence, we obtain the following simple expression for the max-product message
updates, invoking Proposition 5.5:mOR→i = max

1
1, min m−
k→OR
k 6 =i.

(5.77)The mode in the OR-with-output factor is

The OR-with-output Factor.
(∗
POR-out(ω ) = max

K +1∏ Mk→OR-out (0),MK+1→OR-out (1)×k =1
K

Mk→OR-out (yk )
∏ max
y

k =1

k

!



M
(1)
× min 1, max k→OR-out
k ∈[K ] Mk →OR-out (0)).

(5.78)

5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS89

We now turn to the ratio of max-marginals. For i ≤ K, we have:νi (1; ω )νi (0; ω )= mi→OR-out × min



K



 mK+1→OR-out × ∏ max{1, mk→OR-out }, 





k =1
k 6 =i






−
1

 max 1, min mk→OR-out.

(5.79)

k ∈[K ]\{i }For i = K + 1, we have:νK+1 (1; ω )= mK+1→OR-out ×νK+1 (0; ω )K∏ max{1, mk→OR-out }

!





× min 1, max mk→OR-out(5.80)k ∈[K ]

k =1

This leads to the following simple expression for the max-product message updates:

mOR-out→i = min







K



1
mK+1→OR-out ∏ max{1, mk→OR-out }, max 1, min m−
k →OR-outk ∈[K ]\{i }k =1
k 6 =i(5.81)

for i ≤ K, andK

mOR-out→K+1 =∏ max{1, mk→OR-out }!

× min 1, max mk→OR-out .

k =1

5.4.3


k ∈[K ]

(5.82)

Negations

As mentioned in Section 5.3.3, the ability of endowing the factors seen above with negatedvariables adds a lot of expressive power, enabling us to construct formulas in first order logic
by using the factors XOR, OR and OR-with-output as components.
To use factors with negations, we need to be able to compute with them. We are going to see that all quantities of interest in the sum-product and max-product algorithms—
messages, beliefs, entropies, and modes—can easily be computed when some of the variables
are negated, by invoking the same black box procedures that compute the log-partition function and the marginals (in the sum-product case) or the mode and the max-marginals (in the
max-product case), albeit with some pre- and post-processing. This is a consequence as thefollowing proposition, which holds for any factor (soft or hard):
Proposition 5.6 Let α be a factor linked to K variables, and let β be a factor obtained from α bymodifying the potential function as follows:ψβ (y1 , . . . , yK ) := ψα (σ1 (y1 ), . . . , σK (yK )),(5.83)

where each σi : Yi → Yi is a permutation function that relabels the variables. Let θ = (θi (.))iK=1 be avector of variable log-potentials, and consider the scrambled vector
θi0 (yi ) = θi (σi−1 (yi )),(5.84)

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

90

where σi−1 : Yi → Yi is the inverse permutation of σi .
1. Denote by Zα (θ) (resp. Zβ (θ)) the value of the partition function using factor potential ψα(resp. ψβ ) and variable log-potentials θ. Then we have the following:Zβ (θ) = Zα (θ0 ).

(5.85)

2. Furthermore, denote by µi;α (yi ; θ) (resp. µi;β (yi ; θ)) the marginal probability of the event Yi =yi for factor α (resp. β) with variable log-potentials θ. Then we have:µi;β (yi ; θ) = µi;α (σi (yi ); θ0 ).

(5.86)

3. Denote by Pα∗ (θ) (resp. Pβ∗ (θ)) the “mode” of the unnormalized distribution using factor potential ψα (resp. ψβ ) and variable log-potentials θ. Then we have the following:Pβ∗ (θ) = Pα∗ (θ0 ).

(5.87)

4. Furthermore, denote by νi;α (yi ; θ) (resp. νi;β (yi ; θ)) the max-marginal associated with the event
Yi = yi for factor α (resp. β) with variable log-potentials θ. Then we have:
νi;β (yi ; θ) = νi;α (σi (yi ); θ0 ).

(5.88)

Proof. Items 1 and 2 are immediate from the definition of partition function and marginals:∑

Zβ (θ) =y1 ,...,yKK

ψβ (y1 , . . . , yK ) ∏ exp(θi (yi ))
i =1

K

=
=

∑ ψα (σ1 (y1 ), . . . , σK (yK )) ∏ exp(θi0 (σi (yi )))y1 ,...,yKZα (θ0 ),i =1(5.89)

andµi;β (yi ; θ) =∑y ,...,y
1

K

i −1

ψβ (y1 , . . . , yi , . . . , yK ) ∏ exp(θk (yk ))k =1

yi+1 ,...,yK=∑
y ,...,y
1

K

i −1

ψα (σ1 (y1 ), . . . , σi (yi ), . . . , σK (yK )) ∏ exp(θ0k (σk (yk )))yi+1 ,...,yK

k =1= µi;α (σi (yi ); θ0 ).

(5.90)

Items 3 and 4 and analagous, replacing summations with maximizations.
The next corollary follows from applying Proposition 5.6 to a hard constraint factor αwith binary variables, for which the only permutations are the identity and the negation. For
convenience, we state it for the case of a reduced parametrization (see Section 5.2, Eq. 5.7),

5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS
in the form
P(y) ∝ exp∑ si yi91!
Q α ( y ),(5.91)i ∈V

where each yi ∈ {0, 1}.
Corollary 5.7 Let α be a hard constraint factor linked to K variables, with acceptance set Sα , and let
β be a factor obtained from α by negating the ith input variable:noSβ := y ∈ {0, 1}K | (y1 , . . . , ¬yi , . . . , yK ) ∈ Sα .

(5.92)

Let s := (sk )kK=1 be a vector of variable log-potentials in the reduced parametrization.

1. Denote by Zα (s) := ∑y∈Sα exp ∑kK=1 sk yk and Zβ (s) := ∑y∈Sβ exp ∑kK=1 sk yk the valueof the partition function in each of the factors, when parameterized by s. Then we have thefollowing:Zβ (s1 , . . . , si , . . . , sK ) = exp(si ) × Zα (s1 , . . . , −si , . . . , sK ).
(5.93)
2. Furthermore, denote by zk;α (s) (resp. zk;β (s)) the marginal probability of the event Yk = 1 forfactor α (resp. β) with variable log-potentials s. Then we have:(zk;β (s1 , . . . , si , . . . , sK ) =1 − zi;α (s1 , . . . , −si , . . . , sK ) if k = izk;α (s1 , . . . , −si , . . . , sK )if k 6= i.

(5.94)


3. Denote by Pα∗ (s) := maxy∈Sα exp ∑kK=1 sk yk and Pβ∗ (s) := maxy∈Sβ exp ∑kK=1 sk yk thevalue of the unnormalized mode in each of the factors, when parameterized by s. Then we havethe following:Pβ∗ (s1 , . . . , si , . . . , sK ) = exp(si ) × Pα∗ (s1 , . . . , −si , . . . , sK ).

(5.95)

4. Furthermore, denote by νk;α (l; s) (resp. νk;β (l; s)) the marginal probability of the event Yk = l,for factor α (resp. β) with variable log-potentials s. Then we have:(νl;β (s1 , . . . , si , . . . , sK ) =νi;α (1 − l; s1 , . . . , −si , . . . , sK ) if k = i
νk;α (l; s1 , . . . , −si , . . . , sK )if k 6= i.

(5.96)A direct consequence of Corollary 5.7 is that, for both the sum-product and max-productcases, the factor-to-variable messages at factor β can be computed as follows:(m β → k ( ω1 , . . . , ω K ) =1/mα→i (ω10 , . . . , ωK0 ) if k = iif k 6= i.
mα→k (ω10 , . . . , ωK0 )(5.97)

where ωk := mk→ β for k ∈ {1, . . . , K }, and ω 0 is defined as ωk0 = ωk for k 6= i, and ωi0 = 1/ωi .
By invoking this corollary several times, we end up with a procedure for any number ofnegated variables. There are three steps:

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

92

1. A preprocessing step in which one inverts the incoming messages mk→ β that correspond to the negated variables;
2. The main step in which one computes the outgoing messages mα→k using a black box
for the factor α without negations;
3. A postprocessing step in which one inverts back the outgoing messages that correspond to the negated variables, yielding m β→k .
Example 5.2 (AND-with-output.) The AND-with-output factor emulates the logical expression
V
yK+1 := kK=1 yk . This can be represented as a OR-with-output factor where all variables are negatedW(since we have ¬yK+1 := kK=1 ¬yk by de Morgan’s rules). Hence the sum-product messages can beobtained from the ones in Eqs. 5.62–5.63, by invoking Corollary 5.7:
K

mAND−out→i = 1 − (1 − mK+1→AND−out )∏ (1 + m−k→1 AND−out )−1 ,(5.98)

k =1,k 6=i

for i ≤ K, andK

mAND−out→K+1 =∏ (1 + m−k→1 AND−out ) − 1

! −1
.,

(5.99)

k =1

and similarly for the max-product messages.
Algorithms 6 and 7 summarize the procedure of computing sum-product and maxproduct messages, respectively, in a hard constraint factor linked to binary variables, somepossibly negated. In both cases, two procedures are called as subroutines: for the sumproduct case, the procedures are ComputePartitionFunction, a black-box that computes
Z from a vector of log-potentials, and ComputeMarginals, another black-box that computes the marginal vector z from the same vector of log-potentials. For the max-productcase, the oracles are ComputeMode, a black-box that computes the mode P∗ from a vectorof log-potentials, and ComputeMaxMarginals, another black-box that computes the vectorof max-marginals ν from the same vector of log-potentials.7
If one of the goals is to approximate the value of global partition function Z (θ), then we
can invoke the Eq. 5.21 in Proposition 5.3 and compute it from the Bethe approximation of
the entropy. The latter can be computed upon convergence of the loopy BP algorithm, fromthe variable entropies and the factor entropies (Eq. 4.45). Note that Algorithm 6 provides thefactor entropy as output. We will exploit this fact in Chapter 8, where we introduce a newlearning algorithm that needs to evaluate the partition function, in addition to computing its
gradient.

5.4.4

Generalized Message-Passing Algorithms

In the last section, we provided a recipe for running sum-product and max-product loopy
BP in constrained graphs (Algorithms 6–7). That recipe can be extended in a straighforwardmanner to other variants of message-passing algorithms, such as “fractional” BP algorithms
7 Note that, in line 19 of both algorithms, and since we are dealing with constrained factors, there is a chancethat the denominators 1 − zi (or νi (0)) vanish. In that case, the message ratio m β→i should be interpreted as
being +∞—this means Mβ→i = 1 and Mβ→i = 0.

5.4. COMPUTING MESSAGES AND SOLVING LOCAL SUBPROBLEMS

93

Algorithm 6 Computing Sum-Product Messages in a Binary Hard Factor
1: input:• factor β, which is made of α by negating some variables,• set of negated variables A,• incoming messages mi→ β .
2: Form a vector of log-potentials s = (si )iK=1 , with si := log mi→ β
3: Initialize bias b = 0
4: for i = 1 to K do
5:
if i ∈ A then6:
Set si0 ← −si (flip log-potential)
7:
Increment bias b ← b + si
8:
else
9:
Set si0 ← si
10:
end if
11: end for
12:
13: Compute Zα ← ComputePartitionFunction(s0 ; α)
14: Compute z ← ComputeMarginals(s0 ; α)15: for i = 1 to K do
16:
if i ∈ A then17:zi ← 1 − zi
18:
end if19:
Compute outgoing message m β→i ← 1−ziz exp(−si )i20: end for21:
22: Compute partition function Zβ ← Zα × exp(b)23: Compute entropy Hβ ← log Zβ − s · z
24: output:• outgoing messages m β→i ,• current beliefs bi (1) := zi ,• factor entropy Hβ .

(Wiegerinck and Heskes, 2003; Weiss et al., 2007; Hazan and Shashua, 2010), tree-reweighted
algorithms (Wainwright et al., 2005b,a; Kolmogorov, 2006), and block coordinate descentalgorithms (Werner, 2007; Globerson and Jaakkola, 2008), some of which discussed in Sections 4.5–4.6. Those generalized message-passing algorithms pass messages of the form
1/ραρi,α −ρi
M α → i ( y i ) ∝  Mi →α ( yi )M
y α ∼ yiρψαα (yα )∏j∈N(α),j6=i

ρ

j,α

M j→
α (y j )(5.100)where
denotes either max or ∑, and the exponents ρi , ρα , and ρi,α are specific to eachalgorithm (in standard BP, they are all set to 1). By renormalizing the messages, it is simple
to express those messages as a function of the marginals (in the sum-product case) or the
max-marginals (in the max-product case).
LCHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

94

Algorithm 7 Computing Max-Product Messages in a Binary Hard Factor
1: input:• factor β, which is made of α by negating some variables,• set of negated variables A,• incoming messages mi→ β .
2: Form a vector of log-potentials s = (si )iK=1 , with si := log mi→ β
3: Initialize bias b = 0
4: for i = 1 to K do
5:
if i ∈ A then6:
Set si0 ← −si (flip log-potential)
7:
Increment bias b ← b + si
8:
else
9:
Set si0 ← si
10:
end if
11: end for
12:
13: Compute Pα∗ ← ComputeMode(s0 ; α)
14: Compute ν ← ComputeMaxMarginals(s0 ; α)15: for i = 1 to K do
16:
if i ∈ A then17:
Swap νi (0) and νi (1)18:
end ifν (1)19:Compute outgoing message m β→i ← νi (0) exp(−si )20: end for21:
22: Compute mode Pβ∗ ← Pα∗ × exp(b)23: output:i• outgoing messages m β→i ,• current beliefs bi := νi ,• factor mode Pβ∗ .

5.4.5

Local Subproblems in Dual Decomposition

In Section 4.6.2, we have described another technique for LP-MAP inference, called dual
decomposition, with uses the projected subgradient algorithm (Algorithm 5). That algorithm has a similar flavor as message passing algorithms. One can regard the local MAP
estimates (the µ-variables) and the Lagrange multipliers (the λ-variables) as messages thatare exchanged between variables and factors. The analogue of the problem of computing
factor-to-variable messages is that of solving the local MAP subproblems, i.e., the µ-updates
in Algorithm 5. As seen in Section 4.6.2, all that is necessary is a subroutine ComputeMAP
that computes the MAP assignment of the α-subgraph induced by each factor α. In Section 5.4.2, we have already provided expressions for computing the modes of factors XOR,
OR, and OR-with-output; and Corollary 5.7 allows extending those expressions for otherfactors encoding first-order logic expressions.

5.5. APPROXIMATE INFERENCE IN CONSTRAINED FACTOR GRAPHS

95

5.5 Approximate Inference in Constrained Factor Graphs
We now provide a global perspective on how approximate inference works with the machinery described above for computing messages and solving local subproblems. To have a sense
about the quality of the approximations in graphs with cycles, we consider problems thatcan be formulated as inference in constrained graphs, but which are simple enough so thatexact solutions can be derived analytically and compared with the approximations. This will
give insight about the strengths and limitations of the approximate methods.
We start by showing how any arbitrary factor graph with multi-valued variables canbe binarized, a process that has the side effect of adding hard constraints and cycles. We
then show that this transformation preserves the LP-MAP approximation (used in MAP
inference), but not the Bethe approximation (used in marginal inference). The two main
points that will be illustrated are the following:• For a wide range of problems, one can find a representation as a binary constrained
factor graph that does not affect MAP inference (a positive result);• Even for very simple problems, marginal inference and entropy computation can bestrongly affected by using that representation (a negative result).

5.5.1

Binarization of a Factor Graph

So far, we have focused on binary constrained factor graphs. However, there are many
problems for which it is natural to use multi-valued random variables, for which |Yi | > 2.
How to extend our techniques to perform inference in such multi-valued factor graphs?
One possible way is to binarize the graph, as we describe next.
Let G = (V ∪ F, E0 ) be a unconstrained factor graph, where the associated random variables Y1 , . . . , Y|V| are multi-valued (i.e., |Yi | ≥ 2, for each i ∈ V). We will derive a binary
constrained factor graph G 0 = (V0 ∪ F 0 ∪ H0 , E0 ) which is equivalent to G , in the sense thatany probability distribution represented by G can be represented by G 0 and vice-versa.
Recall the definition of the local polytope of G in Eq. 4.40:
∀i ∈ V∑yi ∈Yi µi (yi ) = 1,
R
. (5.101)
LOCAL(G ) = µ ∈ Rµi (yi ) = ∑yα ∼yi µα (yα ), ∀i ∈ V, yi ∈ Yi , α ∈ N(i )µα (yα ) ≥ 0,∀α ∈ F, yα ∈ Yα
A close look reveals that the constraints in Eq. 5.101 are essentially the same as those thatdefine the marginal polytopes of the XOR factor (Eq. 5.25) and the XOR-with-output factor(Eq. 5.36); therefore, an equivalent set of constraints can be obtained by introducing binary
variables for each configuration of Yi and Yα and linking them to XOR and XOR-with-output
factors. This leads to the following procedure for constructing G 0 :1. For any i ∈ V and yi ∈ Yi , introduce a binary variable Ui,yi and add the corresponding
variable node to G 0 . Set the log-potential function on this variable to 0 on Ui,yi = 0 andto θi (yi ) on Ui,yi = 1.
2. For any α ∈ F and yα ∈ Yα , introduce a binary variable Uα,yα and add the corresponding variable node to G 0 . Set the log-potential function on this variable to 0 on Uα,yα = 0
and to θα (yα ) on Uα,yα = 1.

96

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

XOR-OUTXOR

XOR

Figure 5.6: Left: A simple factor graph with multi-valued variables. Right: A transformationof that graph into a binary constrained factor graph, yielding 11 variables and 7 factors.

3. For any i ∈ V, add a XOR factor to G 0 connecting the variables {Ui,yi | yi ∈ Yi }. From
Eq. 5.25, we have that the corresponding marginal polytope imposes the constraints:∑ µi (yi ) = 1,

y i ∈ Yiµi (yi ) ≥ 0, ∀yi ∈ Yi ,(5.102)the first of which corresponding to the original normalization constraints in LOCAL(G ).
4. For any α ∈ F, i ∈ N(α), and yi ∈ Yi , add a XOR-with-output factor to G 0 connecting
the input variables {Uα,yα | yα ∼ yi } to the output variable Ui,yi . From Eq. 5.36, wehave that the marginal polytope associated with this factor imposes the constraints:µi ( yi ) =∑ µ α ( y α ),y α ∼ yiµα (yα ) ≥ 0, ∀yα ∈ Yα ,(5.103)the first of which corresponding to the original marginalization constraints in LOCAL(G ),and the second implying the non-negativity constraints.
Figure 5.6 illustrates the above procedure when the original graph is a simple pairwise MRF
with only two variables.
Proposition 5.8 For any choice of log-potentials θ, the probability distributions induced by G and
G 0 are the same. Moreover, each iteration of the loopy BP algorithm (as well as other message-passing
and dual decomposition algorithms) has the same asymptotic cost in the two graphs.
Proof. The XOR factors guarantee that, for every i ∈ V, exactly one of the variables Ui,yi
is active, and the XOR-with-output factors guarantee that the variables Uα,yα and Ui,yi areconsistent; it is straightforward to show that the set of admissible outputs in G 0 is in oneto-one correspondence with Y1 × . . . × Y|V| , which is the output set of G . In addition, theway we have set the log-potentials makes it clear that any configuration in G 0 will have the5.5. APPROXIMATE INFERENCE IN CONSTRAINED FACTOR GRAPHS

97

same score as the corresponding configuration in G . This proves the first claim. Let us now
compare the runtimes of one round of message passing in both graphs. The original factor
graph G has |V| variables and |F | factors. For each factor α, a total of deg(α) messages
Mα→i (.) need to be passed, each with a computation cost of O(|Yα |). As for the variable-tofactor messages, the total computation cost is ∑i∈V deg(i ) × |Yi |. Hence the total runtime ofa round of message passing isO∑ deg(i) × |Yi | + ∑ deg(α) × |Yα |

i ∈V

!
.

(5.104)α ∈FThe binary constrained factor graph G 0 has ∑i |Yi | + ∑α |Yα | variables, |V| XOR factors, and∑α ∑i∈N(α) |Yi | XOR-with-output factors. The total number of messages is ∑i |Yi | (for the XOR
factors) and ∑α deg(α) × |Yα | + ∑α ∑i∈N(α) |Yi | (for the XOR-with-output ones). Each of thesemessages has a constant computation cost (since the cost per XOR or XOR-with-output islinear in the factor degree). As for the variable-to-factor messages, the total computation costis ∑i∈V (1 + deg(i )) × |Yi | + ∑α∈F deg(α) × |Yα |. We then have a total runtime per iterationof message passing of O (2 ∑i∈V (1 + deg(i )) × |Yi | + 2 ∑α∈F deg(α) × |Yα |). Hence, although
there is some overhead, the two graphs yield the same asymptotic cost per iteration.

5.5.2

Local Polytope Approximation and LP-MAP Inference

Let us turn to the LP-MAP problem in the binarized graph. A consequence of Proposition 5.8
is that the marginal polytopes of G and G 0 are the same (up to a lifting operation). The next
proposition goes farther by establishing that this also happens for the local polytopes:
Proposition 5.9 The local polytopes LOCAL(G ) and LOCAL(G 0 ) are the same (up to lifting). Thishas the following chain of implications:
1. The LP-MAP problems in LOCAL(G ) and LOCAL(G 0 ) have the same solution.
2. If G does not have cycles, then the local polytope approximation LOCAL(G 0 ) is tight (even if
G 0 has cycles!) and the LP-MAP problem on G 0 is guaranteed to have integer solutions (i.e., ityields the true MAP).
3. The projected subgradient method for dual decomposition (Section 4.6.2) leads to the same
solution, when run on LOCAL(G ) or LOCAL(G 0 ).
4. The block coordinate message passing algorithms discussed in Section 4.6.1 (MPLP, max-sum
diffusion, and TRW-S), when converging and satisfying weak agreement conditions both on
LOCAL(G ) or LOCAL(G 0 ), lead to the same solution.
Proof. The main claim follows directly from the construction of G 0 . The following claims are
immediate. We used the fact that the projected subgradient method for dual decomposition
converges to the solution of LP-MAP, and that MPLP, max-sum diffusion and TRW-S, whenconverging and satisfying weak agreement conditions, also yield that solution.
Note that the equivalence established in Proposition 5.9 would not be trivial, had we notexamined the local polytopes of both graphs. In general, the constrained graph G 0 has cycles,
even when G is a very simple graph without cycles, such as an edge connecting two variables

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION98(as in Figure 5.6). Those cycles, however, turn out not to affect the polytope approximationand the MAP inference algorithms mentioned above.
Example 5.3 (Context-free parsing.) We are going to see how context-free grammars can be binarized. Recall the definition of a context-free grammar (Λ, Σ, P, S), in Definition 2.1. Suppose wewant to parse a sentence with L words using this grammar. In Eqs. 2.10–2.11, we considered sets ofparts for anchored constituents and anchored production rules:(
Rs : =(
Rp : =h X, Y, Z, i, j, kih X, i, ji

X ∈ Λ,
1≤i≤j≤Lh X → YZ i ∈ P,
1≤i≤k<j≤L)([h X, i i),(5.105)h X → xi i ∈ P,
1≤i≤L

)
.

(5.106)

Taskar et al. (2004b) has shown that the marginal polytope associated with this context-free grammar
and this sentence is defined by the following set of equations:8
Non-negative marginals:

zr (yr ) ≥ 0,

Normalization:

zhS,1,Li = 1,

Inside constraints:∀r ∈ Rs ∪ Rp ,zhX,i,ji = ∑ zhX,Y,Z,i,k,ji ,(5.107)(5.108)∀h X, i, ji ∈ Rs ,(5.109)

Y,Z,k

Outside constraints:

zhX,i,ji = ∑ zhY,X,Z,i,j,ki + ∑ zhY,Z,X,k,i−1,ji ,Y,Z,k∀h X, i, ji ∈ Rs ,Y,Z,k(5.110)where the sums above are over the sets of non-terminals and indexes that yield valid productions
according to the grammar. Essentially, these constraints express the inside-outside equations in the
dynamic program for context-free parsing. A close look into Eqs. 5.107–5.110 reveals that all these
constraints can be expressed by XOR and XOR-with-output factors, in a binary constrained factor
graph G whose variables are the possible constituents and production rules. There are two kinds offactors: inside factors, which express the constraints in Eq. 5.109, and outside factors, which express the constraints in Eq. 5.110. These are drawn in Figure 5.7. By construction, the local polytopeof G coincides with the marginal polytope associated with the grammar—therefore, the solutions of the
LP-MAP inference problem are integer, and correspond to valid parse trees. In other words, LP-MAP
inference on this graph is exact, and equivalent to context free parsing.9 Since there are no fractionalvertices, we have from Proposition 5.2 we conclude that LOCAL(G ) = MARG(G ). This raises thequestion of whether G has cycles.
It turns out that G has cycles even for very simple instances: the simplest counter-example isfor a three-word sentence and a complete grammar with a single non-terminal (i.e., a grammar forbracketing)—see Figure 5.7 for an illustration of this example. The existence of cycles motivated someauthors to consider a broader representational formalism, case-factor diagrams (McAllester et al.,
8 Actually, in Taskar et al. (2004b), the marginal polytope was presented without proof. A formal proof has

been derived later by Michael Collins (personal communication).
9 Note that even though LP-MAP inference algorithms run on this graph have guarantees of getting the most
likely parse, their cost per iteration is not better than the CKY algorithm. However, this representation ofphrase-based parsing may be useful if used as a starting point to add more global features and constraints into
the model. In Chapter 7 we use a similar rationale for dependency parsing.

5.5. APPROXIMATE INFERENCE IN CONSTRAINED FACTOR GRAPHS

(12)3

(1)23

((1)(2))3((12)(3))

1(2)3

(123)

12(3)

((1)(23))

99

1(23)

1((2)(3))

Figure 5.7: Representation of context-free parsing as a binary constrained factor graph with
XOR and XOR-with-output factors, for a sentence with three words, and a bracketing grammar. In red, a cycle in this graph, of length 8.

2008). Our example shows that cycles are not a major reason for concern if we are only interested in
MAP inference.

5.5.3

Bethe Entropy Approximation and Marginal Inference

The previous section established a “positive” result: the binarization procedure of Section 5.5.1 preserves the local polytope and hence the LP-MAP inference problem. We now
turn our attention to marginal inference. Recall the connection described in Section 4.5 between sum-product loopy BP and the Bethe entropy approximation HBethe . One could be led
to conjecture that the Bethe approximation of the entropy is also preserved by the binarization procedure. Unfortunately, this does not hold even in very simple graphs, as indicatedby the following counter-example.
Example 5.4 Let G be a very simple graph comprised of one factor linked to two variables Y1 and Y2 ,each taking values in Y1 = Y2 = {1, . . . , L}. The true entropy is
H (µ) = − ∑ µ12 (y1 , y2 ) log(µ12 (y1 , y2 )).

(5.111)y1 ,y2

Consider now the binarization of G , which we denote by G 0 , and let us compute its Bethe entropy
approximation using Eq. 4.45. Each of the 2L variable nodes associated with the variable-value pairs
in the original graph has a local entropy given by:
Hi,yi (µi (yi )) = −µi (yi ) log(µi (yi )) − (1 − µi (yi )) log(1 − µi (yi )),(5.112)for i ∈ {1, 2}; each of these variables participates in two factors, hence these entropies have counting
number 1 − 2 = −1. Each of the L2 variable nodes associated with the edge-values in the original

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

100

graph has a local entropy given by:
H12,y12 (µ12 (y1 , y2 )) = −µ12 (y1 , y2 ) log(µ12 (y1 , y2 )) − (1 − µ12 (y1 , y2 )) log(1 − µ12 (y1 , y2 ));(5.113)since each of these variables participates in two factors, their counting number is also 1 − 2 = −1.
The contribution of the variable nodes to the Bethe approximation is thus∑ ∑ (µi (yi ) log(µi (yi )) + (1 − µi (yi )) log(1 − µi (yi )))i ∈{1,2} yi+ ∑ (µ12 (y1 , y2 ) log(µ12 (y1 , y2 )) + (1 − µ12 (y1 , y2 )) log(1 − µ12 (y1 , y2 ))) .

(5.114)

y1 ,y2

As for the factors, we have that each of the two XOR factors connected with the variable-value pairs
in the original graph has an entropy (cf. 5.56):
Hi (µi (.)) = − ∑ µi (yi ) log(µi (yi ))for i ∈ {1, 2};(5.115)yi

as for the 2L XOR-with-output factors connecting the edge-values with a variable-value pair, they
have entropies (cf. 5.56, and Corollary 5.7):
H1,y1 (µ1 (y1 ), µ12 (y1 , .)) = − ∑ µ12 (y1 , y2 ) log(µ12 (y1 , y2 )) − (1 − µ1 (y1 )) log(1 − µ1 (y1 )),y2

H2,y2 (µ2 (y2 ), µ12 (., y2 )) = − ∑ µ12 (y1 , y2 ) log(µ12 (y1 , y2 )) − (1 − µ2 (y2 )) log(1 − µ2 (y2 )).
y1Hence the total factor contribution is:−∑ ∑ µi (yi ) log(µi (yi )) − 2 ∑ µ12 (y1 , y2 ) log(µ12 (y1 , y2 ))i ∈{1,2} yi

y1 ,y2−∑ ∑(1 − µi (yi )) log(1 − µi (yi )).

(5.116)i ∈{1,2} yi

Summing Eqs. 5.114 and 5.116 yields:
HBethe (µ) = − ∑ µ12 (y1 , y2 ) log(µ12 (y1 , y2 )) − ∑ (1 − µ12 (y1 , y2 )) log(1 − µ12 (y1 , y2 )),y1 ,y2

y1 ,y2(5.117)
which differs from the true entropy by the amount:
HBethe (µ) − H (µ) = − ∑ (1 − µ12 (y1 , y2 )) log(1 − µ12 (y1 , y2 )).

(5.118)y1 ,y2In Martins et al. (2010f, extended version, Appendix B), we show another example in which
the original graph G is free of cycles—which guarantees that marginal inference is exact
with the sum-product algorithm—whereas the same algorithm on the binarized graph G 0
fails miserably.

5.6. FUTURE WORK

101

5.6 Future Work
There are many possible avenues for future research in constrained structured prediction.
One of them is to extend this formalism for broader or related models, such as case-factor
diagrams (McAllester et al., 2008), AND/OR search spaces (Dechter and Mateescu, 2007), andsum-product networks (Poon and Domingos, 2011). Case-factor diagrams, for example, area framework that subsumes pairwise MRFs and probabilistic context-free grammars. Not
much is known about their geometry, and it is not known if the duality and variational
perspectives of factor graphs can be extended to this broader class of models. Even though
the results that we have established in this chapter suggest that binary constrained factor
graphs are already very expressive and can be used for MAP inference with probabilistic
context-free grammars, no analogue result exists for marginal inference. It is conceivablethat the geometric properties of case-factor diagrams can shed some light on the problem,and contribute toward better entropy approximations. We plan to address this in futurework.
In fact, obtaining good entropy approximations for constrained graphical models is an
important open problem that deserves further attention. There has been a lot of research forthe unconstrained case (Wiegerinck and Heskes, 2003; Weiss et al., 2007; Hazan and Shashua,2010). Of particular interest are convex approximations and upper bounds of the partition
function, such as the ones underlying tree-reweighted BP (Wainwright et al., 2005b).
Interestingly, we can draw a connection between entropy approximations and the posterior regularization framework recently proposed by Ganchev et al. (2010), which involves
posterior expectation constraints. Although the rationale is totally different from ours—theidea there is to modify the Expectation-Maximization (EM) algorithm by introducing priorknowledge in the form of constraints on the posterior marginals—their constrained E-stepcan be rewritten in the following form:
maximize w> F( x )µ + HPR (µ)
w.r.t. µ ∈ MARG(Gu )s.t. µ β ∈ MARG(G | β ), for each β ∈ H.

(5.119)Above, H is a set of hard constraint factors, where for each factor β ∈ H, the posterior
constraint associated with that factor is represented by the polytope MARG(G | β ); Gu is theoriginal unconstrained factor graph; and the entropic term in the objective, HPR (µ), regards
only the unconstrained part of the graph, ignoring the constraints. The global optimization problem can be seen as an approximate marginal inference problem, with an entropy
approximation of the form
HPR (µ) ≈ ∑ ci Hi (µi ) +i ∈V∑ cα Hα (µα ),(5.120)α ∈F ∪H

where the counting numbers are ci = 1 − |N(i ) \ H|, cα = 1 for α ∈ F, and c β = 0 for β ∈ H.
Note that this entropy approximation in distinct from Bethe’s approximation (cf. Eq. 4.45). It
would be interesting to see how the message-passing algorithms used in factor graphs canhelp solve the E-step in the posterior regularization framework, or vice-versa.
Another problem is that of developing efficient algorithms for computing messages. In

102

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

the scope of sum-product BP, we have seen that this problem is intimately related with the
problem of computing marginals and the partition function, which are instances of counting
problems. In some cases, efficient procedures exist for computing these quantities. We
have seen an example in Section 2.3, which involves inverting a matrix and computing adeterminant. Other instances exist in the literature for which fast Fourier transforms enable
to carry out this computation efficiently (Liao et al., 2005).
The connection established between constrained graphical models and LP-MAP inference, as well as the possible binarization of tractable graphical models, opens up the possibility of determining whether certain linear programs have integral solutions based on the
topology of the graph. This kind of criterion could be an alternative to other properties that
establish sufficient conditions for the integrality of LPs, such as the total unimodularity of
the constraint matrix (Schrijver, 2003), which is usually difficult to check, or the property
of total dual integrality (Martin et al., 1990), which applies to directed hypergraphs and isrelated to dynamic programming algorithms. This connection can be fruitful in providing
better conditions to characterize integral LPs: it is easy to constuct factor graphs without
cycles where the marginal polytope equations do not yield totally unimodular constraint
matrices, and yet admit integral solutions (since the graph is free of cycles). Some examples
are: a single OR-with-output factor with three input variables (and one output), and a single
unconstrained factor with three binary variables. Conversely, it is also simple to find examples of totally unimodular matrices that correspond to graphs with loops. Therefore, neitherof these characterizations subsumes the other.
On a more practical side, future work will also contemplate how the achievements described in this chapter can be integrated with current systems for constrained prediction,such as Markov logic networks. Current software implementations, e.g., Alchemy,10 include
alternative inference algorithms, such as lifted BP and sampling methods. Lifted methods
are orthogonal to the procedures described here for computing messages, and it is likely thatthe two can be used together to tackle hard constraint factors.

5.7 Conclusions
In this chapter, we provided a theoretical study of constrained factor graphs, a unified representational framework for constrained structured prediction. This framework is useful inconstrained conditional models and grounded Markov logic networks, with many potential
applications to NLP. Our results lead to a better theoretical understanding of these models.
We show how to handle arbitrary first-order logic constraints by introducing hard constraint factors that work as building blocks for expressing such constraints. We provided
a concise representation of the marginal polytopes associated with these factors, and we
derived closed-form expressions for computing messages, marginals, max-marginals, thepartition function, the factor entropies, and the mode. We have shown how all these quantities can be computed in linear time, contrasting with dense factors, for which the analogous
operations have exponential runtime. We also saw that all that is necessary for computing sum-product and max-product messages in arbitrary factors is an oracle that computes
marginals, max-marginals, and the partition function.
10 Available at http://alchemy.cs.washington.edu/.

5.7. CONCLUSIONS

103

In addition, we have shown how any factor graph can be binarized through a procedurethat preserves the marginal and local polytope of the original graph. If the original graphdoes not have cycles, then LP-MAP inference in the binarized graph is exact, even if cycles
are present in the latter. This reinforces the idea that the tractability of MAP inference holds
in much broader scenarios than just tree-structured graphs, corroborating previous results(Weiss and Freeman, 2001b,a; Bayati et al., 2005; Huang and Jebara, 2007). On the other
hand, the entropy approximations are not preserved by this binarization procedure, andthey can be poor in the binarized graph, even for very simple examples. This suggests thatapproximate marginal inference in constrained models may be a much more difficult task.
The results obtained in this chapter will be invoked for presenting contributions in later
chapters. For example, Chapter 6 will propose the AD3 algorithm, dealing with the hardconstraint factors introduced here. In Chapter 8, we will reuse the variational representationof the log-partition function, and discuss the impact of approximate inference in the learning
problem.

104

CHAPTER 5. CONSTRAINED STRUCTURED PREDICTION

Chapter 6

Alternating Directions Dual
Decomposition
In this chapter, we introduce a new LP-MAP inference algorithm called AD3 (for Alternating Directions Dual Decomposition). At its heart lies a method well-known in the optimization community called the “alternating direction method of multipliers” (ADMM),1 an augmented Lagrangian method which was first proposed in the 1970s (Glowinski and Marroco,
1975; Gabay and Mercier, 1976) and has seen a recent surge of interest in machine learningand signal processing (Afonso et al., 2010; Mota et al., 2010; Goldfarb et al., 2010; Boyd et al.,2011). We show how AD3 is able to solve the LP-MAP problem more efficiently than othermethods, by allying the effectiveness of ADMM with the modularity of dual decomposition
methods. Our main contributions in this chapter are:
• We derive the AD3 algorithm and establish its convergence properties, blending classical and newer results (Glowinski and Le Tallec, 1989; Eckstein and Bertsekas, 1992; He
and Yuan, 2011; Wang and Banerjee, 2012).
• We derive efficient procedures for projecting onto the marginal polytopes of the hardconstraint factors introduced in Section 5.3. This paves the way for using AD3 inproblems with declarative constraints. Up to a logarithmic term, the cost of projecting
is asymptotically the same as that of passing messages.
• We introduce a new active set method for solving the local quadratic problem forarbitrary factors. This method requires only an oracle that computes the local MAP(the same requirement as the projected subgradient method discussed in Section 4.6.2),
which allows plugging in combinatorial machinery.
• We show how AD3 can be wrapped into a branch-and-bound procedure to retrieve thetrue MAP, rendering the method exact.
The AD3 algorithm was originally introduced in Martins et al. (2010d, 2011a),2 wheresome of the convergence results were established, along with procedures for projecting on
the marginal polytopes of hard constraint factors. We include in this chapter a substantial
1 Also known by other names, such as “method of alternating directions” and “Douglas-Rachford splitting.”
2 The algorithm was called DD-ADMM in those papers.

105

106

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

amount of new material: the O(1/e) rate of convergence of AD3 , the active set method forgeneral factors, and the branch-and-bound procedure for obtaining the exact MAP.

6.1 Motivation and Related WorkIn the current chapter, we address the problem of LP-MAP inference, defined in Section 4.6.
There is a string of previous work concerning algorithms for solving that problem (Wainwright et al., 2005a; Kolmogorov, 2006; Werner, 2007; Globerson and Jaakkola, 2008; Ravikumar et al., 2010). Our focus is on dual decomposition methods, of which the projected subgradient algorithm of Komodakis et al. (2007); Komodakis and Paragios (2009), describedin Section 4.6.2, is a paradigmatic example. We first discuss the intrinsic limitations of thatmethod and suggest ways of overcoming them, pointing to the literature. This will pave theway for the introduction of our AD3 algorithm, which is the subject of the following sections.
A nice property of the projected subgradient algorithm is that it is guaranteed to converge
to the solution of the LP-MAP problem; this is a clear advantage over other message-passing
algorithms, including the coordinate descent algorithms discussed in Section 4.6.1, which
can get stuck at corners. However, this convergence can be painfully slow when the numberof overlapping components is large (cf. Figure 7.8 in the following chapter). This should not
be surprising: being a “consensus algorithm,” it attempts to reach a consensus among all
overlapping components; the larger this number, the harder it gets to achieve the desiredconsensus. This chapter addresses the following question:
Can we get faster consensus with a different dual decomposition algorithm?
To gain intuition about the problem, it is instructive to look at two things:
1. The objective function in the dual problem (Eq. 4.58). This function is non-smooth—this is why we need “subgradients” instead of “gradients.” It is well-known that nonsmooth optimization lacks some of the nice properties of its smooth counterpart. For
example, there is no guarantee of monotonic improvement in the objective after eachsubgradient update (see Bertsekas et al. 1999, p. 611). Convergence is ensured by usinga schedule that progressively decreases the stepsize, leading to slower convergence
rates (in general, O(1/e2 ) iterations are required for e-accuracy).
2. How the projected subgradient method promotes consensus. A close look at Algorithm 5 reveals that this is done solely by the Lagrange multipliers—in a economic
interpretation, these represent “price adjustments” that will lead to a reallocation of
resources. However, no “memory” exists about past allocations or adjustments, so
the workers never know how far they are from consensus before reallocating—it isconceivable that a smart use of these quantities could accelerate convergence.
Johnson et al. 2007, to obviate the first of these problems, proposed smoothing the objective function through an “entropic” perturbation—which essentially boils down to replacing
the max in Eq. 4.58 by a “soft-max.” As a result, all the local subproblems become marginalcomputations (rather than MAP), and can be solved with marginal inference algorithms suchas the sum-product algorithm. The amount of perturbation is controlled by a “temperatureparameter”—at low temperatures, the solution becomes sufficiently close from the one that

6.1. MOTIVATION AND RELATED WORK

107

would be obtained without perturbation.3 A related and asymptotically faster method was
proposed later by Jojic et al. (2010), who, in addition to smoothing the objective function in
Eq. 4.58, addresses the resulting smooth optimization problem with an accelerated gradientmethod invented by Nesterov (1983), which interleaves the gradient updates to account for
momentum.4 This approach guarantees an e-accurate solution after O(1/e) iterations, animprovement over the O(1/e2 ) bound of the subgradient algorithm.
However, the methods above come with some drawbacks:• To be accurate, they need to operate at near-zero temperatures—for example, theO(1/e) iteration bound of Jojic et al. (2010) requires setting the temperature to O(e).
In many problems, this may lead to numerical instabilities—this is so in problems inwhich marginal computation requires inverting a matrix which becomes ill-conditioned
at low temperatures (as in non-projective dependency parsing, cf. the matrix-tree formula in Eq. 2.19); and in the OR and OR-with-output factors defined in Section 5.3,
whose marginal computation requires evaluating the product of K large numbers(cf. 5.57). Typical strategies to deal with numerical overflow and underflow which
involve approximating and thresholding may affect the global convergence rate.
• The solution of the local subproblems is always dense. Some marginals can be low, butthey will never be zero; since this computation is embedded in an iterative algorithm,
thresholding them can also compromise convergence. This is in deep constrast withthe projected subgradient algorithm, for which the solution of each local subproblem
is a MAP estimate and hence always a single configuration. This property leads toimportant runtime savings, achieved by caching the local subproblems; those savings
are not readily available in the methods of Johnson et al. 2007 and Jojic et al. (2010).
In this chapter, we take a different approach that also yields a O(1/e) iteration bound anddoes not suffer from either of these two drawbacks. The key idea is to ally the simplicityof dual decomposition with the effectiveness of augmented Lagrangian methods, which have along-standing history in optimization (Hestenes, 1969; Powell, 1969; Glowinski and Marroco,
1975; Gabay and Mercier, 1976), and have recently been shown to be very competitive insome problems (Afonso et al., 2010; Goldfarb et al., 2010); see Boyd et al. 2011 for a recent
monograph on the subject. The result is AD3 , a new algorithm for LP-MAP inference.
Like the projected subgradient method, AD3 is an iterative “consensus algorithm,” alternating between a broadcast operation, where subproblems are distributed across local workers, and a gather operation, where the local solutions are assembled by a controller. The maindifference is that AD3 also broadcasts to the workers the current global solution, allowing themto regularize their subproblems toward that solution. This has the consequence of speeding up consensus, without sacrificing the modularity of dual decomposition. Additional
properties of AD3 are:• it is suitable for heavy parallelization (many overlapping components);• it is provably convergent, even when local subproblems are only solved approximately;
3 Johnson (2008) noted later a connection with other entropic proximal-point methods.
4 The idea of smoothing a non-smooth function and then applying accelerated gradient techniques is due to

Nesterov (2005), who established the convergence rate of such algorithms.

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

108• it produces optimality certificates for the MAP (when the relaxation is tight);• it keeps track of primal and dual residuals, allowing monitoring the LP-MAP optimization process and stopping whenever a desired accuracy level is attained.
After providing background on augmented Lagrangian methods (Section 6.2), we introduce and analyze AD3 (Section 6.3), and then devote special attention to how to solve thelocal subproblems (Section 6.4). We will see that for all logic factors described in Section 5.3,
those subproblems can be solved efficiently with sort operations. A general procedure isthen introduced in Section 6.5 that solves the subproblems for arbitrary factors, requiring
only a black-box for computing the local MAP. For instances where the LP-MAP relaxation
is not tight, we devise in Section 6.6 a branch-and-bound procedure. Experiments with synthetic Ising and Potts models, and two real-world tasks, protein design and frame-semantic
parsing, demonstrate the success of our approach (Section 6.7).
We postpone to Chapter 7 the evaluation of AD3 algorithm in a set of dependency parsing
experiments. There, we demonstrate that AD3 exhibits better performance than the projected
subgradient method when the factor graph contains many overlapping factors.

6.2 Augmented Lagrangian Methods
Before describing AD3 , we start with a brief overview of augmented Lagrangian methods.
Consider the following general convex optimization problem with equality constraints:
minimize f (u) + g(v)w.r.t.
u ∈ U, v ∈ V
s.t.
Au + Bv = c,

(6.1)

where U ⊆ RP and V ⊆ RQ are convex sets and f : U → R and g : V → R are convex functions. If it were not for the equality constraints in the last line, the problem in Eq. 6.1 would
be separable into two independent problems, min{ f (u) | u ∈ U} and min{ g(v) | v ∈ V},
i.e., the optimal value of (6.1) would be the sum of those two minima.

6.2.1

Method of Multipliers

For any η ≥ 0, an equivalent problem to the one in Eq. 6.1 isminimize f (u) + g(v) + 2 kAu + Bv − ck2
w.r.t.
u ∈ U, v ∈ V
s.t.
Au + Bv = c.
η

(6.2)

The only difference between Eqs. 6.1 and 6.2 is that the latter’s objective function has an extra
Euclidean penalty term which penalizes violations of the equality constraints; the scalarη ≥ 0 controls the intensity of this penalty. Since this term vanishes at feasibility, the two
problems have the same solution. The objective function of Eq. 6.2, however, is always ηstrongly convex, even if f (u) + g(v) is not. The Lagrangian of this problem (call it Lη ) is:Lη (u, v, λ) = f (u) + g(v) + λ · (Au + Bv − c) +η
kAu + Bv − ck2 .
2

(6.3)

6.2. AUGMENTED LAGRANGIAN METHODS

109

Note that Lη equals the Lagrangian of the original problem, plus the Euclidean penalty term.
It is called the η-augmented Lagrangian of Eq. 6.1.
Augmented Lagrangian methods (Bertsekas et al., 1999, Sect. 4.2) are a class of optimization methods that address the problem in Eq. 6.1 by seeking a saddle point of Lηt , for somesequence (ηt )t∈N . The earliest instance is the method of multipliers (Hestenes, 1969; Powell,1969), which alternates between a joint update of u and v through(ut+1 , vt+1 ) := arg min{ Lηt (u, v, λt ) | u ∈ U, v ∈ V}

(6.4)

and a gradient update of the Lagrange multiplier,λt+1 := λt + ηt (Aut+1 + Bvt+1 − c).

(6.5)

Under some assumptions, this method is convergent, and convergence can even be superlinear when the sequence (ηt )t∈N is increasing (see Bertsekas et al. 1999, Sect. 4.2).
A shortcoming of the method of multipliers is that the optimization problem in Eq. 6.4
is often difficult, since the penalty term of the augmented Lagrangian couples the variables
u and v (observe that the problem is separable when ηt = 0).

6.2.2

Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) avoids the shortcoming above, by replacing the joint optimization in Eq. 6.4 by a single block Gauss-Seidel step, in which u andv are alternately updated. That is, the following updates are made instead of Eq. 6.4:
ut+1 := arg min Lηt (u, vt , λt )
u ∈U

=

ηt
kAu + Bvt − ck2 ,
2

(6.6)

ηt
kAut+1 + Bv − ck2 .
2

(6.7)

arg min f (u) + A> λt · u +
u ∈U

and
vt+1 := arg min Lηt (ut+1 , v, λt )v ∈V=arg min g(v) + B> λt · v +v ∈VThis is followed by an update of the Lagrange multipliers as stated in Eq. 6.5. In many problems, the two minimizations in Eqs. 6.6–6.7 are much simpler than the joint maximization in
6.4; one example arises in our LP-MAP inference problem, as we shall see.
ADMM was invented in the 1970s by Glowinski and Marroco (1975) and Gabay and
Mercier (1976) and is related or analogous to other optimization techniques, such as DouglasRachford splitting, Spingarn’s method of partial inverses, and some proximal point methods(see Boyd et al. 2011 for an historical overview). It can also be seen as an instance of themore general problem of finding a root of a maximal monotone operator (Eckstein andBertsekas, 1992), and this insight has been used to derive generalized variants of ADMM.
One such variant was proposed by Fortin and Glowinski (1983), who consider a more general

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

110

λ-update, involving an extra parameter τ > 0:
λt+1 := λt + τηt (Aut+1 + Bvt+1 − c).

(6.8)

In the sequel, we will present the convergence results of ADMM and its variants whenapplied to the LP-MAP inference problem.

6.3 LP-MAP Inference with the AD3 Algorithm
We next apply ADMM to the LP-MAP inference problem described in Section 4.6, which
will result in the AD3 algorithm.

6.3.1

Derivation of AD3

Let us first recall the variable splitting formulation of the LP-MAP inference problem, adapting Eq. 4.52 to incorporate the hard constraint factors:
maximize∑α ∈F

w.r.t.

∑ ∑i ∈ N ( α ) y i ∈ Yiµ|α ∈ MARG(G |α ),ζ i (yi ) ∈ R,s.t.

θiα (yi )µiα (yi ) +∑ θα (yα )µα (yα )!

y α ∈Yα+ ∑∑ ∑ θi ( yi ) µi ( yi )βββ ∈ H i ∈ N ( β ) y i ∈ Yi

∀α ∈ F ∪ H,∀i ∈ V, yi ∈ Yi ,µiα (yi ) = ζ i (yi ),∀i ∈ V, α ∈ N(i ), yi ∈ Yi ,(6.9)

where for each soft factor α ∈ F we define (as in Eq. 4.52) µ|α := ((µi (.))i∈N(α) , µα (.)), andfor each hard constraint factor β ∈ H we define µ| β := (µi (.))i∈N( β) .5 As in Section 4.6.2, the“split” log-potentials θiα (yi ) are chosen so that∑ θiα (yi ) = θi (yi );(6.10)α ∈N ( i )the simplest choice being to set θiα (yi ) := deg(i )−1 θi (yi ).
We refer to the problem in Eq. 6.9 as the relaxed primal. We call the original MAP problem(Eq. 4.34) the primal, and the problem after dual decomposition (Eq. 4.58) the dual. Observethat the relaxed primal problem takes the form in Eq. 6.1 by doing the following:• replacing “max” with “min” and considering the negative of the objective function;• identifying u and v in Eq. 6.1 respectively with µ and ζ;• letting U in Eq. 6.1 be the Euclidean product of marginal polytopes ∏α∈F∪H MARG(G |α ),and setting V := R∑i |Yi | ;• identifying f (u) in Eq. 6.1 with the objective of Eq. 6.9, and letting g ≡ 0;
5 Recall that, by convention (Definition 5.1 and thereafter), the hard constraint factors do not have a factor
log-potential and are not associated with factor marginals; they are defined in terms of their acceptance set Sβ ,and in the binary case, their marginal polytope MARG(G | β ) is geometrically equivalent to conv Sβ .

6.3. LP-MAP INFERENCE WITH THE AD3 ALGORITHM

111• letting in Eq. 6.1 the matrix A be the identity, −B be a matrix with entries in {0, 1}
encoding the matching between the components of µ and ζ, and c = 0.
This falls into the class of a “general consensus optimization” problems, as described by
Boyd et al. (2011, Sect. 7.2).
The η-augmented Lagrangian function associated with Eq. 6.9 isLη (µ, ζ, λ) =∑ ∑∑i ∈ N ( α ) y i ∈ Yi

α ∈F

+ ∑

(θiα (yi ) + λiα (yi ))µiα (yi ) +∑ θα (yα )µα (yα )!

y α ∈Yα∑ ∑ (θi (yi ) + λi (yi ))µi (yi )ββββ ∈ H i ∈ N ( β ) y i ∈ Yi∑−

∑ ∑ λiα (yi )ζ i (yi )α ∈ F ∪ H i ∈ N ( α ) y i ∈ Yi−η2
∑ ∑ (µiα (yi ) − ζ i (yi )) .
2 α∈∑
F ∪H i ∈N ( α ) y ∈Yi

(6.11)i

This is the standard Lagrangian that we had derived in Eq. A.7 with additional terms forthe hard constraint factors, and the additional Euclidean penalty term in the last line. The
ADMM updates are the following:
Broadcast:Gather:Multiplier updates:µ(t) := arg max Lηt (µ, ζ (t−1) , λ(t−1) ),(6.12)

ζ (t) := arg max Lηt (µ(t) , ζ, λ(t−1) ),
ζ


(t)
α ( t −1)α(t)α(t)− τηt µi − ζ i , ∀α ∈ F, i ∈ N( a),λi : = λi(6.13)µ ∈M(6.14)

where, in Eq. 6.12, M := ∏α∈F∪H MARG(G |α ). We next analyze each of these updates.

Broadcast step. Crucially, the maximization with respect to µ (6.12) can be carried out inparallel at each factor, as in the projected subgradient algorithm (Algorithm 5). The only
difference is that, instead of a local MAP computation, each soft-factor worker now needs tosolve a quadratic program of the form:
maximize∑ ∑ (θiα (yi ) + λiα (yi ))µiα (yi ) + ∑ θα (yα )µα (yα )i ∈ N ( α ) y i ∈ Yi

y α ∈Yαη−(µiα (yi ) − ζ i (yi ))2∑∑
2 i ∈N ( α ) y ∈Yiw.r.t.

iµ|α ∈ MARG(G |α ),(6.15)and each hard-factor worker needs to solve a similar quadratic program, but without the term
for the factor marginals:
maximize∑∑ (θi (yi ) + λi (yi ))µi (yi ) −β

i ∈ N ( β ) y i ∈ Yi

w.r.t.

µ| β ∈ MARG(G | β ),

β

β


2
η
β
µ
(
y
)
−
ζ
(
y
)
i i
∑ i i
2 i∈∑
N ( β ) y ∈Yii

(6.16)

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION112

The only difference between the subproblems in Eqs. 6.15–6.16 and the corresponding subproblems in the projected subgradient algorithm is the inclusion of a Euclidean penalty term,
which regularizes toward the current ζ. This term penalizes deviations from the global consensus. We defer to Sections 6.4 and Section 6.5 a detailed derivation of procedures to solve
the local subproblems in Eqs. 6.15–6.16.
Gather step. The maximization of Lηt with respect to ζ (Eq. 6.13) has a closed form. Indeed,this problem is separable into independent optimizations, one for each i ∈ V and yi ∈ Yi :minimizew.r.t.

2


1
α
−1 α
ζ
(
y
)
−
µ
(
y
)
−
η
λ
(
y
)
i i
i
i
i
i
t
2 α∈∑
N (i )
ζ i (yi ) ∈ R,

whose solution isζ i ( yi )(t) : =(6.17)



1
α
−1 α
µ
(
y
)
−
η
λ
(
y
)
.
i
i
i
i
t
deg(i ) α∈∑N (i )(6.18)

This update can be simplified further when the penalty ηt is kept constant:
Proposition 6.1 Let λ0 = 0 and consider a sequence of ADMM iterations indexed by t ∈ N, eachperforming the updates in Eqs. 6.12–6.14, with ηt := η kept constant. Then, we have for each t ∈ N:

∑ λi

α(t)

= 0,

∀i ∈ V.

(6.19)α ∈N ( i )As a consequence, under the assumptions above, the ζ-update is simply an averaging operation:
ζ i ( yi )(t) : =1µiα (yi ).
deg(i ) α∈∑N (i )(6.20)

Proof. Use induction: Eq. 6.19 is trivially satisfied for λ(0) = 0; for t > 0, if λ(t−1) satisfies
Eq. 6.19, then, after the Lagrange multiplier updates in Eq. 6.14, we have∑α ∈N ( i )

α(t)
λi =

=

∑

α ( t −1)
λi
− τηt

∑

α ( t −1)λi− τηtα ∈N ( i )α ∈N ( i )= (1 − τ )∑ λi

α ( t −1)

!

∑

α(t)
(t)
µi − deg(i )ζ i

∑

α(t)
µi −

α ∈N ( i )α ∈N ( i )

= 0.

∑

α ∈N ( i )α(t)α ( t −1)
µi − ηt−1 λi!

(6.21)α ∈N ( i )Putting all the pieces together, we obtain AD3 , depicted as Algorithm 8. AD3 retains themodularity of the projected subgradient algorithm described in Section 4.6.2 (Algorithm 5).
Both algorithms are iterative “consensus algorithms,” alternating between a broadcast operation, where simple subproblems are distributed across local workers (lines 5–9 in Algorithm 8), and a gather operation, where the local solutions are assembled by a controller,

6.3. LP-MAP INFERENCE WITH THE AD3 ALGORITHM

113

Algorithm 8 Alternating Directions Dual Decomposition (AD3 )1: input: factor graph G , parameters θ, penalty constant η, constant τ
2: initialize ζ uniformly (i.e., ζ i (yi ) = 1/|Yi |, ∀i ∈ V, yi ∈ Yi )3: initialize λ = 0
4: repeat5:
for each factor α ∈ F ∪ H do
6:
set unary log-potentials ωiα := deg(i )−1 θi + λiα , for i ∈ N(α)
7:set ω|α := ((ωiα )i∈N(α) , θα ) if α ∈ F, or ω|α := (ωiα )i∈N(α) if α ∈ H
8:set µb|α := SolveQP(ω|α ; ζ |α ) (Eq. 6.15 or Eq. 6.16)
9:
end forbiα
10:compute average ζ i := deg(i )−1 ∑α∈N(i) µ
α
α
α
11:
update λi := λi − τη (µbi − ζ i )12: until convergence.
13: output: primal variables ζ and µ, dual variable λwhich updates the current global solution (line 10) and adjusts multipliers to promote a
consensus (line 11). The main practical difference is that AD3 also broadcasts the current
global solution ζ to the workers, allowing them to regularize their subproblems toward thatsolution. This is expressed in line 8 through the procedure SolveQP, which replaces the procedure ComputeMAP of Algorithm 5. This has the consequence of speeding up consensus,
as we shall see.

6.3.2

Convergence Analysis

We now establish the convergence of AD3 , which follows directly from the general convergence properties of ADMM. Remarkably, convergence is ensured with a fixed stepsize/penalty,
without the need for annealing. This is because, as we approach convergence, the penalty
term must go to zero to ensure (relaxed) primal feasibility.
Proposition 6.2 (Convergence.) Let (µ(t) , ζ (t) , λ(t) )t be the sequence of iterates produced by Algo√rithm 8 with a fixed ηt = η and 0 < τ ≤ ( 5 + 1)/2 ' 1.61. Then the following holds:
1. Primal feasibility of the relaxed primal (Eq. 6.9) is achieved in the limit, i.e.,
α(t)

k µi

(t)

− ζ i k → 0,

∀α ∈ F ∪ H, i ∈ N( a);(6.22)

2. The sequence (ζ (t) , µ(t) )t∈N converges to an optimal solution of the relaxed primal (Eq. 6.9);
3. The sequence (λ(t) )t∈N converges to an optimal solution of the dual (Eq. 4.58);
4. Every element λ(t) in the latter sequence is dual feasible. As a consequence, the objective of
Eq. 4.58 evaluated at λ(t) approaches the optimal value from above.
Proof. 1, 2, and 3 are general properties of ADMM algorithms (Glowinski and Le Tallec,
1989, Thm. 4.2). All necessary conditions are met: the problem in Eq. 6.9 is convex and thecoupling constraints are such that the corresponding matrix B in Eq. 6.1 is full column rank.
Point 4 is a consequence of Proposition 6.1 and the fact that the dual constraint set Λ isprecisely defined by Eq. 6.19 (see Eq. 4.56).

114

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

The first and last points in Proposition 6.2 reveal an important feature of AD3 : after asufficient decrease of the penalty term, we have at hand a nearly-feasible primal-dual pair.
We next make this more precise by describing how AD3 allows us to keep track of a primaland dual residuals, and how it can be stopped once they fall below a threshold.
Primal and dual residuals. Recall that the projected subgradient method is able to provide
optimality certificates when the relaxation is tight (i.e., when the solution of the LP-MAP
problem is the true MAP). While this is good enough when tight relaxations are frequent, asin the settings explored by Rush et al. (2010), Koo et al. (2010), and Rush and Collins (2011),
it is hard to know when to stop when a relaxation gap exists. We would like to have similar
guarantees concerning the relaxed primal.6 A general weakness of subgradient algorithms isthat they do not have this capacity, and so are usually stopped rather arbitrarily by specifying
a maximum number of iterations. In contrast, ADMM allows to keep track of primal anddual residuals (Boyd et al., 2011). This allows providing certificates not only for the exact
primal solution (when the relaxation is tight), but also to terminate when a near optimal(t)solution of the relaxed primal problem has been found. The primal residual r P measures theamount by which the agreement constraints are violated:(t)α(t)∑α∈F∪H ∑i∈N(α) ∑yi ∈Yi (µi (yi ) − ζ i (yi ))2
(t)
rP =
;
∑i∈V ∑yi ∈Yi deg(i )(6.23)(t)where the constant in the denominator ensures that r P ∈ [0, 1]. The dual residual r tD is theamount by which a dual optimality condition is violated (see Boyd et al. 2011, p. 18, fordetails). It is computed via:
(t)

rD =

∑i∈V ∑yi ∈Yi deg(i )(ζ i (yi )(t) − ζ i (yi )(t−1) )2,∑i∈V ∑yi ∈Yi deg(i )(6.24)(t)and also satisfies r D ∈ [0, 1]. Our stopping criterion is thus that these two residuals are
below a threshold, e.g., 10−3 .
Picking the parameters η and τ. An important issue is that of picking the penalty andstepsize coefficients, which boils down to picking the hyperparameters η and τ. In practice,
we found that τ does not have a great impact on the convergence speed, so we often set
τ = 1 in the experiments, which renders the penalty coefficient equal to the stepsize and is
the most common approach in ADMM implementations. However, it is often important tochoose a “good” η. Figure 6.1 shows how different choices of η affect the progress in the dual
objective, for a synthetic Potts grid. Typically, a small value of η leads to a stable albeit slower
progress, while a large value may be a plus in the long run, but leads to strong oscillations
in earlier iterations. A practical alternative, which we follow in the parsing experiments(Section 7.4), is to dynamically adjust η in earlier iterations, as described in Boyd et al. (2011,
6 This problem is more important than it may look. Problems with many slaves tend to be less exact, hence
relaxation gaps are frequent. Also, when decoding is embedded in training, it is useful to obtain the fractional
solution of the relaxed primal (rather than an approximate integer solution). See Kulesza and Pereira (2007) andMartins et al. (2009c) for details.

Dual Objective

6.3. LP-MAP INFERENCE WITH THE AD3 ALGORITHM

115

η=0.1
η=1
η=10

10000
5000
0

50

100
150
Number of Iterations

Figure 6.1: Progress in the dual objective for different values of η (the factor graph is abinarized 20 × 20 Potts grid with 8 states). For η = 0.1, progress is too slow; for η = 10
there are strong oscillations, which makes the algorithm slow to take off. In this example,
convergence is faster with an intermediate value, η = 1.
p.20), and freeze it afterwards, so that convergence is not compromised.
Approximately solving the local subproblems. The next proposition states that convergence may still hold if the local subproblems are only solved approximately, provided thesequence of residual errors is summable. This again follows from a general result established
for the ADMM algorithm. We refer to Eckstein and Bertsekas (1992) for a proof.
Proposition 6.3 (Eckstein and Bertsekas (1992)) Let ηt = η be fixed and τ = 1. Consider the(t)sequence of residual errors ε(t) := (εα )α∈F , where
(t)

(t)

(t)

εα := µb|α − µ̃|α ,(6.25)

where µb|α is the true solution of Eq. 6.15 or 6.16, and µ̃|α is an estimate of µb|α retrieved by an(t)
approximate algorithm. Then, all points 1–4 in Proposition 6.2 still hold, provided ∑∞t=1 k ε k < ∞.
The importance of Proposition 6.3 will be clear in Section 6.5, when we describe a general
iterative algorithm for solving the local quadratic subproblems. Essentially, Proposition 6.3
allows these subproblems to be solved numerically up to some accuracy without compromising global convergence, as long as the accuracy of the solutions improves sufficiently fastover ADMM iterations.
Convergence rate. Even though ADMM was invented in the 1970s, no convergence rate
was known until very recently. The next proposition states the iteration bound of AD3 , whichturns out to be asymptotically equivalent to the one of Jojic et al. (2010); both are O(1/e),
and therefore better than the O(1/e2 ) bound of the projected subgradient algorithm.
Proposition 6.4 (Convergence rate.) Under the conditions stated in Proposition 6.2 and setting
τ = 1, the AD3 algorithm yields an e-accurate objective value after O(1/e) iterations.
Proof. See Appendix D. The proof invokes results recently established by He and Yuan
(2011) and Wang and Banerjee (2012) concerning convergence in a variational inequality
setting, which we adapt to establish a O(1/t) convergence rate in the dual objective. By
setting e = O(1/t), the result follows.

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

116

6.4 Solving the Local Subproblems
We next show how to efficiently obtain an exact solution of Eqs. 6.15–6.16 for a variety ofcases, including binary graphs with logic constraints. The results of this section will becomplemented, in Section 6.5, with a new procedure, which will allow us to handle arbitrary
factors, further extending the scope of problems where AD3 is applicable.
By subtracting a constant to the objective, rescaling, and turning the maximization into aminimization, the problem in Eq. 6.15 can be written more compactly as:
minimize


2
1
α
−1
− η −1 ∑ θ α ( y α ) µ α ( y α )
µ
(
y
)
−
η
ω
(
y
)
i
i
i
∑ i
2 i∈∑
y ∈Yα
N ( α ) y ∈Yiw.r.t.

iαµ|α ∈ MARG(G |α ),(6.26)

where ωi (yi ) := ηζ i (yi ) + θiα (yi ) + λiα (yi ). For hard factors, the corresponding problem iseven more compact:
minimize21βµ i ( y i ) − η −1 ω i ( y i )∑∑2 i ∈N ( β ) y ∈Yiw.r.t.

iµ| β ∈ MARG(G | β ),(6.27)

i.e., it amounts to computing an Euclidean projection onto the marginal polytope.
We will see that, in many cases, Eqs. 6.26–6.27 admit a closed-form solution or can besolved exactly by an efficient procedure: this is so for binary pairwise factors, the logic hardconstraint factors of Section 5.3, and factors linked to multi-valued variables, once binarized.
In the first two cases, and also in the last case if the factors are pairwise, the asymptotic
computational cost is the same as that of MAP computations, up to a logarithmic factor.

6.4.1

Binary pairwise factors

In this section, we obtain the closed form solution of problem Eq. 6.26 for binary pairwise
factors. Define z1 := µ1 (1), z2 := µ2 (1), and z12 := µ12 (1, 1). In Appendix E.1, we rewrite
the problem in Eq. 6.26 as:
minimizew.r.t.
s.t.

11(z1 − c1 )2 + (z2 − c2 )2 − c12 z12
22
z1 , z2 , z12 ∈ [0, 1]3
z12 ≤ z1 ,z12 ≤ z2 ,z12 ≥ z1 + z2 − 1,

(6.28)

where we have substituted
c1 = (η −1 ω1 (1) + 1 − η −1 ω1 (0) − η −1 θ12 (00) + η −1 θ12 (10))/2
c2 = ( η−1ω2 ( 1 ) + 1 − η

c12 = (η−1−1

θ12 (00) − η−1ω2 ( 0 ) − η

θ12 (10) − η−1−1

θ12 (00) + η

θ12 (01) + η−1−1(6.29)

θ12 (01))/2(6.30)

θ12 (11))/2.

(6.31)

Observe that we can assume c12 ≥ 0 without loss of generality—indeed, if c12 < 0, we recover
0 = − c , z0 = 1 − z , z0 = z − z . Thus,
this case by redefining c10 = c1 + c12 , c20 = 1 − c2 , c12
2 12
12 2
1
12

6.4. SOLVING THE LOCAL SUBPROBLEMS

117

assuming that c12 ≥ 0, the lower bound constraints z12 ≥ z1 + z2 − 1 and z12 ≥ 0 are always
inactive and can be ignored. Hence, (6.28) can be simplified to:minimizew.r.t.
s.t.

11(z1 − c1 )2 + (z2 − c2 )2 − c12 z12
22
z1 , z2 , z12z12 ≤ z1 ,z12 ≤ z2 ,z1 ∈ [0, 1],z2 ∈ [0, 1].

(6.32)The next proposition establishes a closed form solution for this problem, which immediatelytranslates into a procedure for SolveQP for binary pairwise factors.
Proposition 6.5 The problem in Eq. 6.32, with c12 ≥ 0, has the following closed form solution:[c2 + c12 ]U )if c1 > c2 + c12
 ([c1 ]U ,(z1∗ , z2∗ ) =([c1 + c12 ]U ,[ c2 ]U )if c2 > c1 + c12 (6.33)
 ([(c + c + c )/2] , [(c + c + c )/2] ) otherwise,
2
2
U
U
1
12
1
12
∗ = min{ z∗ , z∗ }, where [ x ]and z12
U : = min{max{ x, 0}, 1} denotes projection (clipping) onto the
1 2unit interval U := [0, 1].

Proof. See Appendix E.1.

6.4.2

Hard constraint factors

We next turn to hard constraint factors, whose local subproblems in AD3 take the form in
Eq. 6.27. We have seen in Chapter 5 that such factors are important in many applications,
ranging from error-correcting coding to NLP, and we have presented in Section 5.3 an inventory of hard constraint factors that allow expressing constraints in first-order logic, alongwith an algebraic characterization of their marginal polytopes (see Figure 5.4). For a binary
hard constraint factor with acceptance set Sβ , we have seen that its marginal polytope isequivalent to the convex hull Zβ = conv Sβ .
The problem in Eq. 6.27 is precisely a projection onto the marginal polytope of the hard conβ
β
β
β
β
β
straint factor. Noting that kµi − η −1 ωi k2 = (µi (1) − η −1 ωi (1))2 + (1 − µi (1) − η −1 ωi (0))2 ,βββ
which equals a constant plus 2(µi (1) − η −1 (ωi (1) + 1 − ωi (0))/2)2 , we have that Eq. 6.27
can be written as:
1(6.34)
projβ (z0 ) := arg min kz − z0 k2 ,z ∈Z β 2where z0i := (ωi (1) + 1 − ωi (0))/2, for each i ∈ N( β). We next show how to compute thisprojection for the factors introduced in Section 5.3. Several auxiliary results and proofs canbe shown in Appendix E.2.
ββ

XOR Factor. Consider the one-hot XOR factor introduced in Section 5.3. Recall that for thiscase, the marginal polytope ZXOR is the probability simplex, as stated in Eq. 5.25:(
ZXOR =z ∈ [0, 1]KK∑ zk = 1

k =1

)
.

(6.35)

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

118

Algorithm 9 Projection onto simplex (Duchi et al., 2008)
Input: z0
Sort z0 into y0n: y1 ≥ . . . ≥ y K ojFind ρ = max j ∈ [K ] | y0j − 1j (∑r=1 y0r ) − 1 > 0ρ
Define τ = 1ρ ∑r=1 y0r − 1
Output: z s.t. zi = max{z0i − τ, 0}.

Hence the quadratic problem in Eq. 6.26 reduces to that of projecting onto the simplex. That
problem is well-known in the optimization community (see, e.g., Michelot 1986); by writing
the KKT conditions, it is simple to show that the solution z∗ is a soft-thresholding of z0 , andtherefore the problem can be reduced to that of finding the right threshold. Algorithm 9 provides an efficient procedure; it requires a sort operation, which renders its cost O(K log K ).
A proof of correctness appears in Duchi et al. (2008).7
OR Factor. Consider the OR factor introduced in Section 5.3. For this case, Eq. 5.29 gives
us the following expression for the marginal polytope, which we reproduce here:(ZOR =z ∈ [0, 1]KK∑ zk ≥ 1).

(6.36)k =1

In Appendix E.2, we introduce the Sifting Lemma (Lemma E.1), which guarantees the correctness of the following procedure for computing a projection onto ZOR :1. Set z̃ as the projection of z0 onto the unit cube. This can be done by clipping each coordinate to the unit interval U = [0, 1], i.e., by setting z̃i = [z0i ]U = min{1, max{0, z0i }}.
If ∑iK=1 z̃i ≥ 1, then return z̃. Else go to step 2.
2. Return the projection of z0 onto the simplex (use Algorithm 9).
The validity of the second step stems from the following fact: if the relaxed problem inthe first step does not return a feasible point then, from the Sifting Lemma, the constraint∑iK=1 zi ≥ 1 has to be active, i.e., we must have ∑iK=1 zi = 1. This, in turn, implies that z ≤ 1,
hence the problem becomes equivalent to the XOR case. In sum, the worst-case runtime isO(K log K ), although it is O(K ) if the first step succeeds.
OR-with-output Factor. Consider now the OR-with-output factor introduced in Section 5.3,
whose marginal polytope is given by Eq. 5.39:(
ZOR−out =z ∈ [0, 1]

K +1

K

∑ zk ≥ zK+1 , zk ≤ zK+1 , ∀k ∈ {1, . . . , K}).

(6.37)

k =1

Solving the quadratic problem for this factor is slightly more complicated than in the previous two cases; however, we next see that it can also be addressed in O(K log K ) with a sort
operation.
7 In the high-dimensional case, a red-black tree can be used to reduce this cost to O ( K ) (Duchi et al., 2008). In
later iterations of AD3 , great speed-ups can be achieved in practice since this procedure is repeatedly invokedwith only small changes to the coefficients.

6.4. SOLVING THE LOCAL SUBPROBLEMS

119

The polytope ZOR−out can be expressed as the intersection of the following three sets:8
UK+1 := [0, 1]K+1
A1 : = { z ∈ R(
A2 : =

K +1(6.38)| zk ≤ zK+1 , k = 1, . . . , K })z ∈ [0, 1]K+1(6.39)

K∑ z k ≥ z K +1.

(6.40)

k =1

We further define A0 := [0, 1]K+1 ∩ A1 . From the Sifting Lemma (Lemma E.1), we have thatthe following procedure is correct:
1. Set z̃ as the projection of z0 onto the unit cube. If z̃ ∈ A1 ∩ A2 , then we are done: justreturn z̃. Else, if z̃ ∈ A1 but z̃ ∈/ A2 , go to step 3. Otherwise, go to step 2.
2. Set z̃ as the projection of z0 onto A0 (we will describe how to do this later). If z̃ ∈ A2 ,return z̃. Otherwise, go to step 3.
3. Return the projection of z0 onto the set {z ∈ [0, 1]K+1 | ∑kK=1 zk = zK+1 }. This set isprecisely the marginal polytope of the XOR-with-output factor (cf. Eq. 5.36), hence
the projection corresponds to the local subproblem for that factor. As described in
Section 5.3, XOR-with-output is equivalent to a XOR with the last input negated. For
this, we can employ the procedure already described for the XOR factor, which consistsof a projection onto the simplex (using Algorithm 9).9
Note that the first step above can be omitted; however, it avoids performing step 2 (which
requires a sort) unless it is really necessary. To completely specify the algorithm, we only
need to explain how to compute the projection onto A0 (step 2):
Procedure 6.1 To project onto A0 = [0, 1]K+1 ∩ A1 :2a. Set z̃˜ as the projection of z0 onto A1 . Algorithm 10 shows how to do this.
2b. Set z̃ as the projection of z̃˜ onto the unit cube (with the usual clipping procedure).
The proof that the composition of these two projections yields the desired projection onto
A0 is a bit involved, and we included it in the appendix (Proposition E.2).10 We only need todescribe how to project onto A1 (step 2a), which is written as the following problem:1
min kz − z0 k2
z 2

s.t.

zk ≤ zK+1 , ∀k = 1, . . . , K.

8 Actually, the set UK +1 is redundant, since we have A

2 ⊆U

K +1 and therefore Z(6.41)

OR−out = A1 ∩ A2 . However
it is computationally advantageous to consider this redundancy, as we shall see.
9 This will be clarified in the sequel, when we show how to handle negations.
10 Note that in general, the composition of individual projections is not equivalent to projecting onto theintersection. In particular, commuting steps 2a and 2b would make our procedure incorrect.

120

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

Algorithm 10 Projection onto A1
Input: z0
Sort z01 , . . . , z0K
n into y1 ≥ . . . ≥
 yK


o
j −1
Find ρ = min j ∈ [K + 1] | 1j z0,K+1 + ∑r=1 yr > y jρ −1Define τ = 1ρ z0,K+1 + ∑r=1 yr
Output: z s.t. zK+1 = τ and zi = min{z0i , τ }, i = 1, . . . , K.

It can be successively rewritten as:
K1
1
min (zK+1 − z0,K+1 )2 + ∑ min (zk − z0k )2
z K +1 2z ≤z
2i = 1 k K +1
K1
1
= min (zK+1 − z0,K+1 )2 + ∑ (min{zK+1 , z0k } − z0k )2
z K +1 2
2
k =1

1
1
= min (zK+1 − z0,K+1 )2 +z K +1 2
2 k∈I∑(z(zK+1 − z0k )2 .

(6.42)

K +1 )

where I(zK+1 ) , {k ∈ [K ] : z0k ≥ zK+1 }. Assuming that the set I(zK+1 ) is given, the previous
is a sum-of-squares problem whose solution isz∗K+1 =

z0,K+1 + ∑k∈I(zK+1 ) z0k
1 + |I(zK+1 )|.

(6.43)The set I(zK+1 ) can be determined by inspection after sorting z01 , . . . , z0K . The procedure isshown in Algorithm 10.
Negations. Finally, we show how a factor that contains negated inputs (see Section 5.3.3) canhave its problem reduced to that of a factor without negations.
Let factor β0 be constructed from β by negating one of the inputs (without loss of generality, the first one, y1 )—i.e., Sβ0 = {y β0 | (1 − y1 , y2 , . . . , yK ) ∈ Sβ }. Then, if we have aprocedure for evaluating the operator projβ , we can use it for evaluating projβ0 through thechange of variable z10 := 1 − z1 , which turns the objective function into (1 − z10 − z01 )2 =(z10 − (1 − z01 ))2 . Naturally, the same idea holds when there is more than one negated input.
The overall procedure computes z = projβ0 (z0 ):0 = z if it is not negated and z0 = 1 − z otherwise.
1. For each input i, set z0i
0i
0i
0i

2. Obtain z0 as the solution of projβ (z00 ).
3. For each input i, set zi = zi0 if it is not negated and zi = 1 − zi0 otherwise.

6.4.3

Larger factors and multi-valued variables

For general factors, a closed-form solution of the problem in Eq. 6.26 is not readily available.
This is a disadvantage relative to the projected subgradient algorithm, which can easily handle factors with multi-valued variables, and even certain structured factors such as chains6.5. AN ACTIVE SET METHOD FOR HANDLING ARBITRARY FACTORS

121

or trees, by using dynamic programming (or other combinatorial algorithm) for the corresponding local MAP computations. A similar strategy for computing an exact solution forthese special structures does not seem possible in AD3 .
However, AD3 can still tackle general factors with multi-valued variables by employing
the binarization procedure described in Section 5.5.1. After binarization, the resulting graphis a binary constrained graph for which we already developed the machinery necessary forsolving the AD3 subproblems. As described in Section 5.5.1, this transformation preserves
the LP-MAP problem and, up to log factors, it results in the same asymptotic complexity(per iteration) of message-passing algorithms in the original graph.
An alternative strategy that sidesteps binarization is to use an inexact algorithm that
becomes sufficiently accurate as Algorithm 8 proceeds (exploiting the convergence result in
Proposition 6.3); this can be achieved by warm-starting with the solution obtained in theprevious iteration. We reserve Section 6.5 to describe in full detail an active-set method that
implements this strategy, and which can gracefully handle coarser decompositions, in which
each factor is a subgraph such as a chain or a tree.

6.5 An Active Set Method for Handling Arbitrary Factors
We next describe an iterative procedure for tackling the local AD3 subproblems of arbitrary
factors. The only interface between our procedure and the factor is a black-box implementation of the function ComputeMAP, just like the projected subgradient algorithm (Algorithm 5). This enables plugging combinatorial algorithms that are tailored for computing
the MAP on certain factors.
There are several ways to convert a quadratic problem into a sequence of linear problems—
the Frank-Wolfe algorithm is perhaps the simplest example (Frank and Wolfe, 1956). However, that algorithm is not suitable for our task, since it is too slow to converge and does not
exploit the sparsity of the solution (which we establish in Proposition 6.6 below). Instead, we
propose an active set method (Nocedal and Wright, 1999, Sect. 16.4). For simplicity, we limit
our description to the soft factor case (Eq. 6.26), but the method extends in a straightforwardmanner to hard factors, as will be discussed later.
Let us start by writing the local subproblem in Eq. 6.26 in the following compact form:
minimizewith respect tosubject to1
k u − a k2 − b > v
2u ∈ R ∑ i ∈ N ( α ) | Yi | , v ∈ R | Y α | u = Mv
1> v = 1 v ≥ 0,(6.44)

where we have introduced variables u and v to denote the variable marginals u := (µiα (.))i∈N(α)
and the factor marginals v := µα (.), respectively; and similarly for the parameters a :=(η −1 ωi (.))i∈N(α) and b := η −1 θα (.). The three sets of constraints in (6.44) come from the122

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

following representation of the marginal polytope MARG(G |α ):11
α (y ) =


µ
(
y
)
,
∀
i
∈
N
(
α
)
,
y
∈
Y
,
µ
∑
α
i
i
i
y α ∼ yiαi,
MARG(G |α ) =µ|α∑yα ∈Yα µα (yα ) = 1,µα (yα ) ≥ 0,
∀ y α ∈ Yα

(6.45)

where we have introduced the matrix M in Eq. 6.44 with ∑i |Yi | rows and |Yα | columns, toencode the consistency constraints, i.e., given by [M](i,yi ),yα := [[yα ∼ yi ]].
A crucial result is that the problem in Eq. 6.44 always admits a sparse solution, as we show
in the next proposition:
Proposition 6.6 The problem in Eq 6.44 admits a solution v∗ ∈ R|Yα | with at most ∑i∈N(α) |Yi | −N(α) + 1 nonzeros.
Proof. See Appendix E.3.
The fact that the solution lies in a low dimensional subspace, as established in Proposition 6.6, makes active set methods appealing. Such methods keep track of an active set
given by the nonzero components of v; we now know that we only need to maintain at
most O(∑i |Yi |) elements in the active set—note the additive, rather than multiplicative, dependency on the number of values of the variables. This alleviates eventual concerns about
memory and storage. Before introducing the method in full detail, we need to analyze thedual and write the KKT conditions of the problem in Eq. 6.44.
Lagrangian and Dual Problem. The problem (6.44) has O(Yα ) variables and constraints,
a number which grows exponentially fast with deg(α). We next derive a dual formulation
with only O(∑i |Yi |) variables. The Lagrangian of (6.44) is
L(u, v, w, τ, λ) =1ku − ak2 − b> v − w> (Mv − u) − τ (1 − 1> v) − λ> v.
2(6.46)

Equating ∇u L(u) and ∇v L(v) to zero leads to the equations:u = a−w(6.47)

M> w + b = τ1 − λ.

(6.48)

Since the Lagrange variables λ are constrained to be non-negative, the dual problem takes
the form (after replacing the maximization with a minimization and subtracting a constantto the objective):
minimizewith respect tosubject to1
k w − a k2 + τ
2w ∈ R ∑ i | Yi | , τ ∈ R
M> w + b ≤ τ1.

(6.49)

11 Eq. 6.45 follows directly from the constraints that characterize the local polytope (Eq. 4.40), by notingthat summing over the first constraints µiα (yi ) = ∑yα ∼yi µα (yα ), we obtain that, for each i ∈ N(α), it holds∑yi ∈Yi µiα (yi ) = ∑yα µα (yα ).

6.5. AN ACTIVE SET METHOD FOR HANDLING ARBITRARY FACTORS

123

This problem can be seen as a “projection with slack” onto the set {w | M> w + b ≤ 0}.12
We will apply the active set method of Nocedal and Wright (1999, Sect. 16.4) to the dual
problem (6.49). Let mr denote the rth column of matrix M. This method keeps track of aworking set of constraints which are assumed to be active:no
W := r ∈ {1, . . . , |Yα |} | mr> w + br − τ = 0 ;(6.50)

by complementary slackness, at the optimum this set contains the support of v∗ : W ⊇ r ∈{1, . . . , |Yα |} | vr > 0 .

KKT conditions.

The KKT equations of (6.49) are:u−a+w = 0(∇u L = 0)

(6.51)

M w + b = τ1 − λ(∇v L = 0)

(6.52)

Mv = u(Primal feasibility)(6.53)>

1 v=1(Primal feasibility)(6.54)

y≥0(Primal feasibility)(6.55)

λ≥0(Dual feasibility)(6.56)(Complementary slackness).

(6.57)>

λ> v = 0

We can eliminate variables u and w and reduce the above set of equations to
M> Mv + τ1 = M> a + b + λ(6.58)

1> v = 1(6.59)v ≥ 0(6.60)

λ ≥ 0(6.61)>

λ v = 0.

(6.62)Let I := {r ∈ {1, . . . , |Yα |} | vr > 0} denote the support of v, i.e., the indices corresponding
to nonzero entries. Obviously, we do not know the set I beforehand; along the algorithm,
we maintain our best guess through the active set W (Eq. 6.50). Denote by vI and λI thesubvectors formed by only those indices. Denote by MI the submatrix of M with columns
in I. The above set of equations imply λI = 0 and the following system in matricial form:"

MI> MI 1
1>
0

#"

vI
τ

#

"

=

MI> a + bI
1#.

(6.63)

If the matrix on the left-hand side is non-singular, the system has a unique solution (vbI , τb).13
Padding v
bI with zeros, we form vb (this never needs to be done explicitly, of course). If v
b and
12 Interestingly, this problem has affinities with well-known formulations in machine learning using L 2
regularization, such as support vector machines and the MIRA algorithm.
13 If at some point of the active-set algorithm the matrix becomes singular, we can still compute a solution bylooking at its null space and proceed with the execution.

124

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITIONτb satisfy the KKT conditions (6.58–6.62), i.e., if we have v
b ≥ 0 andM> Mb
v + τb1 ≥ M> a + b,

(6.64)

then the set I is correct and v
b is a solution of the QP (Eq. 6.44).
To use this rationale algorithmically, the following two operations need to be performed:• Solving the KKT system (6.63). This involves decomposing or inverting a matrix of size
|I|-by-|I|. Since we need to perform this operation repeatedly after inserting or removing one element from the active set, this procedure can be done efficiently (namely,
in time O(|I|2 )) by keeping track of the inverted matrix. From Proposition 6.6, thisruntime is manageable, since we can limit the size of the active set to |I| = O(∑i |Yi |).
Note also that after inserting a new configuration yα to the active set, this will originate
a new column in the matrix MI , and for the matrix inversion stated above, we need to
update the product MI> MI . From the definition of M and simple algebra, we have thatthe (yα , y0α ) entry in M> M is just the number of common values between the configurations yα and y0α . Hence, when a new configuration yα is added to the active set W, that
configuration needs to be compared with all the others already in W.
• Checking if any of the constraints (6.64) is violated. By setting u
b = Mbv and wb = a−ub,the constraints can be written as M> w
b + b ≤ τb1, and they hold if and only if
max mr> w
b + br ≤ τb.
r ∈Yα(6.65)

Hence, to verify the constraints we need to compute the maximum on the left handside; interestingly that maximum is nothing but the MAP response of the factor α tovariable log-potentials w
b and the factor log-potentials b, and this value can be computed through the function ComputeMAP.

Active set algorithm. Algorithm 11 depicts the whole procedure, which is an instantiationof Nocedal and Wright (1999, Alg. 16.1) adapted to our purposes. At each iteration s, an active set Ws is maintained which stores our current guess about the support of v.14 The active
set is initialized arbitrarily: e.g., in the first outer iteration of AD3 it can be initialized withthe single output yα which is the MAP given log-potentials a and b; and in subsequent AD3
iterations it can be warm-started with the solution obtained in the previous outer iteration.
At each inner iteration, the KKT system (Eq. 6.63) is solved given the current active set.
If the solution is the same as in the previous round, a black-box ComputeMAP is then invoked to check for violations of the KKT constraints; if there are no violations, the algorithm
returns; otherwise it adds the most violated constraint to the active set.
If the solution of the KKT system v
b is different from the one in the previous round (bvs ),a line search is made to set v
bs+1 in-between v
bs and v
b. The stepsize αs is chosen to yield
the most possible progress while keeping the constraints satisfied. This has a closed form
14 The meaning of our active set is the opposite of the active set as defined by Nocedal and Wright 1999, since

we are tackling the dual problem.

6.6. EXACT INFERENCE WITH BRANCH-AND-BOUND

125

Algorithm 11 Active Set Algorithm for Solving a General AD3 Subproblem
1: input: Parameters a, b, M, starting point v0
2: initialize W0 as the support of v0
3: for s = 0, 1, 2, . . . do
4:solve the KKT system and obtain v
b and τb (Eq. 6.63)
5:if v
b = vs then6:
compute ub := Mbv and w
b := a − ub7:obtain the tighter constraint r = ComputeMAP(w,b b)8:if mr> wb + br ≤ τb then9:
return solution ub and v
b
10:
else
11:
add the most violated constraint to the active set: Ws+1 := Ws ∪ {r }12:end if
13:
else
14:
compute the interpolation constant αs (Eq. 6.66)
15:
set vs+1 := (1 − αs )vs + αs vb16:
if there are blocking constraints then17:pick a blocking constraint r
18:
remove that blocking constraint from the active set: Ws+1 := Ws \ {r }19:
end if20:
end if
21: end for
22: output: ub and vbsolution, as derived by Nocedal and Wright (1999, p. 457):
vs,rαs := min 1,minbrr ∈Ws s.t. vs,r >vbr vs,r − v

.

(6.66)A constraint r is called blocking if vs,r > vbr . If there are blocking constraints, the minimizer r
above is picked and removed from the active set.
Each iteration of Algorithm 11 always improves the dual objective, and with a suitablestrategy to prevent cycles and stalling, the algorithm is guaranteed to stop after a finite
number of steps (Nocedal and Wright, 1999, Theo. 16.5). In practice, since it is run as a
subroutine of AD3 , we don’t need to run Algorithm 11 to optimality, which is particularly
convenient in early iterations of AD3 —note that this is enabled by Proposition 6.3, whichshows that global convergence is preserved as long as one makes the algorithm sufficiently
precise in later AD3 iterations. The ability of warm-starting each outer AD3 iteration withthe solution computed in the previous round is very useful in practice: we have observed
that, thanks to this warm-starting strategy, very few inner iterations are typically necessary
until the correct active set is identified.

6.6 Exact Inference with Branch-and-Bound
Finally, it is worthwhile to recall that AD3 , as just described, solves the LP-MAP relaxation of
the actual problem. In some problems, this relaxation is tight (in which case a certificate of
optimality will be obtained), but this is not always the case. When a fractional solution is ob-

126

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

tained, it is desirable to have a strategy to recover the exact MAP solution. Two observations
are noteworthy. First, as we saw in Section 4.6, the optimal value of the relaxed problem
LP-MAP provides an upper bound to the original problem MAP. In particular, any feasible dual point provides an upper bound to the original problem’s optimal value. Second,
during execution of the AD3 algorithm, we always keep track of a sequence of feasible dual
points (as guaranteed by Proposition 6.2, item 4). Therefore, each iteration constructs tighterand tighter upper bounds. We thus have all that is necessary for implementing a branchand-bound search that finds the exact solution of the ILP. The procedure works recursively as
follows:
1. Initialize L = −∞ (our best value so far).
2. Run Algorithm 8. If the solution ζ ∗ is integer, return ζ ∗ and set L to the objective value.
If along the execution we obtain an upper bound less than L, then Algorithm 8 canbe safely stopped and return “infeasible”—this is the bound part. Otherwise (if ζ ∗ isfractional) go to step 3.
3. Find the “most fractional” component of ζ ∗ (call it ζ ∗j (.)) and branch: for every y j ∈ Y j ,constrain ζ j (y j ) = 1 and go to step 2, eventually obtaining an integer solution ζ ∗ |y j or
infeasibility. Return the ζ ∗ ∈ {ζ ∗ |y j }y j ∈Yj that yields the largest objective value.
Although this procedure may have worst-case exponential runtime (which is not surprisingsince the problem is NP-hard), in many problems for which the relaxations are near-exact it
is found empirically very effective. We describe an example for frame-semantic parsing in
Section 6.7.4.

6.7 Experiments
We next provide an empirical comparison between AD3 (Algorithm 8) and four other approximate MAP inference algorithms, all of which address the LP-MAP problem:• Star-MSD (Sontag et al., 2011), an acceleration of the max-sum diffusion algorithm(Kovalevsky and Koval, 1975; Werner, 2007) based on star updates. This is an instanceof a block coordinate descent algorithm, as briefly mentioned in Section 4.6.1;• Generalized MPLP (Globerson and Jaakkola, 2008), the algorithm that we have described in detail in Section 4.6.1;• The projected subgradient algorithm for dual decomposition (Komodakis et al., 2007),
depicted as Algorithm 5 in Section 4.6.2;• The accelerated version of dual decomposition introduced by Jojic et al. 2010, which
we described briefly in the beginning of this chapter (Section 6.1).
All these algorithms address the LP-MAP problem; the first two use message-passing, performing block coordinate descent in the dual; and the last two are dual decomposition
algorithms. The accelerated dual decomposition algorithm is guaranteed to converge to ae-accurate solution after O(1/e) iterations, where e is a prescribed hyperparameter. The
primal and dual objectives are the same for all algorithms. All the baselines have the same

6.7. EXPERIMENTS

127

algorithmic complexity per iteration—this complexity is asymptotically the same as AD3 run
on a binarized graph, but is different from AD3 with the active set method. In all experimentswith AD3 , we set τ = 1.

6.7.1

Binary Pairwise MRFs

Figure 6.2 shows typical plots for an Ising model (binary pairwise MRF) on a random grid,with single node log-potentials chosen as θi (1) − θi (0) ∼ U[−1, 1] and mixed edge couplingsin U[−ρ, ρ], where ρ ∈ {0.5, 1, 1.5, 2}. Decompositions are edge-based for all methods. For
MPLP and Star-MSD, primal feasible solutions (ybi )i∈V were obtained by decoding the single
node messages, as in Globerson and Jaakkola (2008); for the dual decomposition methods
we use ybi = argmaxyi µi (yi ).
We observe that the projected subgradient is the slowest, taking a long time to find a“good” primal feasible solution, arguably due to the large number of components. The
accelerated dual decomposition method of Jojic et al. (2010) is also not competitive in thissetting, as it consumes many iterations before it reaches a near-optimal region.15 MPLP
performs slightly better than Star-MSD and both are comparable to AD3 in terms of convergence of the dual objective. However, AD3 outperforms all competitors at obtaining a“good” feasible primal solution in early iterations (it retrieved the true MAP in all cases, in≤ 200 iterations). We conjecture that this rapid progress in the primal is due to the penalty
term in the augmented Lagrangian (Eq. 6.11), which is very effective at pushing for a feasible
primal solution of the relaxed LP.

6.7.2

Multi-valued Pairwise MRFs

To assess the effectiveness of AD3 in the non-binary case, we evaluated it against the accelerated dual decomposition algorithm and MPLP in a Potts model (multi-valued pairwiseMRF) with single node log-potentials chosen as θi (yi ) ∼ U[−1, 1] and edge log-potentials as
θij (yi , y j ) ∼ U[−10, 10] if yi = y j and 0 otherwise. For the two baselines, we used the sameedge decomposition as before, since they can handle multi-valued variables; for AD3 , webinarized the graph as described in Section 6.4.3. Figure 6.3 (left) shows the best dual solutions obtained at each iteration for the three algorithms. We observe that MPLP decreases
the objective very rapidly in the beginning and then slows down. The accelerated dual decomposition algorithm has much better performance than in the binary case and manages
to converge faster (which matches the encouraging results obtained by Jojic et al. 2010), but
it is relatively slow to take off. AD3 has the best features of both methods.
We also evaluate AD3 in the original graph, employing the active set method describedin Section 6.5 to tackle the quadratic subproblems. We set the maximum number of inner
iterations in the active set method to 10, and we compared the performance with AD3 runin the binarized graph. The result is shown in Figure 6.3 (right). Since now the iterations
are not directly comparable (since AD3 with the active set method needs to do more work
per iteration), we plot the objectives over runtime. The variant of AD3 with the active set
method is clearly superior.
15 It is conceivable that the early iterations could make faster progress by annealing e. Here we have just usedthe variant described by Jojic et al. (2010).

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION128

ρ = 1.0

Objective

Edge Coupling ρ = 0.5

Star−MSD dual
MPLP dual

676

359.2

Subgrad. dual

675

359.15

674

359.1

673

Acc. DD dual
AD3 dualStar−MSD primal

672
0

100

200
300
Number of Iterations

400

500

100

200

300

400

500

ρ = 2.0

ρ = 1.5

958

1333

956

1332.5954

Subgrad. primal
Acc. DD primalAD3 primal

1333.5960

MPLP primal

1332
100

200

300

400

500

100

200

300

400

500

Figure 6.2: Results for 30 × 30 random Ising models with several edge couplings. We plot thedual objectives and the best primal feasible solution at each iteration. For the subgradientmethod, we set ηt = η0 /t, with η0 yielding the maximum dual improvement in 10 iterations,with halving steps (those iterations are not plotted). For accelerated dual decomposition,
we plot the most favorable e ∈ {0.1, 1, 10, 100}. For AD3 , we set η = 5.0 and τ = 1.0. All
decompositions are edge-based.

6.7.3

Protein Design

We next evaluate AD3 in benchmark protein design problems, using the dataset of Yanover
et al. (2006). In these problems, the input is a 3D shape, and the goal is to find the moststable sequence of amino-acids in that shape. The problem can be represented as a pairwise
Markov network whose variables denote the identity of amino-acids and rotamer configurations, yielding hundreds of possible states for each node.16
We compare the performance of AD3 with MPLP, using David Sontag’s implementation.17 Figure 6.4 shows the progress in the dual objective over runtime, for two of the mostlargest problem instances. In both cases, MPLP steeply decreases the objective at first, butthen reaches a plateau and eventually halts, with no significant improvement over consecutive iterations. AD3 rapidly surpasses MPLP in getting a better dual objective.18 For AD3 ,we use the active set method for solving the subproblems, which led to a better performance
than the graph binarization technique.

6.7.4

Frame Semantic Parsing

Finally, we report experiments on a natural language processing task which involves logic
constraints: frame-semantic parsing, using the FrameNet lexicon (Fillmore, 1976). For each
16 The protein design data set is available from http://www.jmlr.org/papers/volume7/yanover06a/

Rosetta_Design_Dataset.tgz.
17 Available at http://cs.nyu.edu/ dsontag/code/mplp_ver1.tgz. Note that Sontag’s code imple˜ments a tightening approach that is able to retrieve the true MAP (Sontag et al., 2008); here, we only run theirinitial set of iterations that solves the LP-MAP relaxation, the same problem that AD3 addresses.
18 Note however that the ultimate goal of this task is to get a good primal solution. Experimentally, we observedthat MPLP often gets better primal solutions than AD3 , regardless being consistently worse in optimizing thedual. However, the ability of quickly decreasing the dual is very desirable in tightening approaches, where larger
factors are iteratively added to the model to tighten the relaxation. It is likely that a tightened AD3 would obtaina competitive performance in this task.

6.7. EXPERIMENTS

MPLP
Acc. DD
AD3 with binarization
Dual Opt.

2020
2015
2010

AD3 with binarization
AD3 with active set method

2210
Dual Objective

Dual Objective

2025

129

2200
2190
2180
2170
2160
2150

2005
0

200

400
600
Number of Iterations

800

0

1000

1

2

3
4
5
Runtime (sec.)

6

7

Figure 6.3: Left: Results for a random 20 × 20 random Potts model with 8-valued nodes andcoupling factor ρ = 10. We plot the dual objectives, and the value of the true dual optimum.
For the accelerated dual decomposition algorithm, we set e = 1.0; for AD3 , we set η = 0.5.
Right: Comparison between AD3 with the active set method, and AD3 run on the binarizedgraph, for a random model of the same kind. In both cases, we set η = 1, the most favourable
choice in {0.01, 0.1, 1, 10}. For the active set method, we limited the maximum number of
inner iterations to 10.
MPLP
every 100 iterations
3
AD
every 100 iterations

455
Dual Objective

Protein Design (1kw4)

450
445
440
435

Dual Objective

Protein Design (1fpo)

MPLP
every 100 iterations
3
AD
every 100 iterations

160

155

150

430
1000

2000

3000 4000 5000
Runtime (secs.)

6000

7000

500

1000
Runtime (secs.)

1500

2000

Figure 6.4: Experiments on two of the largest protein files in the protein design dataset of
Yanover et al. (2006), with 3167 (1fbo) and 1163 (1kw4) factors. The results are representativeof the performance obtained in other protein files. In AD3 , we self-adjust η as proposed
by Boyd et al. (2011), with an initial value of η = 1.0. We also show the iterations of bothmethods. Observe that the greater cost per iteration in AD3 is amortized in later iterations,since we can take advantage of warm-starting in the active set method.
instance, the goal is to predict the set of arguments and roles for a predicate word in a
sentence, while respecting several constraints about the frames that can be evoked. The
resulting graphical models are binary constrained factor graphs with XOR and AtMostOne
factors.19
Figure 6.5 shows a comparison between AD3 , MPLP and the projected subgradient algorithm for the five most difficult examples, the ones in which the LP relaxation is not tight.
Again, AD3 outperforms all the competitors. Since these examples have a fractional LP-MAP
solution, we applied the branch-and-bound procedure described in Section 6.6 to obtain theexact MAP for these examples. The whole dataset contains 4,462 instances, which wereparsed by this exact variant of the AD3 algorithm in only 4.78 seconds, against 43.12 secondsof CPLEX, a state-of-the-art commercial ILP solver.
In the next chapter (Sections 7.4–7.5), we will illustrate the application of AD3 for dependency parsing, which we tackle as a constrained inference problem. We defer to Section 7.5 thedetailed empirical analysis of AD3 in that task. We will also analyze runtime and describe
19 We refer to the recent thesis by Das (2012, Sect. 5.5) for more details on this task.

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

130

Dual Objective

Frame−Semantic Parsing
26.1

42.18

26

42.16

25.9

42.14

25.8

200 400 600 800 1000
Number of Iterations

61.4
61.2
61
60.8
60.6

42.12
0

20.4
20.3
20.2
200

400

600

800

20.1

100

66

MPLP

65

Subgrad.

64

AD3

200

300

400

500

63
200

400

600

800 1000

200

400

600

800 1000

Figure 6.5: Experiments in frame-semantic parsing, using models described in Das (2012).
We selected the five most difficult examples, which have between 321 and 884 variables,and between 32 and 59 factors. Unlike MPLP and the projected subgradient algorithms,
which did not converge after 1000 iterations, AD3 achieves convergence in a few hundreds ofiterations for all but one example. For the projected subgradient algorithm, we set ηt = η0 /t,
with η0 = 1.0 being the most favorable choice for all examples. For AD3 , we used theprocedure described above to self-adjust η.
how one may greatly benefit from caching and warm-starting the subproblems.

6.8 Future Work
The contributions presented in this chapter open up several directions for future research.
The active set method requires only a black box for computing the local MAP for eachfactor. By taking larger components of the graph, such as sequences or trees, for which MAP
inference is tractable, we can reduce substantially the number of subproblems, which maylead to speed-ups. In addition, there are many problems which are characterized by combinatorial subproblems, such as the composition of head automata with maximum spanning
tree algorithms for dependency parsing (Koo and Collins, 2010), as well as several problems
in parsing and machine translation (Auli and Lopez, 2011; Rush and Collins, 2011; Chang
and Collins, 2011). The active set method can be applied to such problems, using combinatorial algorithms for local MAP computations. It would be interesting to compare AD3
with the projected subgradient algorithm in the number of oracle calls, for a broad range ofproblems. We leave this for future work.
It could also be interesting to try other algorithms for solving the quadratic problems,
instead of the active set algorithm. Interior point methods could be a good fit for some problems; however they usually fail to capture the sparsity. Other options are cyclic projection
algorithms and the smoothening method of Nesterov (2005); the latter would require a blackbox for computing marginals (rather than MAP).
Another idea is to replace the quadratic penalty of ADMM by a general Bregman divergence, in an attempt to obtain easier subproblems. We point out that an entropic penalty
would not lead to the same subproblems as in Jojic et al. (2010): it would lead to the problemof minimizing non-strictly convex free energies with different counting numbers. Althoughextensions of ADMM to Bregman penalties have been considered in the literature, to the bestof our knowledge, convergence has been shown only for quadratic penalties (see also Boyd

6.8. FUTURE WORK131

et al. 2011, p. 24). Quadratic penalties also have the important advantage that they lead tosparse solutions, which can be exploited algorithmically (as we do in our active set method).
The convergence proofs, however, can be extended to Mahalanobis distances, since they correspond to an affine transformation of the subspace defined by the equality constraints of
Eq. 6.2. Simple operations, such as scaling these constraints, do not affect the algorithms thatare used to solve the subproblems. Therefore, we can generalize AD3 by adding a a numberof free parameters to specify these scalings; tweaking those parameters might eventuallylead to even faster convergence.
Note also that AD3 is dual decomposable, hence the subproblems can all be solved inparallel. There is a good chance that significant speed-ups may be achieved in large problems
through parallelization, for example in multi-core architectures, or through GPU programming. This sort of techniques has been applied with great success to other message-passing
algorithms, giving rise to parallel frameworks such as GraphLab20 (Low et al., 2010). A significant amount of computation can also be saved by caching the subproblems, which tend
to become more and more similar across later iterations. We will see instances of this in
Chapter 7.
The branch-and-bound algorithm for obtaining the exact MAP deserves further experimental study. One advantage of AD3 over other algorithms for solving the LP-MAP problem
is that AD3 is able to quickly produce sharp upper bounds, which is a useful property whenembedded in a branch-and-bound procedure. For many problems, there are effective rounding procedures that can also produce lower bounds, which can be exploited for guiding thesearch. There are also alternatives to branch-and-bound, for example tightening proceduresof the same kind as Sontag et al. (2008), which keep adding larger factors to decrease theduality gap. The variant of AD3 with the active set method can be used for handling theselarger factors.
Interestingly, the AD3 algorithm, unlike other message-passing and dual decomposition
algorithms, does not have any connection at all with dynamic programming, nor it is likelythat dynamic programming algorithms can be useful for solving its quadratic subproblems.
These quadratic subproblems are interesting per se, as they can also arise in other proximal
point methods in which Euclidean penalties are used. Therefore, investigating efficient procedures for projecting onto the marginal polytopes of other factors can be a fruitful line ofresearch. In particular, AD3 can be adapted to solve other problems than LP-MAP, in which
the objective function may not be linear. For example, if we change the objective functionof LP-MAP (or MAP) by incorporating an Euclidean penalty of the form ∑i∈V ∑yi ∈Yi µi (yi ),this will promote a sparse solution, in which some “marginals” will be integer, indicating afactor configuration, while others will be fractional, implicitly representing a list of outputs
with non-negligible probability. This can be useful for developing pruning models, as an alternative to typical approaches based on hard thresholding or K-best lists. Note that, for thiskind of problem, the AD3 algorithm can be employed with minimal changes and the same
subproblem solvers, since all subproblems are still quadratic.

20 Available at http://graphlab.org/.

132

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

6.9 Discussion
In this chapter, we introduced AD3 , a new LP-MAP inference algorithm based on the alternating direction method of multipliers of (Glowinski and Marroco, 1975; Gabay and Mercier,1976). AD3 maintains the modularity of dual decomposition methods, but achieves fasterconsensus, by penalizing, for each subproblem, deviations from the current global solution.
We established the convergence properties of AD3 , blending general results for ADMM
algorithms obtained by Glowinski and Le Tallec (1989); Eckstein and Bertsekas (1992); He
and Yuan (2011); Wang and Banerjee (2012). Importantly, AD3 converges to an e-accurate
solution with an iteration bound of O(1/e), like the accelerated dual decomposition methodof Jojic et al. (2010).
We have shown how the AD3 algorithm can be applied to the constrained factor graphs
introduced in the previous chapter, by deriving efficient procedures for projecting onto themarginal polytopes of the hard constraint factors introduced in Section 5.3. This paves theway for using AD3 in problems with declarative constraints. Up to a logarithmic term, thecost of projecting on those factors is asymptotically the same as that of passing messages.
We also provided a closed-form solution of the AD3 subproblem for pairwise binary factors.
We suggested two approaches for dealing with factors involving multi-valued variables:
one is binarizing the graph, as described in Section 5.5.1, and reverting to the already known
projection procedures. The other is a new active set method, which requires only an oracle thatcomputes the local MAP (the same requirement as the projected subgradient method discussed in Section 4.6.2). The active set method seems more adequate than other algorithms
for quadratic programming, since it deals well with sparse solutions—which are guaranteedin this problem, as we show in Proposition 6.6—and it can take advantage of warm starting.
Experiments in synthetic and real-world datasets have shown that AD3 is able to solve
the LP-MAP problem more efficiently than other methods for a variety of tasks, including
synthetic problems involving Ising and Potts grid models, as well as protein design andframe-semantic parsing problems. We also show how AD3 can be wrapped in a branch-andbound procedure to retrieve the true MAP, rendering the method exact. This approach looks
particularly useful for problems in which the LP-MAP relaxation is not very loose, whichhappens with the frame-semantic parsing application that we use as a testbed. In the next
chapter, we will include additional experiments in dependency parsing, where we show thatAD3 can handle a large number of factors better than other algorithms. We will also see how
the algorithm can benefit from caching the subproblems.
During the preparation of this thesis, some related methods have been proposed in the literature. Meshi and Globerson (2011) also propose applying ADMM to inference in graphicalmodels, although they insert the quadratic penalty in the dual problem (the one underlying
the MPLP algorithm) rather than in the primal. Quite recently, Yedidia et al. (2011) proposed
the divide-and-concur algorithm for decoding of low-density parity check codes (LDPC),
which shares resemblances with AD3 .21 Even more recently, Barman et al. (2011) applies analgorithm analogous to AD3 for the same LDPC decoding problem, by deriving an efficientalgorithm for projecting onto the parity polytope. It would be interesting to compare theperformance of this specialized projection algorithm with our active set method.22
21 Their approach can be seen as non-convex ADMM.
22 Recall that our active set method requires only a black box for solving the local MAP problem. In the case

6.9. DISCUSSION

133

of the parity polytope this can be done in linear time, while projecting onto the polytope takes O( L log L) time,using the algorithm of Barman et al. (2011).

134

CHAPTER 6. ALTERNATING DIRECTIONS DUAL DECOMPOSITION

Chapter 7

Turbo Parsers: Linear Programming
Relaxations for Dependency Parsing
In this chapter, we introduce and characterize turbo parsers1 —dependency parsers that run
approximate inference in a loopy factor graph, ignoring the global effects caused by the loops.
Several parsers fall into this class, most noticeably the ones proposed by Smith and Eisner(2008), Martins et al. (2009b, 2010f, 2011c) and Koo et al. (2010). All these can be regarded
as approximate inference algorithms over a loopy factor graph. In this chapter, we give
special emphasis to parsers built from linear relaxations (i.e., LP-MAP inference, as describedin Section 4.6). The contributions presented on this chapter are:
• We consider new models for dependency parsing that can accommodate a wide rangeof global features (Figure 7.1).
• To perform inference under such models, we derive a new concise ILP formulation using multi-commodity flows. By “concise” we mean that the size of the program (number of variables and constraints) grows polynomially with the length of the sentence,
an improvement over previous work, for which this growth is exponential (Riedel and
Clarke, 2006).
• We show that the resulting parser, once a linear relaxation is made, is an instance ofa turbo parser. In doing so, we establish some equivalence results with other turbo
parsers (Smith and Eisner, 2008; Koo et al., 2010), by characterizing the underlying
factor graphs and the optimization problems that are addressed in the corresponding
inference algorithms.
• We show how to apply the AD3 algorithm (introduced in Chapter 6) to dependency
parsing, allowing us to exploit the structure of the linear program described by thefactor graph. We also describe how certain computational tricks (such as caching)enable important speed-ups.
We show the robustness of our parsers by providing an empirical evaluation for 14 languages, with state-of-the-art results. We provide some error analysis, and compare the per1 The name stems from “turbo codes,” a class of high-performance error-correcting codes introduced byBerrou et al. (1993) for which decoding algorithms are equivalent to running belief propagation in a graphwith loops (McEliece et al., 1998).

135

CHAPTER 7. TURBO PARSERS

136

formance of the AD3 algorithm with the projected subgradient algorithm and a commercial
LP solver (CPLEX).
The concise ILP formulation and the use of linear relaxations in statistical parsing wasintroduced in Martins et al. (2009b); this chapter extends that paper with some new material:
new parts for head bigrams and for directed paths, and a tighter relaxation built from thelinear program of head automata. The unified view of these and other parsers as “turbo
parsers” was established in Martins et al. (2010f). The AD3 algorithm was originally applied
to this problem in Martins et al. (2011c).

7.1 Motivation and Related Work
The motivation for introducing turbo parsers comes from two important observations withrespect to previously existing linear models in statistical parsing, such as the ones we havedescribed in Chapter 2:• Models that permit exact inference are usually forced to make unrealistic independence
assumptions, or to have a stringent decomposition into parts, for the sake of tractability.
As a result, these models are strongly limited in the amount of output context that canbe encoded in the features, becoming under-expressive. This is the case, e.g., of arcfactored models for dependency parsing (Eisner, 1996; McDonald et al., 2005b), and of
first-order probabilistic context-free grammars (Charniak, 1997; Johnson, 1998).
• On the other hand, empirical performance typically improves whenever one breaks
these strong locality assumptions, even though this has the common downside of making exact inference very costly or intractable. In practice, approximate decoders using
greedy search, beam-search, or sampling methods seem to lead to good accuracyruntime tradeoffs, and search error is not usually seen as a problem (McDonald et al.,2006; Huang, 2008; Finkel et al., 2008). However, many of these methods are either
heuristic in nature or the underlying approximation is not well understood.
The recent advances in the graphical models literature, summarized in Chapter 4, area natural fit to statistical parsing. In fact, the approximate inference algorithms described
therein have important advantages over other traditional methods: firstly, the nature of theapproximation is reasonably well understood, and secondly, they often come with guarantees or optimality certificates. Given our achievements already described in Chapters 5 and
6, we are well equipped to devise feature-rich statistical parsing, without a great deal ofconcern about the locality of the features:• We need not worry about hard constraints (inherent in any parsing problem), since wehave already described in Chapter 5 how this machinery extends to constrained factor
graphs.
• We have proposed in Chapter 6 the AD3 algorithm, which solves the LP-MAP problem
and is suitable for constrained models.
Our formulations are very flexible and can accommodate a wide range of non-arc-factored
parts. In our experiments, we encode features involving arcs, siblings (consecutive or arbitrary), grandparents, head bigrams, directed paths, and non-projective arcs; a graphic representation of these parts is shown in Figure 7.1. Many extensions are possible and relatively

7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM

137

Figure 7.1: Parts used by our parser. Arcs are the basic parts: any dependency tree canbe “read out” from the arcs it contains. Consecutive siblings and grandparent parts introduce
horizontal and vertical Markovization (McDonald et al., 2006; Carreras, 2007). We break thehorizontal Markov assumption via all siblings parts (which look at arbitrary pairs of siblings,
not necessarily consecutive) and the vertical one through parts which indicate a directed path
between two words. Inspired by transition-based parsers, we also adopt head bigram parts,
which look at the heads attached to consecutive words. Finally, we have parts which indicate
if an arc is non-projective (i.e., if it spans words that do not descend from its head).

simple to include. For example, if expert knowledge is available, one can include additional
hard or soft first-order logic constraints, as typically done in Markov logic networks andconstrained conditional models (Richardson and Domingos, 2006; Chang et al., 2008).
This chapter is organized as follows: in Section 7.2, we derive a concise ILP formulation
for dependency parsing. In Section 7.3, we proceed to show how the resulting parser, once a
linear relaxation is made, is an instance of a turbo parser, by deriving its underlying graphical model; we also characterize other turbo parsers, by writing down their optimization
problems. This paves the way for Section 7.4, where we show how to apply the AD3 algorithm (introduced in Chapter 6) to dependency parsing. Section 7.5 provides an empiricalevaluation on data in 14 languages, along with a qualitative error analysis and a comparison
with competing algorithms. Section 7.6 concludes and points directions for future work.

7.2 Dependency Parsing as an Integer Linear Program
Much attention has recently been devoted to integer linear programming formulations of
NLP problems, with interesting results in applications like semantic role labeling (Roth andYih, 2005; Punyakanok et al., 2004), machine translation (Germann et al., 2001), word alignment (Lacoste-Julien et al., 2006), summarization (Clarke and Lapata, 2008), and coreference
resolution (Denis and Baldridge, 2007), among others. The rationale for the development of
ILP formulations is to incorporate non-local features or global constraints, often difficult tohandle with traditional dynamic programming algorithms. ILP formulations focus on the
declarative representation of problems, rather than procedural design. While solving an ILP
is NP-hard in general, fast solvers exist today that make it a practical solution for many NLP
problems.
In this section, we present new concise ILP formulations for projective and non-projective
dependency parsing. By focusing on the modeling part and abstracting away from algo-CHAPTER 7. TURBO PARSERS

138

rithm design, our formulations pave the way for efficient exploitation of global features andconstraints in parsing applications, leading to models which are more powerful than others
considered in the literature.
We point out that casting dependency parsing as an ILP is not new: Riedel and Clarke(2006) were the first to come up with an ILP formulation, which had exponentially many
constraints; they tackle the problem using a cutting-plane method. However, this approach
appears to be too slow for practical purposes, hence efficient formulations have remained an
open problem, which we now address.
Our formulations offer the following comparative advantages:• The numbers of variables and constraints are polynomial in the sentence length, as
opposed to requiring exponentially many constraints, eliminating the need for incremental procedures like the cutting-plane algorithm;• LP relaxations permit fast online discriminative training of the constrained model;• Soft constraints may be automatically learned from data. In particular, our formulations handle higher-order arc interactions (like siblings and grandparents), look at theheads of consecutive words in the sentence, model word valency, and can learn to favornearly-projective parses whenever this is observed in the data.
• We handle interactions that cannot be captured by horizontal and vertical Markovization without an explosion of the number of states. For example, we include scores forarbitrary sibling arcs (not necessarily consecutive) and for pairs of words which have a
directed path in the dependency tree.
Many other extensions are possible and relatively simple to include. This will become clear
when we represent our formulations as constrained factor graphs, in Section 7.3. We postpone to Section 7.5 the experimental analysis of our turbo parsers on standard parsing tasks.

7.2.1

The Arborescence Polytope

At the heart of our formulation lies a concise characterization of the arborescence polytope,
which we introduce in this section. For basic definitions concerning dependency parsing,projective and non-projective arcs and trees, we refer to the background material presented
in Section 2.3.
Recall the definition of a dependency tree (Definition 2.2). Let x := ( x0 , x1 , . . . , x L ) be asentence with L words, where x0 := ∗ is a special symbol for the designated root. We willreason about the entire set Y( x ) of potential dependency parse trees for x.
Let us first regard each token in {0, . . . , L} as a node in a fully connected directed graph(or digraph) D = (N, A); i.e., N = {0, . . . , L} is the set of nodes, and A = N2 is the set of
all possible arcs connecting the nodes.2 Using terminology from graph theory, we say thatB ⊆ A is an r-arborescence of the directed graph D (also called a “directed spanning treeof D with designated root r”) if (V, B) is a (directed) tree rooted at r. This is illustrated
2 The general case where A ⊆ N2 is also of interest; it arises whenever a constraint or a lexicon forbids somearcs from appearing in dependency tree. It may also arise as a consequence of a first-stage pruning step wheresome candidate arcs are eliminated; a practical instance of this will be described in the experimental section(Section 7.5).

7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM

139

*

*

We

learnedalesson

in

1987

about

volatility

learnedWe

lesson

in

about
1987

a

volatility

Figure 7.2: The dependency tree plotted in Figure 2.10, and its depiction as an arborescenceof the fully connected graph connecting all words.

Figure 7.3: The arborescence polytope Ztree . Each vertex corresponds to a dependency parsetree in the output set Y( x ).
for a particular sentence in Figure 7.2. From Definition 2.2, we have that the set Y( x ) oflegal dependency parse trees of x is in bijection with the set of 0-arborescences of D—i.e.,each arborescence can be seen as a potential dependency tree for x. We can thus regard thedependency parsing problem as a search problem in the set of arborescences of D.
We next provide a polyhedral characterization of this set of arborescences. Given a dependency tree y ∈ Y( x ), we define its incidence vector z ∈ R|A| as:z : = ( z a ) a ∈A ,with z a = [[ a ∈ y]];(7.1)

that is, z is a binary vector indexed by all potential arcs, where each component z a is 1 if arc
a is in the tree y, and 0 otherwise. The set of all incidence vectors of the trees in Y( x ) is a setof points in {0, 1}|A| . Its convex hull is a polyhedron called the arborescence polytope, which
we denote by Ztree . Since all incidence vectors lie in {0, 1}|A| , none can be expressed as aconvex combination of the others; therefore they are precisely the vertices of Ztree . In otherwords, each vertex of the arborescence polytope Ztree can be identified with a dependency
tree in Y( x ). Figure 7.3 provides a schematic representation.
So far, the fact that we have been able to characterize our output set geometrically as theset of vertices of a polytope did not buy us much—we still have an exponential number ofvertices to represent. It is desirable to represent the arborescence polytope in terms of a setof linear inequalities. The Minkowski-Weyl theorem (Rockafellar, 1970) guarantees that sucha representation must exist—i.e., there must be a P ∈ N, some P-by-|A| matrix A, and some
vector b in RP such that Ztree can be represented as
Ztree = {z ∈ R|A| | Az ≤ b}.

(7.2)

However, it is not obvious how to obtain a concise representation—one in which the numberof linear inequalities P grows only polynomially with the number of words L. This is theCHAPTER 7. TURBO PARSERS

140

subject of the following sections. We will describe three possible representations of Ztree :• a cycle-based representation, where P has exponential growth with L; this was the representation implicitly used by Riedel and Clarke (2006);• a single-commodity flow representation requiring O( L2 ) variables and constraints—this is
only an approximate representation, as it defines an outer bound of Ztree and not Ztree
itself;• a multi-commodity flow representation requiring O( L3 ) variables and constraints, andwhich we will show to be exact.
The last two representations are inspired by the commodity flow models described by Magnanti and Wolsey (1994) in the combinatorial optimization literature; while Magnanti and
Wolsey (1994) addressed the undirected minimum spanning tree problem, their analysis carries over to our case, where arcs are directed.3

7.2.2

A Cycle-Based Representation

Recall the digraph D = (N, A) defined in the previous section. Clearly, for an arbitrary
subset of arcs B ⊆ A to be a 0-arborescence of D it is necessary and sufficient that it touches
all the nodes, and that the overall structure is a directed tree rooted at 0, i.e., a graph free ofcycles where 0 is the only node without a parent. Formally:
Proposition 7.1 A subgraph y = (N, B) is a legal dependency tree (i.e., y ∈ Y( x )) if and only if thefollowing conditions are met:
1. Each node in N \ {0} has exactly one incoming arc in B,
2. 0 has no incoming arcs in B,
3. B does not contain cycles.
Proof. Straightforward from the definition of directed trees and arborescences.
For each node n ∈ N, let δ− (n) := {(i, j) ∈ A | j = n} denote its set of incoming arcs,and δ+ (n) := {(i, j) ∈ A | i = n} denote its set of outgoing arcs. The two first conditions
can be easily expressed by linear constraints on the incidence vector z:∑ za = 1, for every j ∈ N \ {0},

(7.3)

∑

(7.4)

a∈δ− ( j)z a = 0.

a ∈ δ − (0)The third condition is somewhat harder to express. We need to enumerate each possible
cycle and forbid all configurations that contain that cycle. Denote by C ⊆ 2A the set of all
3 A similar representation has been proposed by Wong (1984). See Schrijver (2003, p. 903) for other pointersto the literature.

7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM

141

possible cycles, where each cycle C ∈ C is represented by the set of arcs it contains (hence C
is a subset of A). The following constraints do the job:∑ za ≤ |C| − 1, for every C ∈ C .

(7.5)

a ∈C

Unfortunately, the set C grows exponentially fast with L, which yields an exponential number of constraints of the form in Eq. 7.5. To obviate this problem, Riedel and Clarke (2006),
when optimizing over the constraint set defined by Eqs. 7.3–7.5, employed a cutting-plane
method, where constraints are added incrementally as violations are detected. The resulting
algorithm is still slow, and an arc-factored model is used as a surrogate during training (i.e.,the hard constraints are only used at test time), which implies a discrepancy between themodel that is optimized and the one that is actually going to be used. The next sections
propose ILP formulations that eliminate the need for cycle constraints; in fact, they require
only a polynomial number of constraints.

7.2.3

A Single Commodity Flow Representation

The reason why the cycle-based representation is not concise is that it requires to forbid
every cycle, in accordance to condition 3 of Proposition 7.1. However, there is an alternative
condition, as we state next.
Proposition 7.2 A subgraph y = (N, B) is a legal dependency tree (i.e., y ∈ Y( x )) if and only if itsatisfies conditions 1 and 2 of Proposition 7.1 plus the following replacement of condition 3:
30 . B is connected. In other words, there is a directed path from the root 0 to every other node in
N \ {0} consisting only of arcs in B.
Proof. Clearly, any directed tree which spans {0, . . . , L} must satisfy conditions 1 and 2 andbe connected, hence it also satisfies 30 . It suffices to show that conditions 1, 2 and 30 imply
condition 3 of Proposition 7.1, i.e., that these conditions prevent cycles. We prove this bycontradiction. Let y satisfy 1, 2 and 30 . Conditions 1 and 2 imply that there are L arcs in B.
Suppose that y has a cycle C of length L0 ; then there are L0 arcs in C, covering L0 nodes, and
L − L0 arcs in B \ C. In order for B be connected, B \ C must cover the remaning L − L0 + 1
nodes plus at least one node in C, a total of L − L0 + 2 nodes. This cannot be done with only
L − L0 arcs while maintaining connectivity within B \ C.
Proposition 7.2 states that conditions 1-2-3 are equivalent to 1-2-30 , in the sense that bothdefine the same set Y( x ). However, as we will see, the latter set of conditions is moreconvenient. Connectedness of graphs can be imposed via flow constraints, by requiring that,for any n ∈ N \ {0}, there is a directed path in B connecting 0 to n.
We first describe an inexact single commodity flow formulation that requires only O( L2 )variables and constraints. Our formulation adapts the one described by Magnanti and
Wolsey (1994, p. 39), originally proposed for the undirected minimum spanning tree problem. Under this model, the root node must send one unit of flow to every other node. Bymaking use of extra variables, φ := (φa ) a∈A , to denote the flow of commodities through eacharc, we are led to the following constraints in addition to Eqs. 7.3–7.4:

CHAPTER 7. TURBO PARSERS142• Root sends flow L:∑

φa = L.

(7.6)∑ φa − ∑+ φa = 1, for every j ∈ N \ {0}.

(7.7)

a ∈ δ + (0)• Each node consumes one unit of flow:a∈δ− ( j)a∈δ ( j)• Flow is zero on disabled arcs:
φa ≤ Lz a , for every a ∈ A.

(7.8)

• Each arc indicator lies in the unit interval:z a ∈ [0, 1], for every a ∈ A.

(7.9)

These linear constraints project an outer bound of the arborescence polytope, i.e.,
Zsc :=⊇nz ∈ R| A| | ∃φ : (z, φ) satisfy Eqs. 7.3–7.4 and Eqs. 7.6–7.9

Ztree .

o(7.10)

Furthermore, the integer points of Zsc are precisely the incidence vectors of dependency treesin Y( x ); these are obtained by inserting integer constraints, or equivalently by replacing
Eq. 7.9 byz a ∈ {0, 1}, for every a ∈ A.

(7.11)A representation as in Eq. 7.10 is called a lifted representation, in the sense that it involves
extra variables (in this case, the flows φ) to build a higher-dimensional polytope which isthen projected onto the original z-space (see the formal definition of lifting in Appendix B).

7.2.4 A Multi-Commodity Flow Representation
We next present a multi-commodity flow representation of the arborescence polytope, which isstill polynomial (even though it has cubic rather than quadratic dependency on the sentence
length L), but has the advantage of being exact. Like the single-commodity representation,
it relies on the characterization of arborescences in terms of a connectedness constraint,
as put forth by Proposition 7.2. It is adapted from the multicommodity directed flow model of
Magnanti and Wolsey (1994, p. 44), albeit not exactly copied, since the latter is for undirectedspanning trees. We prove in Proposition 7.3 that this representation is exact.
In the multi-commodity flow model, every node k 6= 0 defines a commodity: one unit ofcommodity k originates at the root node and must be delivered to node k; the variable φijk
denotes the flow of commodity k in arc (i, j). This model differs from the single commodity
model by replacing the constraints in Eqs. 7.6–7.9 by the following ones in Eqs. 7.12–7.15:

7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM

143• The root sends one unit of commodity to each node:

∑−

φak −

a ∈ δ (0)∑+φak = −1,for every k ∈ N \ {0}.

(7.12)a ∈ δ (0)• Any node consumes its own commodity and no other:∑− φak − ∑+ φak = [[ j = k]],a∈δ ( j)for every j, k ∈ N \ {0}.

(7.13)a∈δ ( j)• Disabled arcs do not carry any flow:for every a ∈ A and k ∈ N.

φak ≤ z a ,(7.14)• All variables lie in the unit interval:z a ∈ [0, 1],

φak ∈ [0, 1],for every a ∈ A and k ∈ N.

(7.15)

Consider the polytope obtained by imposing the constraints in Eqs. 7.3–7.4 and Eqs. 7.12–
7.15 and projecting out the flow variables φ:
Zmc :=noz ∈ R| A| | ∃φ : (z, φ) satisfy Eqs. 7.3–7.4 and Eqs. 7.12–7.15 .

(7.16)We then have the following:
Proposition 7.3 The multi-commodity flow representation is exact:
Zmc = Ztree .

(7.17)

Proof. Let S ∈ N be an arbitrary subset of the nodes. We denote by δ+ (S) := {(i, j) ∈ A | i ∈S, j ∈/ S} the set of arcs which depart from S. The set of these arcs form a cut of the graph,associated with the partition S ∪ (N \ S). One way of imposing connectedness is by requiring
every possible cut to have at least one arc installed. This can be expressed by the following
set of constraints:∑z a ≥ 1,for every proper subset S ⊂ N s.t. 0 ∈ S.

(7.18)a ∈ δ + (S)Magnanti and Wolsey (1994, Proposition 5.2) show that the constraints in Eq. 7.18, alongwith Eqs. 7.3–7.4 and the non-negativity constraint z ≥ 0, are an exact representation of
the arborescence polytope Ztree .4 We show that our multi-commodity flow representation isequivalent to that set of constraints, by applying the min-cut max-flow theorem (Schrijver,
2003). For every node k ∈ N \ {0}, consider the set Sk of all possible proper subsets S ⊂ N
s.t. 0 ∈ S and k ∈/ S. Then, express all constraints in Eq. 7.18 as L non-linear constraints, one
per k ∈ N \ {0}:min ∑ z a ≥ 1.
(7.19)S ∈Sk

a ∈ δ + (S)

4 Note that there are exponentially many constraints in Eq. 7.18, one per each proper subset of nodes, hence

that formulation, albeit exact, is not practical.

CHAPTER 7. TURBO PARSERS

144

Note that this inequality is actually an equality, since for the set S = N \ {k }, the set δ+ (S)equals δ− (k) and from the single parent constraint (Eq. 7.3) we have ∑ a∈δ− (k) z a = 1. Hence,
we can write insteadmin ∑ z a = 1.
(7.20)S ∈Sk

a ∈ δ + (S)Now, for a fixed z, we can interpret the minimization in Eq. 7.20 as a min-cut problem withsource 0 and sink k, where the capacity of each arc a is given by z a . The constraint statesthat the min-cut is 1; from the min-cut max-flow theorem, this value must equal the solutionof the equivalent max-flow problem; the capacity and the conservation-of-flow constraintsof that max-flow problem are precisely the constraints that we have written in Eqs. 7.12–
7.14. Putting all the pieces together, we have that the multi-commodity flow representation
is equivalent to the exact cut-based representation with the constraint in Eq. 7.18; therefore
it is also exact.
To sum up, we have introduced multi-commodity flow constraints in Eqs. 7.12–7.14 which
provide a lifted representation of the arborescence polytope Ztree . By Proposition 7.3, thisrepresentation is exact, without the need of integer constraints. We will exploit this fact inthe sequel.

7.2.5

Arc-Factored Model

Now that we have a polyhedral representation of the arborescence polytope, it is straightforward to formulate arc-factored dependency parsing as an LP.
At first sight, this may look like a futile exercise: as we have seen in Section 2.3, thearc-factored model, which is rather simplistic, already permits efficient parsing with graphbased algorithms—asymptotic runtimes are O( L3 ) with the Chu-Liu-Edmonds’ algorithm(Chu and Liu, 1965; Edmonds, 1967), or O( L2 ) with Tarjan’s or Gabow’s algorithm (Tarjan,
1977; Gabow et al., 1986). In fact, the O( L2 ) bound is optimal, since any algorithm needs
to touch every potential arc, and there is a quadratic number of them.5 Hence, there is noadvantage at all in casting arc-factored parsing as a linear program and employing an off-theshelf solver, as those asymptotic runtimes are unbeatable, and gains in the non-asymptotic
regime are very unlikely. It is legitimate to ask, why are we interested in these LP formulations in
the first place? We will defer the answer to this question until we reach Sections 7.2.6–7.2.10,
when we enrich the model with non-arc-factored features. For now, consider the arc-factoredcase as an instructive exercise.
By storing the arc-local feature vectors into the columns of a matrix F( x ) := [ f a ( x )] a∈A ,and defining the score vector s := F( x )> w (each entry is an arc score) the inference problem,
which originally ismaximize w · f ( x, y)(7.21)
w.r.t.
y ∈ Y( x )5 In practice, most parsers include a preprocessing stage that prunes the set of potential arcs beforehand, hencein full rigor the bound may not be optimal if we consider this pruning. Gabow’s algorithm takes O(|A| + L log L)
time for a general set of potential arcs A; with sufficiently aggressive pruning, we could have a constant numberof potential heads for each word, making A have size O( L). The asymptotic runtime would then be O( L log L)—
which is still optimal up to logarithmic terms.

7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM

145

can be rewritten, using the multi-commodity flow representation, as the LP
maximize s · z
w.r.t.
z ∈ Zmc ( x );(7.22)

which in terms of linear inequalities (and adding the multi-commodity flow variables) takes
the form
maximize s · z
w.r.t.
z, φ"#(7.23)
z
s.t.
A
≤ b,
φ
where A is a sparse constraint matrix (with O( L3 ) non-zero elements), and b is the constraint
vector; A and b encode the constraints in Eqs. 7.3–7.4 and Eqs. 7.12–7.15. Note that we do
not need any integer constraints, since the solution of this LP is guaranteed to be integer, by
virtue of Proposition 7.3.
An alternative (pursued in Martins et al. 2009b) is to cast arc-factored dependency parsing as an ILP using the single-commodity flow formulation (i.e., letting A and b encode
the constraints in Eqs. 7.3–7.4 and Eqs. 7.6–7.9 instead, and constraining z to be integer).
That approach yields only a quadratic number of variables and constraints, but requires theinteger constraint for exactness; by dropping it the problem becomes the LP relaxation. Although this procedure gives competitive results (as attested in the experimental section ofMartins et al. 2009b), we will not pursue that alternative here for the sake of simplicity.

7.2.6

Pairwise Interactions: Siblings, Grandparents and Head Bigrams

In practice, as we stated in Section 2.3, the performance of the arc-factored model can berather poor, due to the strong factorization assumptions implied by this model. As shownby McDonald and Pereira (2006) and Carreras (2007), the inclusion of features that correlate sibling and grandparent arcs may be highly beneficial.6 Unfortunately, any attempt toextend the arc-factored model to include pairwise arc information renders the problem NPhard (McDonald and Satta, 2007), so we must rely on approximate methods. This is where
our (integer) concise LP formulations can be useful: if we manage to extend them to accommodate this additional pairwise information then we will have a concise ILP for dependency
parsing with rich features; relaxing this ILP will yield an approximate problem which issolvable in polynomial time.
To accomplish the aforementioned goal, we employ a linearization strategy which was
first proposed in the context of pseudo-Boolean optimization, for handling higher-orderinteractions among Boolean variables (Boros and Hammer, 2002). Suppose that we havea model with features of the form f a1 ,...,aK ( x ) (i.e., features whose values depend on thesimultaneous inclusion of arcs a1 , . . . , aK on a candidate dependency tree). Define extra
variables z a1 ...aK := z a1 ∧ . . . ∧ z aK . This logical relation can be expressed by the following
6 By sibling features we mean features that depend on pairs of sibling arcs (i.e., of the form ( h, m ) and ( h, s ));by grandparent features we mean features that depend on pairs of grandparent arcs (of the form (h, m) and ( g, h)).
See Figure 7.1.

CHAPTER 7. TURBO PARSERS

146
O(K ) agreement constraints:
z a1 ...aK≤ z ai ,for each i = 1, . . . , K,Kz a1 ...aK≥∑ za − K + 1.
i

(7.24)

i =1

We apply this strategy to pairwise interactions, in which we assume to be given a set of arc
pairs P. In the sequel, we consider the following pairs, all illustrated in Figure 7.1:
• Sibling arcs, of the form (h, m) and (h, s);7
• Grandparent arcs, of the form (h, m) and ( g, h);• Head bigram arcs, of the form (h, m) and (h0 , m + 1).
Each of these three groups contains O( L3 ) arcs, so |P| = O( L3 ).8 For each pair of arcs
( a, b) ∈ P, we assume that we have a score s ab given by the inner product of a feature vector
f ab ( x ) with the weight vector w; this score contributes to the total objective if arcs a and b
are both included. Accordingly, we add new variables zP := (z ab )(a,b)∈P . The linearization
strategy above yields the following agreement constraints between the zP and the alreadyexisting z-variables, for every ( a, b) ∈ P:
z ab ≤ z az ab ≤ zb
z ab ≥ z a + zb − 1.

(7.25)

Composing this with the optimization problem in Eq. 7.22, we finally get
maximize ∑ a∈A s a z a + ∑(a,b)∈P s ab z ab
w.r.t.
z, zPs.t.
z ∈ Zmc ,z ab ≤ z a ,z ab ≤ zb ,z ab ≥ z a + zb − 1, for every ( a, b) ∈ P.

(7.26)

This is also a linear program with O( L3 ) variables and constraints. However, even though
Zmc coincides with the arborescence polytope Ztree (and therefore all its vertices are integer),the inclusion of the extra pairwise variables and the extra constraints make the linear program no longer guaranteed to have integer solutions. For the optimization problem to be
exact, we need to add the constraint
z integer.
(7.27)
This fact is not surprising, since we know from McDonald and Satta (2007) that the original
problem is NP-hard. Yet, relaxing the integer constraint in Eq. 7.27 yields a very powerfulapproximate parser, as we will see in the experimental section.
7 Note that these are arbitrary siblings, not necessarily consecutive as the ones considered, e.g., by Eisner (1996)
and McDonald and Pereira (2006). Those will be discussed in Section 7.2.7.
8 Other pairs of arcs can be useful, for example Smith and Eisner (2008) consider pairs of crossing arcs which
are O( L4 ) in total.

7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM*

We

learnedalesson

in

1987

about

147

volatility

Figure 7.4: Head automata for the word learned in the sentence of Figure 7.2, seen as sequencial models. We depict in blue the left automaton and in red the right automaton. The
states of each model (shows as numerals inside the circles) represent the position of the lastmodifier.

7.2.7

Head Automata and Consecutive Siblings

We have just described a strategy that allows us to incorporate features that look at siblingarcs; however, those features ignore the relative position of the siblings. In particular, theycannot tell if two sibling arcs (h, m) and (h, s) are consecutive, in the sense that there is no arc(h, m0 ) in the tree with m0 between m and s. In constrast, previous models for projective andnon-projective parsing (Eisner, 1996; Alshawi, 1996; Koo et al., 2010) have made use of head
automata for modeling consecutive siblings. Such models allow incorporating features that fire
when m is the closest child of h (either on the left or on the right side), or when (h, m) and(h, s) are two consecutive sibling arcs on the same side.
Note that consecutive sibling features are neither more nor less expressive than the ones
that refer to arbitrary siblings: the latter allow correlating siblings that are far apart, and oneither side of the head, regardless of what happens in-between; they appear to be particularly
useful in free-order languages, but not only (we will see in Section 7.5 that these features
also improve the accuracy in English datasets). The former only look at consecutive ones on
the same side, but model the fact that no other sibling exists in-between—this is commonlyrefered to as horizontal Markovization (the term vertical Markovization refers to grandparentarcs). We point out that the strategy that we describe here to handle consecutive siblings isdifferent from the one that we have employed in our previously published work (Martinset al., 2009b); the one we present now is better, since it leads to tighter relaxations while
requiring the same number of variables and constraints, namely O( L3 ).
We start by describing head automata models and their underlying polytopes, which wecall head automata polytopes. Then, we will see how a head automaton can be seen as a solverof a linear optimization problem over the head automaton polytope.
Without loss of generality, let us fix a word position h (which will be the head word) andconsider its potential dependents on the right side—it is straightforward to generalize what
follows for the sequence of left-side dependents. Define the following “consecutive sibling”
indicator variables, for each h, m, s with 0 ≤ h ≤ m < s ≤ L + 1:9(
1 if (h, m) and (h, s) are consecutive siblings,
zcs(7.28)
hms : =
0 otherwise.
Given h, we can stack all these variables in a vector of right-dependent indicators zcs
h,→ : =
9 For convenience, we define also these variables for the case where m = h (in which case they indicate that s
is the first child of h) and for s = L + 1 (which indicate that m is the last child of h). When h = m and s = L + 1,
they indicate that h does not have children on the right side.

CHAPTER 7. TURBO PARSERS

148

cs
(zcs
hms ) h≤m<s≤ L+1 , and analogously we can form a vector for the left-dependents, z h,← : =
3
cs
(zcs
hms ) h≥m>s≥0 . We can further stack all these vectors in a O ( L )-dimensional vector, z : =
cs
cs
((zcs
h,→ ) h , ( z h,← ) h ). We assume there is a score vector s containing the corresponding scores,in addition to the already described arc-factored scores. That is:• shm is the score associated with word m being a modifier of h,• scs
hhs is the score associated with word s being the first modifier of h,• scs
is the score associated with word m being the last modifier of h,hm( L+1)• scs
hms is the score associated with words ( h, m ) and ( h, s ) being consecutive siblings.
In a head automaton, the goal is, for a fixed h, to find the most likely sequence of dependents
on one of the sides. We will assume that the automaton is for h = 0 and is looking for rightdependents; the general case follows easily. The problem is equivalent to that of finding a
Viterbi path (m1 , . . . , m L ) for a chain model whose possible states for M j are {0, . . . , j}; theevent M j = a means that “the last modifier, up to j, is a.” This is illustrated in Figure 7.4.
Between words j and j + 1, only two transitions may occur: either M j+1 = m j (which happens
if j + 1 is not a modifier), or M j+1 = j + 1 (which happens otherwise). Since this is a chain
model, the marginal polytope is exactly characterized by local consistency constraints andhard constraints (see Chapter 5). Let µi ( a) be the posterior marginal for the event Mi = a,and µi,i+1 ( a, b) the posterior marginal for the event Mi = a ∧ Mi+1 = b. Local consistency
constraints assert thati

∑ µi (a) = 1,i ∈ {1, . . . , L}

(7.29)i ∈ {1, . . . , L}, a ∈ {1, . . . , L}

(7.30)i ∈ {1, . . . , L}, b ∈ {1, . . . , i + 1}

(7.31)i ∈ {1, . . . , L}, a ∈ {1, . . . , i }(7.32)i ∈ {1, . . . , L}, a ∈ {1, . . . , i }, b ∈ {1, . . . , i + 1}.

(7.33)

a =0i +1∑ µi,i+1 (a, b) = µi (a),b =0i

∑ µi,i+1 (a, b) = µi+1 (b),a =0µi ( a) ≥ 0,
µi,i+1 ( a, b) ≥ 0,

Hard constraints assert that impossible configurations must receive zero marginals:µi,i+1 ( a, b) = 0,i ∈ {1, . . . , L}, a ∈ {1, . . . , i }, b ∈/ { a, i + 1}.

(7.34)

Plugging (7.34) in (7.30)–(7.31) yields:µi,i+1 ( a, a) + µi,i+1 ( a, i + 1) = µi ( a),µi,i+1 (b, b) = µi+1 (b),i ∈ {1, . . . , L}, a ∈ {1, . . . , i }(7.35)i ∈ {1, . . . , L}, b ∈ {1, . . . , i + 1}

(7.36)i ∈ {1, . . . , L},(7.37)i ∈ {1, . . . , L}, a ∈ {1, . . . , L}.

(7.38)i∑ µi,i+1 (a, i + 1) = µi+1 (i + 1),

a =0

and plugging further (7.36) in (7.35) yields:µi+1 ( a) + µi,i+1 ( a, i + 1) = µi ( a),7.2. DEPENDENCY PARSING AS AN INTEGER LINEAR PROGRAM

149

The marginal polytope is thus characterized by (7.29), (7.38), (7.37), and (7.32)–(7.33). We
next make the variable replacements• zhi = µi (i ), the posterior marginal for the event that i is a modifier of h;• zsc= µi,i+1 ( a, i + 1), the posterior marginal for the event that (h, a) and (h, i + 1)
ha(i +1)
are consecutive siblings;• ωhai := µi ( a), an extra variable for the event that, up to i, the last modifier of h is a.
The overall optimization problem associated with the (h, →)-automaton becomes that ofmaximizingLLL∑ shj zhj + ∑ ∑ shjk zhjk

(7.39)

ωhii = zhi ,i ∈ {1, . . . , L}

(7.40)∑ ωhai = 1,i ∈ {1, . . . , L}

(7.41)i ∈ {1, . . . , L}, a ∈ {1, . . . , i }(7.42)i ∈ {1, . . . , L}

(7.43)i ∈ {1, . . . , L}, a ∈ {1, . . . , i }(7.44)i ∈ {1, . . . , L}, a ∈ {1, . . . , i }.

(7.45)

j = h +1

j = h k = j +1

subject to:

i

a =0

ωha(i+1) + zcs
ha(i +1) = ωhai ,i∑ zcsha(i+1) = zh(i+1) ,a =0

ωhai ≥ 0,cszha(i+1) ≥ 0,

L
Let zh := (zhm )m=h+1 and ω h : = ( ωham ) h≤ a≤m≤ L . We define the ( h, →)-head automaton
polytope as the set

Zhead,h,→ :=(zh , zcs
h,→ )

∃ω :

(zh , zcs
h,→ , ω h } satisfy Eqs. 7.40–7.45

.

(7.46)

This polytope is isomorphic to the marginal polytope underlying the chain model associatedwith the head automaton. As such, its vertices are integer and correspond to the valid
assignments of the (zh , zcsh,→ )-variables. Note that the head automata are all disjoint, in thesense that each of them involves their own set of variables, with no overlaps. We define
the head automaton polytope Zhead as the Euclidean product of all (h, →)- and (h, ←)-head
automaton polytopes:
L

L

h =0

h =0

Zhead := ∏ Zhead,h,← × ∏ Zhead,h,→ .

(7.47)

To sum up, our LP formulation can be extended for accommodating consecutive siblings. All
we need is to add the O( L3 ) extra variables and constraints in Eqs. 7.40–7.45. Even thoughboth polytopes Zmc and Zhead have only integer vertices, that does not happen with theirintersection. Therefore, the resulting LP formulation that composes the arc-factored modelwith the head automata is not exact; if one adds the integer constraints in Eq. 7.27, then we
obtain an ILP which is exact. Without the integer constraints, one obtains an LP relaxation
which is equivalent to one used in the sibling model in Koo et al. (2010).

CHAPTER 7. TURBO PARSERS

150

7.2.8

Valency Indicators

A crucial fact about dependency grammars is that words have preferences about the number
and arrangement of arguments and modifiers they accept. Therefore, it may be desirable toinclude features that indicate, for a candidate dependency tree, how many dependency links
depart from each word; denote these quantities by vi := ∑ a∈δ+ (i) z a , for each i ∈ N. We callvi the valency of the ith word.
In Martins et al. (2009b), we added valency indicators zval
ik : = [[ vi = k ]] for i ∈ N andk = 0, . . . , L. This way, we were able to penalize candidate dependency trees that assign
unusual valencies to some of their words, by specifying a individual cost for each possible
value of valency. The following O(| L|2 ) constraints encode the agreement between valency
indicators and the other variables:L∑ kzvalik = ∑ z a ,+i∈Na ∈ δ (i )

k =0

L

∑ zvalik = 1,

i∈N

k =0

zval
ik ≥ 0,i ∈ N, k ∈ {0, . . . , L}.

(7.48)

Experimentally, valency features did not help much, so we do not consider them further.

7.2.9

Directed Path Indicators

One of the key advantages of the multicommodity flow model that we have presented in
Section 7.2.4 is that it directly models ancestor-descendant relationships—namely, if an arc(i, j) carries flow directed to a node k (i.e., if φijk = 1), then there is a directed path in thepathtree between j and k. Hence, we may define variables z jk that indicate if there is a pathfrom j to k. Since each node except the root has only one incoming arc, the following linear
equalities are enough to describe these new variables:
path

z jk

=

∑− φak ,j, k ∈ N \ {0}a∈δ ( j)
path

z0k

= 1,

k ∈ N \ {0}.

(7.49)We define features that fire when two words bear an ancestor-descendant relationship,
giving rise to a score term in the objective that rewards/penalizes trees for which that relationship holds. For example, we might learn to prefer trees for which certain nouns descendfrom verbs that are semantically related.
In the very same way that arbitrary siblings can be seen as a way of relaxing the horizontal Markov property that underlies consecutive siblings, directed path features relax thevertical Markov property underlying grandparent features. This was illustrated in Figure 7.1.
Note that this is only one possible usage of ancestor-descendant relationships. There areother ways in which one can exploit this important information given as a by-product ofthe multicommodity flow model. For example, it is possible to include features that look atspans headed by a given word—the span boundaries are simply the leftmost and rightmost
descendant of that word.

7.3. TURBO PARSERS AND THEIR FACTOR GRAPHS

7.2.10151

Nonprojective Arc Indicators

For most languages, dependency parse trees tend to be nearly projective (cf. Buchholz andMarsi 2006). We may wish to make our model capable of learning to prefer “nearly” projective parses whenever that behavior is observed in the data. We refer to the backgroundsections on dependency parsing (Section 2.3) for the definitions of “projective arc” and “projective tree” (Definitions 2.3–2.4).
In our model, we consider features that fire for arcs which are non-projective. By definition, these are arcs whose span contains words which do not descend from the head, and assuch, this can be captured by making use of the directed path variables zpath (Section 7.2.9).
np
The procedure is as follows. Define indicators znp := (z a ) a∈A , wherenpz a := [[ a ∈ y and a is nonprojective]].

(7.50)
np

From the definition of projective arcs (Definition 2.3), we have that z a = 1 if and only if thepath
arc is active (z a = 1) and there is some node k in the span of a = (i, j) such that zik = 0.
We are led to the following O( L3 ) constraints for (i, j) ∈ A:
zij

np

≤ zij

np
zij

≥ zij − zik ,

np
zij

path

max(i,j)−1≤ −∑min(i, j) ≤ k ≤ max(i, j)
path

zik

+ | j − i | − 1.

(7.51)

k =min(i,j)+1

7.3 Turbo Parsers and Their Factor Graphs
In this section, we show a formal connection between three different approximate dependency parsers recently proposed by Smith and Eisner (2008), Martins et al. (2009b) and Koo
et al. (2010). While these parsers are differently motivated, we show that all correspond toinference in a factor graph, and all optimize objective functions over local approximationsof the marginal polytope. The connection is made clear by showing the factor graphs that
underlie each of these parsers, and by writing the explicit declarative optimization problem
that they attempt to solve. The machinery described in Chapter 5 for constrained graphical models will allow us to recognize that each of these parsers can be regarded as a turbo
parser: they all can be seen as approximate inference algorithms that ignore loops in a factor graph. Their success parallels similar approximations in other fields, such as statisticalimage processing and error-correcting coding.
We summarize the situation as follows:• Smith and Eisner (2008) proposed a factor graph (which we reproduce in Figure 7.5) inwhich they run sum-product loopy BP. For the configuration that includes arc-factored
and consecutive sibling features, this factor graph is precisely the one that is used by
Koo et al. (2010) in their dual decomposition framework. Since the underlying factor
graph explicitly contains a factor that emulates the “tree” constraint, we call it a treebased factor graph.10
10 Note that we are refering here to two graphs of a different nature:

one is the factor graph, whose nodes

CHAPTER 7. TURBO PARSERS

152

Figure 7.5: Factor graph corresponding to the dependency parsing model of (Smith andEisner, 2008) with sibling and grandparent features. Circles denote variable nodes, andsquares denote factor nodes. Note the loops created by the inclusion of pairwise factors(grand and sib).

• Martins et al. (2009b) approximate parsing as the solution of a linear program, wherea wide set of linear constraints encapsulate the requirement that the output is a valid
dependency tree, while allowing to accommodate a large set of rich features. This is
the parser to which we have devoted the entire Section 7.2.
At first sight, it is not clear what is being optimized in Smith and Eisner (2008) and what, if
any, factor graph is being used by Martins et al. (2009b). Here, we fill the blanks in the two
approaches: we derive explicitly the variational problem addressed in the former reference,
and we provide the underlying factor graph in the latter, which will be called a flow-based
factor graph since it is related with the multi-commodity flow formulation presented in Section 7.2. Furthermore, we will show that for some of the configurations, the local polytope
approximations of all the three aforementioned parsers are exactly the same. So, where dothese parsers differ? Essentially, in three aspects:• the kind of inference they run: marginal inference in Smith and Eisner (2008), versus
MAP inference in Martins et al. (2009b) and Koo et al. (2010);• the optimization algorithms they implement: loopy belief propagation in Smith andEisner (2008), the dual simplex method implemented by an off-the-shelf LP solver in
Martins et al. (2009b), and the projected subgradient algorithm for dual decompositionin Koo et al. (2010).
• The features they use. Even when the parsers use the same parts, the part-local features
might not be the same. This explains some differences in performance, e.g., between
Martins et al. (2009b) and Koo et al. (2010).
The next subsections characterize the tree-based and flow-based factor graphs, as well as theinference procedures run on them.

7.3.1

A Tree-Based Factor Graph

We depict in Figure 7.5 the factor graph used in the dependency parser of Smith and Eisner(2008); we denote this factor graph by Gtree . There are O( L2 ) variable nodes in Gtree , each of
represent dependency links; the other is the dependency tree, whose nodes are words.

7.3. TURBO PARSERS AND THEIR FACTOR GRAPHS

153

them a binary variable representing a potential arc a = (h, m). There are O( L3 ) pairwise soft
factors connecting pairs of arcs that form sibling and grandparent relationships, of the kinddescribed in Section 7.2.6. There is also a hard constraint “tree” factor connected to all thevariables, which imposes that the set of active arcs form a well-defined dependency tree (i.e.,an arborescence). Finally, there are O( L) factors that emulate head automata, of the kinddescribed in Section 7.2.7.11
Let us recall how all the computations involving the tree and head automata factors
are performed efficiently in Smith and Eisner’s sum-product loopy BP and in Koo et al.’s
projected subgradient algorithms:• From Proposition 5.4, we have that all that is necessary for computing the sum-productmessages sent by a factor is to be able to compute the marginals of the subgraph comprised only of that factor (see also Algorithm 6). As described in the backgroundsection (Section 2.3), for the tree factor this can be done in O( L3 ) time by invoking
the matrix-tree theorem. As a consequence, O( L3 ) is also the time necessary to compute all outgoing messages in that factor. What about the head automata factors? As
shown in Section 7.2.7, a head automaton is equivalent to a sequence model, representable as a chain graph. Therefore, the marginals can be computed by running theforward-backward algorithm, which costs O( L2 ) time for each automaton. Since there
are O( L) such automata, and we still have O( L3 ) pairwise factors whose messages canbe computed in constant time, each iteration of loopy BP has a total runtime of O( L3 ).
• Let us turn to the projected subgradient algorithm of Koo et al. (2010). In this algorithm, the only operations that are necessary at factor-level are local MAP computations. For the tree factor, a MAP computation can be done in time O( L3 ) with ChuLiu-Edmonds’ algorithm (or even faster using Tarjan’s or Gabow’s algorithms, as wehave seen). For the head automata, all we need is to run the Viterbi algorithm whichtakes O( L3 ) time in total (O( L2 ) per automaton). Hence, one iteration of the projected
subgradient algorithm also has a runtime of O( L3 ).
What are these algorithms optimizing? The graphical model machinery described in
Chapters 4–5 allows us to interpret both approximate parsers as procedures for optimizing afunction over the local polytope LOCAL(Gtree ). Let us start by characterizing this polytope.
As before, we denote by A the set of candidate arcs, and by P ⊆ A2 the set of pairs ofarcs that have factors. Let µ := (µA , µP ) with µA = (µ a (.)) a∈A be the variable marginals,
and µP = (µ ab (., .))(a,b)∈P be the factor marginals, for the pairwise soft factors. Since all
variables are binary, we will use a “Boltzmann” parametrization and will write, for eacha ∈ A, µ a (1) = z a and µ a (0) = 1 − z a , where z a is a variable constrained to [0, 1]. Let
zA := (z a ) a∈A ; since the tree factor is a hard constraint factor, its marginal polytope isdefined as the convex hull of its acceptance set, which is precisely the arborescence polytope
Ztree defined in Section 7.2.1. Hence we have the constraintzA ∈ Ztree .

(7.52)For the head automata, we denote by zcs := (zcshms ) hms the vector of marginal probabilities
cs
for consecutive siblings, where zhms denotes a marginal probability for the event that (h, m)
11 Smith and Eisner (2008) also proposed other variants with more factors, which we omit for brevity.

CHAPTER 7. TURBO PARSERS

154

and (h, s) are consecutive siblings on the same side of h. The marginal polytope for eachhead automaton (h, →) is the set Zhead,h,→ defined in Eq. 7.46 (and analogously for (h, ←)).
The marginal polytope for the entire set of head automata is the polytope Zhead defined in
Eq. 7.47. We thus have(zA , zcs ) ∈ Zhead .
(7.53)Let us now turn to the pairwise factors. It is straightforward to write a contingency tableand obtain the following local agreement constraints at each pairwise factor involving arcs a
and b:µ ab (1, 1) = z ab ,µ ab (0, 0) = 1 − z a − zb + z ab
µ ab (1, 0) = z a − z ab , µ ab (0, 1) = zb − z ab .
Noting that all these marginals are constrained to the unit interval, one can get rid of all
variables µ ab and write everything asz a ∈ [0, 1], zb ∈ [0, 1], z ab ∈ [0, 1],z ab ≤ z a ,z ab ≤ zb ,z ab ≥ z a + zb − 1,

(7.54)The local polytope LOCAL(Gtree ) is thus formed by the sets of constraints in Eqs. 7.52, 7.53
and 7.54.
Now let us write the variational problem that loopy BP optimizes. For simplicity, we will
follow Martins et al. (2010f) and refer only to the configuration without the head automata(i.e., just the tree factor and the pairwise soft factors). For deriving the Bethe entropy approximation, we need first to compute the factor entropies. Start by noting that the tree-factor
entropy Htree can be obtained in closed form by computing the marginals z̄A and the partition function Z from the incoming messages, as shown in Algorithm 6. Both quantities canbe computed using the matrix-tree theorem. Let
Ia;b (z a , zb , z ab ) = ∑ µ ab (y a , yb ) log
y a ,ybµ ab (y a , yb )µ a (y a )µb (yb )(7.55)

be the mutual information associated with each pairwise factor. From Eq. 4.45, we can write
the overall Bethe entropy approximation as:
HBethe (µ) =∑ (1 − deg(a)) Ha (za ) + ∑ Hab (zab ) + Htree (zA )a ∈A( a,b)∈P= Htree (zA ) −∑ Ia;b (za , zb , zab ).

(7.56)( a,b)∈P

The approximate variational expression becomeslog Z (θ, x ) ≈ maxz θ> F( x )z + Htree (zA ) −s.t.

∑ Ia;b (za , zb , zab )( a,b)∈P

z ab ≤ z a , z ab ≤ zb ,z ab ≥ z a + zb − 1, ∀( a, b) ∈ P,
zA ∈ Ztree ,(7.57)

whose maximizer corresponds to the beliefs returned by Smith and Eisner’s loopy BP algorithm (if it converges). The reader familiar with Bethe approximations for pairwise graphs

7.3. TURBO PARSERS AND THEIR FACTOR GRAPHS

XOR

155

OR

XOR-OUTXOR-OUTOR-OUT

AND-OUT

Figure 7.6: Details of the factor graph underlying the parser of (Martins et al., 2009b). Dashedcircles represent auxiliary variables.

will note that Eq. 7.57 only differs from the pure pairwise case without hard constraints byreplacing the sum of variable entropies, ∑ a∈A H (z a ) by the tree entropy Htree (zA ).

7.3.2

A Flow-Based Factor Graph

We now turn to our concise ILP formulation (Martins et al., 2009b), described in Section 7.2.
As mentioned above, this formulation is exact but the corresponding problem is NP-hard,hence we suggest dropping the integer constraints to obtain an LP relaxation of the original
problem.
This section sheds some light on this approximation. Namely, we will construct a factor graph Gflow , and we will show that the LP relaxation corresponds to LP-MAP inferencein that graph. In other words, we will show that it corresponds to an optimization of theform in Eq. 4.50, where the marginal polytope MARG(Gflow ) is replaced by the local surrogate LOCAL(Gflow ). The flow-based factor graph is represented in Figure 7.6, and we next
describe it in detail.
For completeness, we consider all the parts illustrated in Figure 7.1, which were describedin Sections 7.2.5–7.2.10 (we leave out the valency features, whose interpretation would be alittle bit more involved). Let us start by describing the variables nodes of Gflow . There are
two kinds of variable nodes:• those that correspond to binary variables that contribute score terms to the objective of
the ILP;• some additional auxiliary variables that are necessary to build some of the constraints.
The ones of the first kind are:• Arc variables z := (z a ) a∈A , denoting potential arcs in the tree;

CHAPTER 7. TURBO PARSERS

156• Consecutive sibling variables zcs := (zcshms ) h,m,s , each denoting a pair of consecutive
siblings (h, m) and (h, s) in the tree;
path

path

L
• Directed path variables zpath := (zij )i,j
=0 , where each zij
word i is an ancestor of word j in the dependency tree;np

represents the event thatnp

• Non-projective arc variables znp := (z a ) a∈A , each z a denoting that a is a non-projective
arc in the dependency tree.
The auxiliary variable nodes are:
• The flow variables φ := (φak ) a∈A,k=1,...,L that denote multi-commodity flows;• The auxiliary variables ω := (ωham )h,a,m that denote states in the head automata.
For clarity, we represent the auxiliary variables as dashed circles in Figure 7.6. (In general,
any lifted formulation involves dashed circles.)Let us now turn to the factors of Gflow . Again, there are two kinds:• soft factors, all of them pairwise, allowing to deal with pairwise arc features (for siblings,
grandparents and head-bigrams). These factors connect pairs of arc variables, precisely
as was done in the tree-based factor graph (Section 7.3.1).
• hard constraint factors, which represent the linear constraints of the ILP.
It remains to describe the hard constraint factors. A few of them are unary and serve only
to seed some of the path and flow variables, imposing the constraints
path

z0k

path

= zkk

= 1, ∀k,φ(hh,m) = 0,∀h, m;(7.58)(7.59)

i.e., that any word descends from the root and from itself, and that arcs leaving a word carry no flow
to that word. Then, there is a set of factors which serve to impose the multi-commodity flow
constraints, and which replace the tree factor in Figure 7.5:
• O( L) XOR factors, each connecting all arc variables of the form {(h, m)}h=0,...,L for a
fixed m. These ensure that each word has exactly one parent. Each factor yields a local
agreement constraint (see the marginal polytope expression in Eq. 5.25):
L∑ z(h,m) = 1,m ∈ {1, . . . , L};(7.60)

h =0

these are equivalent to the single-parent constraints in Eq. 7.3. The constraint that theroot has no parent (Eq. 7.4) can be trivially imposed by removing all arc candidates ofthe form (h, 0), which are structurally impossible.
• O( L3 ) IMPLY factors, each expressing that if an arc carries flow, then that arc must be
active. Such factors are pairwise OR factors with the first input negated, hence, thelocal agreement constraints are:φak ≤ z a ,a ∈ A, k ∈ {1, . . . , L}.

(7.61)

7.3. TURBO PARSERS AND THEIR FACTOR GRAPHS

157

These correspond to the multi-commodity flow constraints in Eq. 7.14.
• O( L2 ) XOR-with-output factors, which impose the constraint that each path variablepath
zmk is active if and only if exactly one incoming arc in {(h, m)}h=0,...,L carries flow to k. Suchfactors are XOR factors with the last input negated, and hence their local constraints
are:nzmk = ∑ φ(kh,m) ,path

m, k ∈ {1, . . . , L}.

(7.62)

h =0

These correspond to the constraints in Eq. 7.49 that are the very definition of the pathvariables.
• O( L2 ) XOR-with-output factors to impose the constraint that words do not consume otherwords’ commodities; i.e., if h 6= k and k 6= 0, then there is a path from h to k if and only
if exactly one outgoing arc in {(h, m)}m=1,...,L carries flow to k. The resulting marginal
polytopes are defined by the constraints:
path

zhk

n

= ∑ φ(kh,m) , h, k ∈ {0, . . . , L}, k ∈/ {0, h}.

(7.63)

m =1

It is easy to see that these correspond to the multi-commodity flow constraints in
Eq. 7.13 for the case j 6= k, once we make the substitution in Eq. 7.62.
These factors are equivalent to the tree factor, in the sense that composing all their marginal
polytopes yields the marginal polytope of the tree factor, i.e., the arborescence polytope. In
addition, we have another set of factors which replace the head automata factors in Figure 7.5:
• A XOR factor requiring that there is exactly one word which is the last modifier of h up to i.
This imposes the constraint in Eq. 7.41:
i∑ ωhai = 1,i ∈ {1, . . . , L},(7.64)a =0• A XOR-with-output factor imposing that, for each word i and each state a (the lastmodifier up to i), either that word preserves the previous state or it modifies h (cf. Eq. 7.42):
ωha(i+1) + zcs
ha(i +1) = ωhai ,i ∈ {1, . . . , L}, a ∈ {1, . . . , i }(7.65)• A XOR-with-output factor imposing that each potential arc (h, i + 1) is active if and only
if it is the next sibling of some other arc, which must be unique (or a first child of h, inwhich case zcs= 1 by definition). The corresponding marginal polytope emulateshh(i +1)
Eq. 7.43:i∑ zcsha(i+1) = zh(i+1) ,i ∈ {1, . . . , L}

(7.66)

a =0

We must also collapse the variables ωhii and zhi , in accordance to Eq. 7.40. Finally, there is alast set of factors that serve to accommodate the non-projective arc features:

CHAPTER 7. TURBO PARSERS

158• For each arc a ∈ A of the form a = (h, m), we have one OR-with-output factor conpath
nected to all the path variables zhk for k between h and m, each negated, whose output
is an extra variable (call it γa ) that indicates that there is some word k in the span of a which
does not descend from h. Suppose without loss of generality that h < m. From Eq. 5.39,
we have that the corresponding marginal polytope is defined by the following set of
constraints:path

γa ≥ 1 − zhk ,γa ≤ −∑

for every k ∈ {h + 1, . . . , m − 1},path

zhk + |m − h| − 1.

(7.67)k ∈{ h+1,...,m−1}• For each arc a ∈ A of the form a = (h, m), one AND-with-output factor connected tonp
z a and γa , whose output is z a . This signals that the tree contains the arc a and that thisarc is non-projective. The corresponding constraints are:np

za ≤ za ,np

z a ≤ γa ,np

z a ≥ z a + γa − 1.

(7.68)

Taken together, the constraints in Eqs. 7.67–7.68 are equivalent to the non-projectivity constraints in Eq. 7.51.
Let us summarize what we have built. We defined a factor graph Gflow , represented
in Figure 7.6; MAP inference in such graph is equivalent to the ILP formulation that wehave introduced in Section 7.2. Moreover, LP-MAP inference, which replaces the marginal
polytope of this graph by the local polytope LOCAL(Gflow ), defined by the constraints in
Eqs. 7.54–7.68, is precisely the LP relaxation of that ILP formulation. Moreover, if we resort
only to the parts considered in Section 7.3.1—namely, the ones for pairwise arcs (grandparents, arbitrary siblings, and head bigrams) as well as the consecutive siblings—then we also
have
LOCAL(Gflow ) = LOCAL(Gtree ),(7.69)
since, as we have seen, the multi-commodity flow model is an exact representation of thearborescence polytope, and the same holds for the head automaton polytope. Therefore,
although the approaches of Smith and Eisner (2008) and Martins et al. (2009b) look very
different, in reality both are variational approximations emanating from Proposition 5.3, respectively for marginal and MAP inference. However, they operate on distinct factor graphs,
respectively the ones in Figure 7.5 and Figure 7.6.12
12 Given what was just exposed, it seems appealing to try max-product loopy BP on the factor graph of Figure 7.5, or sum-product loopy BP on the one in Figure 7.6. Both attempts present serious challenges: the former
requires computing messages sent by the tree factor, which requires O( L2 ) calls to the Chu-Liu-Edmonds algorithm and hence O( L5 ) time. No obvious strategy seems to exist for simultaneous computation of all messages,
unlike in the sum-product case. The latter is even more challenging, as standard sum-product loopy BP has
serious issues in the factor graph of Figure 7.6; we construct in Martins et al. (2010f, extended version, Appendix
B) a simple example with a very poor Bethe approximation. This might be fixed by using other variants ofsum-product BP, e.g., ones in which the entropy approximation is concave.

7.4. DEPENDENCY PARSING WITH AD3

159

7.4 Dependency Parsing with AD3
In this section, we describe how the AD3 algorithm presented in Chapter 6 can be applied
to our rich feature models for dependency parsing. Before doing so, we briefly review the
related work on dual decomposition.

7.4.1

Related Work: Dual Decomposition in NLP

A typical source of intractability in NLP problems comes from the combinatorial explosion
inherent in the composition of two or more tractable models (Bar-Hillel et al., 1964; Tromble
and Eisner, 2006). In a recent paper, Rush et al. (2010) proposed a dual decomposition
framework as a way of combining models which alone permit efficient decoding, but whose
combination is intractable. These are problems in which the global score decomposes asf (y) = f 1 (z1 ) + f 2 (z2 ), where z1 and z2 are two overlapping “views” of the output, so thatthe problem of finding the output y ∈ Y with the largest score f (y) can be written as
maximize f 1 (z1 ) + f 2 (z2 )w.r.t.
z 1 ∈ Y1 , z 2 ∈ Y2s.t.
z1 ∼ z2 .

(7.70)

Above, the notation z1 ∼ z2 means that z1 and z2 “agree on their overlaps,” and an isomorphism Y ' {hz1 , z2 i ∈ Y1 × Y2 | z1 ∼ z2 } is assumed. One way of addressing this problem
is to introduce Lagrange multipliers for the agreement constraints and to tackle the dualwith the projected subgradient algorithm, a technique known as Lagrangian relaxation. This
results in a relaxation of the original problem which is equivalent to the one described inour background chapters (Section 4.6.2). This technique has proven quite effective in parsing(Koo et al., 2010; Auli and Lopez, 2011) as well as machine translation (Rush and Collins,
2011; Chang and Collins, 2011). The application to dependency parsing by Koo et al. (2010)
has already been mentioned in Section 7.3.2, when we described inference in the tree-based
factor graph.
We will next formalize these notions and proceed to compositions of an arbitrary numberof models. Of special interest is the unexplored setting where this number is very large
and each component very simple. We show here that the success of dual decomposition
with the projected subgradient algorithm is strongly tied to the ability of finding a “good”
decomposition, i.e., one involving few overlapping components (or slaves). With many components, the algorithm exhibits extremely slow convergence (cf. Figure 7.8). Unfortunately,
a lightweight decomposition is not always at hand, either because the problem does notfactor in a natural way, or because one would like to incorporate features that cannot beeasily absorbed in few tractable components. Examples include some of the features that wehave discussed in Section 7.2, as well as features generated by statements in first-order logic,
features that violate Markov assumptions, or history features such as the ones employed intransition-based parsers.

CHAPTER 7. TURBO PARSERS

160

7.4.2

Dual Decomposition With Many Overlapping Components

We will see that the AD3 algorithm, introduced in Chapter 6, is a good fit for the kind ofproblems mentioned above. AD3 retains the modularity of the subgradient-based method,
but it speeds up consensus by regularizing each slave subproblem towards the averaged
votes obtained in the previous round (cf. Algorithm 8). While this yields more involved
subproblems (with a quadratic term), we have shown in Chapter 6 how exact solutions canstill be efficiently computed for all kinds of first-order logic, which is all we need for the flowbased factor graph described in the previous section. As a result, we obtain parsers that canhandle very rich features, do not require specifying a decomposition, and can potentially beparallelized.
Let us first establish the basic setup, where we characterize the kind of NLP problems
for which AD3 is potentially useful. These are problems characterized by many overlapping
components or parts; such problems abound in the scope of text analysis at sentence level(parsing, compositional semantics, etc.) and at document or corpus level (summarization,
entity resolution, etc). The ingredients are the following:
Basic parts. We let R be a set of basic parts, such that each element y ∈ Y can be identified
with a subset of R. The exact meaning of a “basic part” is problem dependent. For example,
in dependency parsing, R can be the set of all possible dependency arcs (see Figure 7.1); in
phrase-based parsing, it can be the set of possible spans; in sequence labeling, it can be theset of possible labels at each position. Our only assumption is that we can “read out” y fromthe basic parts it contains. For convenience, we represent y as a binary vector, y = (y(r ))r∈R ,where y(r ) = 1 if part r belongs to y, and 0 otherwise.
Decomposition. We generalize the decomposition in Eq. 7.70 by considering sets Y1 , . . . , YS
for S ≥ 2. Each Ys is associated with its own set of parts Rs , in the same sense as above;
we represent the elements of Ys as binary vectors zs = (zs (r ))r∈Rs . Examples are vectors
indicating a tree structure, a sequence, or an assignment of variables to a factor, in whichcase it may happen that only some binary vectors are legal. Some parts in Rs are basic, whileothers are not. We denote by R̄s = Rs ∩ R the subset of the ones that are. In addition, we
assume that:• R1 , . . . , RS jointly cover R, i.e., R ⊆s =1 Rs ;SS• Only basic parts may overlap, i.e., Rs ∩ Rt ⊆ R, ∀s, t ∈ {1, . . . , S};• Each zs ∈ Ys is completely defined by its entries indexed by elements of R̄s , fromwhich we can guess the ones in Rs \ R̄s . This implies that each y ∈ Y has a uniquedecomposition (z1 , . . . , zS ).
Fig. 7.1 shows several parts used in dependency parsing models; in phrase-based parsing,these could be spans and production rules anchored in the surface string; in sequence labeling, they can be unigram, bigram, and trigram labels.13
13 There is a lot of flexibility about how to decompose the model into S components: each set R can corresponds
to a single factor in a factor graph (Smith and Eisner, 2008), or to a entire subgraph enclosing several factors(Koo et al., 2010), or even to a formula in Markov logic (Richardson and Domingos, 2006). In these examples,the basic parts may correspond to individual variable-value pairs.

7.4. DEPENDENCY PARSING WITH AD3

161

Global consistency. We want to be able to read out y ∈ Y by “gluing” together the components (z1 , . . . , zS ). This is only meaningful if they are “globally consistent,” a notion which
we make precise. Two components zs ∈ Ys and zt ∈ Yt are said to be consistent (denotedzs ∼ zt ) if they agree on their overlaps, i.e., if zs (r ) = zt (r ), ∀r ∈ Rs ∩ Rt . A complete assignment (z1 , . . . , zS ) is globally consistent if all pairs of components are consistent. This isequivalent to the existence of a witness vector (u(r ))r∈R such that zs (r ) = u(r ), ∀s, r ∈ R̄s .
With this setup, assuming that the score function decomposes as f (z) = ∑Ss=1 f s (zs ), thedecoding problem (which extends Eq. 7.70 for S ≥ 2) becomes:
P : maximize ∑Ss=1 f s (zs )w.r.t.
z s ∈ Ys , ∀ s(u(r ))r∈R ∈ R|R| ,s.t.
zs (r ) = u(r ), ∀s, r ∈ R̄s .

(7.71)

We call the equality constraints expressed in the last line the “agreement constraints.” It isthese constraints that complicate the problem, which would otherwise be exactly separable
into S subproblems. Note that each set Ys in Eq. 7.71 is combinatorial, hence non-convex.
Let Zs denote the convex hull of Ys . The relaxed problem is
P0 :maximizew.r.t.
s.t.

∑Ss=1 f s (zs )z s ∈ Zs , ∀ shu(r )ir∈R ∈ R|R| ,zs (r ) = u(r ), ∀s, r ∈ R̄s .

(7.72)

AD3 updates. Regarding each component as a factor in a factor graph, we can apply theAD3 algorithm in a straighforward manner. The updates become the following (cf. Algorithm 8):14• z-updates, for each s = 1, . . . , S:
zts+1 = arg max

z s ∈Zs ( x )ηf s (zs ) + ∑ λs (r )zs (r ) − ∑ (zs (r ) − ut (r ))2
2 r∈R̄
r ∈R̄

!
,(7.73)ss• u-updates:u t +1 ( r ) =1
zts+1 (r ),|{s : r ∈ R̄s }| s:r∑∈R̄

(7.74)s• λ-updates:
λts+1 (r ) = λts (r ) − τη (zts+1 (r ) − ut+1 (r )).

(7.75)

We end this section by listing in Table 7.4 all the factors in the flow-based factor graph,including their interpretation as first order logic statements. Each of these factors is a components in the AD3 algorithm. We also show in Table 7.4 the asymptotic runtimes for solving
the corresponding subproblems.
14 We set τ = 1.5 and follow the procedure described in Section 6.3.2 for adjusting the penalty constant. We
initialize η = 0.03 and then increase/decrease η by a factor of 2 whenever the primal residual becomes > 10times larger/smaller than the dual residual.

CHAPTER 7. TURBO PARSERS

162

Arc-factored

Grandparents
Siblings, all-siblings
Head bigram
Directed path

Nonprojective arc

bilexical, unilexical, POS
head and modifier contexts
in-between
arc direction and length
morphologicalbilexical, unilexical, POS
arc directions and lengths
bilexical, unilexical, POS
arc directions and lengths
bilexical, unilexical, POS
arc directions and lengths
bilexical, unilexical, POS
ancestor
descendant
bilexical, unilexical, POS
bias
in-between
arc direction and lengthwh ∧ wm and POS back-offs
p h −1 ∧ p h ∧ p h +1 ∧ p m −1 ∧ p m ∧ p m +1
ph ∧ pi ∧ pm , for some i between h and m[[h < m]], |h − m| (binned)
all combinations of morphological features of heads andmodifiers (if available)w g ∧ ph ∧ wm , w g ∧ wh ∧ pm , p g ∧ wh ∧ wm , etc.
[[h < m]] ∧ [[ g < h]], | g − h| + |h − m|, etc.
wh ∧ pm ∧ ws , wh ∧ wm ∧ ps , wh ∧ wm ∧ ws , etc.
[[h < m]] ∧ [[m < s]], |h − m| + |h − s|, etc.
wh ∧ wh0 ∧ pm ∧ pm−1 , etc.
[[h < m]] ∧ [[h0 < m + 1]], |h0 − m + 1| + |h − m|, etc.
wa ∧ wd and POS back-offs, for noun/verb pairs only
wa , p a (this counts the number of descendants)
wd , pd (this computes the depth of a word/POS in the tree)wh ∧ wm and POS backoffs
1 (this counts the number of non-projective arcs)
ph ∧ pi ∧ pm , for some i between h and m[[h < m]], |h − m| (binned)

Table 7.1: Features used in our parser. Each w designates a word appended with a POS tag,and each p denotes a POS tag only. We also incorporate backed off versions of these features(reducing the context and replacing lexical items by POS tags). We also incorporate versionsof some of these features with lemmas instead of words, and coarse POS tags (when that
information exists in the dataset). The arc lengths are binned (lengths 1, . . . , 5, ≥ 5, ≥ 10)
and converted to binary features. Some variants of these features were first proposed by
McDonald et al. (2005a, 2006); Carreras (2007).

7.5 Experiments
In this section, we evaluate empirically several configurations of turbo parsers.15
Dataset Description. We used 14 datasets with non-projective dependencies from the CoNLL2006 and CoNLL-2008 shared tasks (Buchholz and Marsi, 2006; Surdeanu et al., 2008). These
datasets contain tokenized sentences with pre-computed information regarding the words,
lemmas, fine and coarse part-of-speech (POS) tags predicted by an external tagger, and morphological information (about gender, number, case, etc.). We point to the references above
for additional language-specific details.
In addition, we also used a projective English dataset derived from the Penn Treebank by
applying the standard head rules of Yamada and Matsumoto (2003). As usual, we train on
sections §02–21, use §22 as validation data, and test on §23. We ran SVMTool (Giménez andMarquez, 2004) to obtain automatic POS tags for §22–23.
Training and Test Procedures. We trained by running 10 iterations of the cost-augmented
MIRA algorithm (Crammer et al., 2006) with LP-relaxed decoding (Martins et al., 2009c); we
describe this strategy in further detail in Chapter 8. In our experiments, we did not force
the parser to output projective trees or unique roots for any of the datasets; everything islearned from the data. Following common practice (Charniak and Johnson, 2005; Carreras
et al., 2008), we employed a coarse-to-fine procedure to prune away unlikely candidate arcs,
15 We focus on parsers based on LP-MAP inference; later in Chapter 8 we will compare this sort of parsers
with the ones based on marginal inference, namely the loopy BP parser of Smith and Eisner (2008). Additional
experiments comparing exact decoding with relaxed decoding appear in Martins et al. (2009b).

7.5. EXPERIMENTS

163

as described by Koo and Collins (2010): the pruner is a probabilistic arc-factored model
which we use to eliminate arcs with posterior probability inferior to 10−4 . In addition, we
limit the number of candidate heads to each word to 10.
To ensure valid parse trees at test time, we rounded fractional solutions as describedin Martins et al. (2009b): first, solve the LP relaxation; if the solution is integer, we are
done; otherwise, we consider the restriction of the z-variables which are indexed by arcs,z̃ := (z a ) a∈ A , and project it onto the feasible set Y( x ). That projection can be computed in astraightforward way by finding a maximal arborescence in the directed graph whose weights
are defined by z̃, which can done with the Chu-Liu-Edmonds algorithm. In practice, we havefound that solutions were integral most of the time.
The parts used in our full model are the ones depicted in Figure 7.1. For all the datasets,
we used a common set of features, which we describe in Table 7.1. Note that a subgradientbased method could handle some of those parts efficiently (arcs, consecutive siblings, grandparents, and head bigrams) by composing arc-factored models, head automata, and a sequencelabeler. However, no lightweight decomposition seems possible for incorporating parts forall siblings, directed paths, and non-projective arcs. Table 7.4 shows the first-order logical formulae that encode the constraints in our model. Each formula gives rise to a subproblem
which is efficiently solvable (see Chapter 6).
By ablating some of rows of Table 7.4 we recover known methods:• Resorting to the tree and consecutive sibling formulae gives one of the models in Koo
et al. (2010), with the same linear relaxation (see Section 7.3.2 for a proof of this fact);• Resorting to tree, all siblings, grandparent, and non-projective arcs, recovers the multicommodity flow configuration originally proposed by Martins et al. (2009b); the relaxation is also the same.16
The experimental results are shown in Table 7.2. For comparison, we include the best
published results for each dataset (to the best of our knowledge), among transition-basedparsers (Nivre et al., 2006; Huang and Sagae, 2010), graph-based parsers (McDonald et al.,2006; Koo and Collins, 2010), hybrid methods (Nivre and McDonald, 2008; Martins et al.,2008a), and turbo parsers (Martins et al., 2010f; Koo et al., 2010). Our full model achieved
the best reported scores for 7 datasets. The last two columns show a consistent improvement(with the exceptions of Chinese and Arabic) when using the full set of features over a second
order model with grandparent and consecutive siblings, which is our reproduction of themodel of Koo et al. (2010).17
Feature ablation and error analysis. We conducted a simple ablation study by training
several models on the English PTB with different sets of features. Table 7.3 shows the results.
As expected, performance keeps increasing as we use models with greater expressive power.
Figure 7.7 shows examples of parses that were correctly predicted by the full model, but notby the G+CS model.
16 As pointed out in Section 7.2, although Martins et al. (2009b) also incorporated consecutive siblings in oneof their configurations, our constraints are tighter than theirs.
17 Note however that the actual results of Koo et al. (2010) are higher than our reproduction, as can be seenin the second column. The differences are due to the features that were used and on the way the models
were trained. The cause is not search error: exact decoding with an ILP solver (CPLEX) revealed no significant
difference with respect to our G+CS column.

CHAPTER 7. TURBO PARSERS164

ArabicBulgar.
ChineseCzech
EnglishDanishDutch
GermanJapane.
Portug.
SloveneSpanishSwedishTurkishPTB §23

80.18
92.88
91.89
88.78
92.57
91.78
85.81
91.49
93.42
93.03
86.21
87.04
91.36
77.55
93.04

Best known UAS
Martins et al. (2008a)
Martins et al. (2010f)
Martins et al. (2010f)
Martins et al. (2010f)
Koo et al. (2010)
Koo et al. (2010)
Koo et al. (2010)
Martins et al. (2010f)
Martins et al. (2010f)
Koo et al. (2010)
Koo et al. (2010)
Martins et al. (2010f)
Koo et al. (2010)
Koo et al. (2010)
Koo and Collins (2010)

G+CS
81.12
93.04
91.05
88.80
92.45
91.70
84.77
91.29
93.62
92.05
86.09
85.99
89.94
76.24
92.19

Full
81.10 (-0.02)
93.50 (+0.46)
90.62 (-0.43)
89.46 (+0.66)
92.68 (+0.23)
91.86 (+0.16)
85.53 (+0.76)
91.89 (+0.60)
93.72 (+0.10)
92.29 (+0.24)
86.95 (+0.86)
86.74 (+0.75)
90.16 (+0.22)
76.64 (+0.40)
92.53 (+0.34)

Table 7.2: Unlabeled attachment scores, excluding punctuation. In columns 3–4, “Full” is ourfull model, and “G+CS” is our reproduction of the model of Koo et al. (2010), i.e., the sameas “Full” but with all features ablated excepted for grandparents and consecutive siblings.

PTB §22
PTB §23

AF
91.02
91.36

+G+CS
92.13
92.19

+AS
92.32
92.41

+NP
92.36
92.50

Full
92.41
92.53

Table 7.3: Feature ablation experiments. AF is an arc-factored model; +G+CS adds grandparent and consecutive siblings; +AS adds all-siblings; +NP adds non-projective arcs; Full
adds the bigram and directed paths.

7.5. EXPERIMENTS

165

(A)

*

He

added

:

(B)

*

‘‘

This

further

confusesretailers

,

(C)

* How

such

remarkstranslateinto

policy

(D)

*

1987

,

In

‘‘

such

We

learnedsellingalessoncontributed

in

’’

she

wo

to

1987

volatility

says

n’t

a

about

becomesnowball

clear

for

monthseffectFigure 7.7: Sentences of the English non-projective dataset (CoNLL 2008) that were correctly
parsed by the full model, but not by the G+CS model. Arcs shown in blue are those thatwere missed by the G+CS model. See text for an explanation.
In (A), the G+CS model has predicted 1987 as the parent of about. The full model got
it right, arguably due to the nonprojectivity features that find the nonprojective arc lesson→about to be likely.
In (B), the G+CS model attached further to retailers. The word further forms an adverbial
phrase which enjoys considerable freedom about where it can be placed in a sentence. Hence,features that look at all siblings attached to a head word (rather than just consecutive ones)
may help parsing sentences without a rigid word ordering.
(C) is an example where path features seem to help. The G+CS model predicted 3 arcs
incorrectly, because it assumed that policy won’t become clear for months was a phrase (hence
it predicted *→translate→wo(n’t)→policy). The full model may have found unlikely the long
path that would descend from translate, and prefered a more horizontal parse.
Example (D) seems simple to parse; the word snowball, however, was incorrectly tagged
as a verb by the part-of-speech tagger and confused the G+CS model, which predicted
to→snowball→effect. The full model got it right, arguably because of the head bigram features, which give a low score to configurations in which two consecutive words (in this casea and snowball) have crossing dependency attachments in opposite sides. This shows that aparser with many features may gain robustness to errors in the pipeline.
Convergence speed and optimality. Figure 7.8 compares the performance of AD3 and the
projected subgradient algorithms in the validation section of the PTB.18 For the second order model, the subgradient method has more slaves than in Koo et al. (2010): it has a slave
imposing the tree constraint (whose subproblems consists on finding a minimum spanningtree) and several for the all-sibling parts, yielding an average number of 310.5 and a maxi18 The learning rate in the subgradient method was set as η = η / (1 + Nt0incr ( t )), as in Koo et al. (2010), where
Nincr (t) is the number of dual increases up to the tth iteration, and η0 is chosen to maximize dual decrease after
20 iterations (on a per sentence basis). Those preliminary iterations are not plotted in Figure 7.8.

CHAPTER 7. TURBO PARSERS
92
91
90
89
88
87
86
850

Accuracy

ADMM Full
Subgrad Full
ADMM Sec Ord
Subgrad Sec Ord
200

400
600
Iterations

Stopping Criteria

100

800

1000

Certificates (%)

UAS (%)

166

80

ADMM Full (Tol<0.001)
ADMM Full (Exact)
Subgrad Full (Exact)
ADMM Sec Ord (Tol<0.001)
ADMM Sec Ord (Exact)
Subgrad Sec Ord (Exact)

60
40
20
00

200

400
600
Iterations

800

1000

Figure 7.8: UAS including punctuation (left) and fraction of optimality certificates (right)
accross iterations of the subgradient and AD3 algorithms, in PTB §22. “Full” is our fullmodel; “Sec Ord” is a second-order model with grandparents and all siblings, for which thesubgradient method uses a coarser decomposition with the tree-based factor graph. Since
subgradient and AD3 are solving the same problems, the solid lines (as the dashed ones)
would meet in the limit, however subgradient converges very slowly for the full model. The
right plot shows optimality certificates for both methods, indicating that an exact solutionof P has been found; for AD3 we also plot the fraction of instances that converged to an
accurate solution of P0 (primal and dual residuals < 10−3 ) and hence can be stopped.
mum of 4310 slaves. These numbers are still manageable, and we observe that a “good” UAS
is achieved relatively quickly. The AD3 method has many more slaves due to the multicommodity flow constraints (average 1870.8, maximum 65446), yet it attains optimality sooner,as can be observed in the right plot. For the full model, the subgradient-based method becomes extremely slow, and the UAS score severely degrades (after 1000 iterations it is 2% lessthan the one obtained with AD3 , with very few instances having been solved to optimality).
The reason is the number of slaves: in this configuration and dataset the average number ofslaves per instance is 3327.4, and the largest number is 113207. On the contrary, AD3 keeps
a robust performance, with a large fraction of optimality certificates in early iterations.
Runtime and caching strategies. Despite its suitability to problems with many overlapping
components, our parser is still 1.6 times slower than Koo et al. (2010) (0.34 against 0.21
sec./sent. in PTB §23), and is far below the speed of transition-based parsers (e.g., Huang
and Sagae (2010) take 0.04 sec./sent. on the same data, although accuracy is lower, 92.1%).
Our implementation, however, is not fully optimized. We next describe how considerable
speed-ups are achieved by caching the subproblems, following a strategy similar to Koo et al.
(2010).
Figure 7.9 illustrates the point. After a few iterations, many variables u(r ) see a consensus
being achieved (i.e., ut (r ) = zts+1 (r ), ∀s) and enter an idle state: they are left unchanged by
the u-update in Eq. 7.74, and so do the Lagrange variables λts+1 (r ) (Eq. 7.75). If by iterationt all variables in a subproblem s are idle, then zts+1 (r ) = zts (r ), hence the subproblem does
not need to be resolved.19 Fig. 7.9 shows that many variables and subproblems are left
untouched after the first few rounds.
Finally, Fig. 7.10 compares the runtimes of our implementation of AD3 with those achieved
19 Even if not all variables are idle in s, caching may still be useful: note that the z-updates in Eq. 7.73 tend tobe sparse for our subproblems (these are Euclidean projections onto polytopes with 0/1 vertices, which tend tohit corners). Another trick that may accelerate the algorithm is warm-starting: since many subproblems involvea sort operation, storing the sorted indexes may speedup the next round.

7.6. CONCLUSIONS AND FUTURE WORKSec Ord ADMM
% active msgs
% active subproblems% active vars

% active

80
60

80

40
20
0

Full ADMM% active msgs
% active subproblems% active vars

100
% active

100

167

60
40
20

200

400

600
Iterations

800

0

1000

200

400

600
Iterations

800

1000

Figure 7.9: Fraction of active variables, subproblems and messages along AD3 iterations(second order model and full model). The number of active messages denotes the total
number of variables (active or not) that participate in an active factor.
Elapsed Times
Time CPLEX (sec.)

101
100
10-1
10-2
10-3
10-3

10-2

10-1
100
Time ADMM (sec.)

101

Figure 7.10: Runtimes of AD3 and CPLEX on PTB §22 (each point is a sentence). Averageruntimes are 0.362 (AD3 ) and 0.565 sec./sent. (CPLEX).
by a state-of-the-art LP solver, CPLEX, in its best performing configuration: the simplex algorithm applied to the dual LP. We observe that AD3 is faster in some regimes but slower
in others. For short sentences (< 15 words), AD3 tends to be faster. For longer sentences,
CPLEX is quite effective as it uses good heuristics for the pivot steps in the simplex algorithm; however, we observed that it sometimes gets trapped on large problems. Note alsothat AD3 is not fully optimized, and that it is much more amenable to parallelization than
the simplex algorithm, since it is composed of many independent slaves. This suggests
potentially significant speed-ups in multi-core environments.

7.6 Conclusions and Future WorkIn this chapter, we have introduced turbo parsers, feature-rich dependency parsers that run
approximate inference in a loopy factor graph, ignoring the global effects caused by theloops. We have publicly released our turbo parsers as an open-source project.20
We started by deriving a new polynomial-size ILP formulation for dependency parsing
based on multi-commodity flows, contrasting with a previous formulation that requires exponentially many constraints (Riedel and Clarke, 2006). Then, we constructed a factor graphwhose LP-MAP inference problem is equivalent to the linear relaxation of the previous ILP;
hence, the resulting parser is a turbo parser. For other turbo parsers (Smith and Eisner,
2008; Koo et al., 2010), we characterize the underlying factor graphs and the optimization
problems that are addressed by the corresponding inference algorithms.
We then applied the AD3 algorithm, introduced in Chapter 6, to solve our linear relax20 See http://www.ark.cs.cmu.edu/TurboParser/.

CHAPTER 7. TURBO PARSERS

168

ation. We devised a caching technique that provides significant speed-ups. We evaluatedour parsers in 14 languages, with state-of-the-art results. Our experiments confirmed that
the AD3 algorithm is particularly suitable for handling many overlapping components, comparing favourably against the projected subgradient method in several aspects: it is faster toreach a consensus, it has better stopping conditions, and it works better in non-lightweightdecompositions.21
There are several ways in which we could extend the work presented in this chapter.
Many other global features or constraints could be thrown into the parsing model, for example using prior linguistic knowledge. An example of global features that might be usefulare features that depend on word spans—this sort of features is useful in phrase-structure
parsers, and can be incorporated here by defining logical operations on the path variables.
Basically, the span of a word h is (i, j) if i is the leftmost word for which there is a path fromh to i, and j is the rightmost word for which there is a path from h to j. It is straighforward
to incorporate those features by employing the logic factors described in Chapter 5.
Other syntactic formalisms, such as phrase-structure grammars, tree-adjoining grammars, or lexical-functional grammars may also be addressed with similar approaches. AD3
may be useful in other frameworks involving logical constraints, such as the models for compositional semantics presented by Liang et al. (2011). Non-logical constraints may also yieldefficient subproblems, e.g., the budget constraints in summarization and compression (Clarkeand Lapata, 2008; Martins and Smith, 2009; Berg-Kirkpatrick et al., 2011). The marginal polytope of such factors (for a budget B) is(
ZB :=(z1 , . . . , zK ) ∈ [0, 1]

KK∑ zk ≤ B);(7.76)

k =1

computing the MAP amounts to obtaining the B variables with the highest positive scores.
Projecting onto the set ZB can be done with cyclic projection algorithms, or employing theactive set method described in Chapter 6.

21 In Martins et al. (2011a), we have also shown that AD3 outperforms MPLP and Star-MSD, although ourexperiments there are for dependency parsers with fewer features.

∃!h : arc(h, m), m 6= 0
flow(h, m, k ) ⇒ arc(h, m)path(m, d) := ∃!h : flow(h, m, d), m 6= 0path(h, d) := ∃!m : flow(h, m, d)path(0, m) := true, flow(h, m, m) := truesibl(h, m, s) := arc(h, m) ∧ arc(h, s)
grand( g, h, m) := arc( g, h) ∧ arc(h, m)bigram(b, h, m) := arc(b, m − 1) ∧ arc(h, m), m 6= 0
lastsibl(h, m, m) := arc(h, m)∃!m ∈ [h, k] : lastsibl(h, m, k)lastsibl(h, m, k) := lastsibl(h, m, k + 1)
⊕ nextsibl(h, m, k + 1)
arc(h, m) := ∃!s ∈ [h, m] : nextsibl(h, s, m)
nonproj(h, m) := arc(h, m) ∧ ∃k ∈ [h, m] : ¬path(h, k)

RuntimeO( L log L)
O (1)O( L log L)
O(n log n)
O (1)
O (1)
O (1)
O( L log L)
O (1)O( L log L)
O( L log L)# Slaves
O( L)
O ( L3 )O ( L2 )O ( n2 )O ( L3 )O ( n3 )O ( n3 )O ( L2 )O ( L3 )O ( L2 )O ( L2 )By definition

Head automaton model

By definition
By definition
By definition

Paths and flows are consistent

Description
Each non-root word has a head
Only active arcs may carry flow

Table 7.4: First-order logic formulae underlying our dependency parser. The basic parts are the predicate variables arc(h, m) (indicating an
arc linking head h to modifier m), path( a, d) (indicating a directed path from ancestor a to descendant d), nextsibl(h, m, s) (indicating that(h, m) and (h, s) are consecutive siblings), nonproj(h, m) (indicating that (h, m) is a non-projective arc), as well as the auxiliary variables
flow(h, m, d) (indicating that arc (h, m) carries flow to d), and lastsibl(h, m, k ) (indicating that, up to position k, the last seen modifier ofh occurred at position m). The non-basic parts are the pairwise factors sibl(h, m, s), grand( g, h, m), and bigram(b, h, m); as well as each
logical formula. Columns 3–4 indicate the number of parts of each kind, and the time complexity for solving each subproblem with AD3 .
For a sentence of length L, there are O( L3 ) parts and the total complexity is O( L3 log L).

Nonproj. Arc

All siblings
Grandp.
Head Bigram
Consec. Sibl.

Tree

7.6. CONCLUSIONS AND FUTURE WORK
169

170

CHAPTER 7. TURBO PARSERS

Part III

Learning

171

Chapter 8

Learning Structured Classifiers with
Dual Coordinate AscentIn the previous three chapters, we have presented novel contributions for structured inference, with applications to natural language processing. Along the way, we have always
assumed possession of a pre-trained structured model, ready to make predictions. We now
switch gears and address the learning problem, which concerns the training of such models.
The following original contributions are presented in this chapter:• We present a unified framework for learning structured classifiers that handles a widefamily of convex loss functions, properly including CRFs, structured SVMs, and thestructured perceptron (see Chapter 3 for background on these losses).
• We introduce a new class of online algorithms that optimize any loss in this family. For
the structured hinge loss, the algorithm reduces to 1-best MIRA (Crammer and Singer,
2003; McDonald et al., 2005a).1 For the structured logistic loss, we obtain a variant of
MIRA for CRFs.
• We show that these algorithms implicitly perform coordinate ascent in a dual formulation, generalizing the framework established by Shalev-Shwartz and Singer (2006) toa larger set of loss functions and to structured output prediction.
• We address the impact of approximate inference on the learning problem.
Our experiments on two NLP problems (named entity recognition and dependency parsing)show that our algorithm converges to accurate models at least as fast as stochastic gradientdescent, without the need to specify any learning rate parameter.
Some of the results presented in this chapter—namely the family of loss functions and thenew online algorithm—were originally introduced in Martins et al. (2010f), and are described
more broadly in a technical report (Martins et al., 2010c). We will refer occasionally to resultspresented in Martins et al. (2009c), which concerns the impact of approximate LP-MAP
inference in the learning procedure.
1 Also called “online passive-aggressive” by Crammer et al. (2006).

173

174

CHAPTER 8. LEARNING WITH DUAL COORDINATE ASCENT

8.1 Motivation and Previous Work
We have seen in Chapter 2 that many important problems in NLP, such as tagging, namedentity recognition, and parsing, require classifiers with structured outputs. Learning thoseclassifiers discriminatively typically involves the minimization of a regularized loss function, of the kind described in Section 3.4. The well-known cases of CRFs (Lafferty et al.,2001) and structured SVMs (Taskar et al., 2003; Tsochantaridis et al., 2004; Altun et al., 2003)
correspond to different choices of loss functions. For large-scale settings, the underlying
optimization problem is often difficult to tackle in its batch form, making online algorithms
desirable. Some examples, discussed in Section 3.5, are the structured perceptron (Collins,
2002a), stochastic gradient descent (SGD) (LeCun et al., 1998), and the margin infused relaxed algorithm (MIRA) (Crammer and Singer, 2003; Crammer et al., 2006).
In this chapter, we start by establishing a unified representation for several convex loss
functions traditionally used in structured classification (Section 8.2). We will use results
derived in Section 5.2.2 (namely, Proposition 5.3) to describe how all these losses can be
expressed in variational form as optimization problems over the marginal polytope. After
doing so, we make use of convex duality to derive new online learning algorithms (Section 8.4)
that share the “passive-aggressive” property of MIRA but can be applied to a wider varietyof loss functions. This includes as a particular case the logistic loss that underlies CRFs,
yielding a procedure analogous to MIRA for training probabilistic models. The primal-dual
interpretation of such algorithms builds on previous work by Shalev-Shwartz and Singer(2006) and Kakade and Shalev-Shwartz (2008), who regard them as dual coordinate ascentprocedures. We generalize their framework to a larger set of loss functions and to structured
prediction.
The updates that we derive in Section 8.4 share the remarkable simplicity of SGD, with
an important advantage:

Our algorithms do not require the specification of a learning rate parameter.

That is, they sidestep what is arguably the most annoying aspect of SGD: tuning a learning
rate parameter and specifying an annealing schedule. Instead, the step sizes are automatically computed as a function of the loss and of its gradient. This is a difference with respect
to standard gradient methods, which only require gradient information for computing the
updates, dispensing the evaluation of the loss function itself. Nevertheless, the additional
computation required for loss evaluations is negligible in the case of the family of loss functions herein addressed, since the methods used to compute the gradient also provide thevalue of the loss for free.
Two important problems in NLP provide an experimental testbed (Section 8.5): namedentity recognition and dependency parsing. We employ feature-rich models where exact inference is sometimes intractable. To be as general as possible, we devise a framework thatfits any structured classification problem representable as a factor graph with soft and hardconstraints (Section 8.2); this includes problems with loopy graphs, such as some variants ofthe dependency parsers of Smith and Eisner (2008).

8.2. A FAMILY OF LOSS FUNCTIONS FOR STRUCTURED CLASSIFICATION

175

8.2 A Family of Loss Functions for Structured Classification
We assume the general framework discussed in Chapter 3 for learning structured linear
models, in which the learning problem takes the form:
minimize Ω(w) + N1 ∑nN=1 L(w; x n , yn )w.r.t.
w ∈ RD ,(8.1)

where Ω : RD → R is a convex regularizer and L : RD × X × Y → R is a convex loss function.
In this chapter, we assume L2 -regularization, Ω(w) := ΩλL2 (w) = λ2 kwk2 , and consider thefollowing family of loss functions parametrized by non-negative scalars β and γ:
L β,γ (w; x, y) :=

h 
i

1
log ∑ exp β w · f ( x, y0 ) − f ( x, y) + γρ(y0 , y) .
β
y 0 ∈Y( x )(8.2)

This family subsumes some well-known cases:• The logistic loss (in CRFs), LCRF (w; x, y) := − log Pw (y| x ), corresponds to β = 1 and
γ = 0 (cf. Eq. 3.16).
• The hinge loss of structured SVMs, LSSVM (w; x, y) := maxy0 ∈Y(x) w> ( f ( x, y0 ) − f ( x, y)) +ρ(y0 , y), corresponds to the limit case β → ∞ and any γ > 0 (cf. Eq. 3.20).
• The loss LSP underlying the structured perceptron is obtained for β → ∞ and γ = 0(cf. Eq. 3.31).
• The softmax-margin loss recently proposed in Gimpel and Smith (2010) is obtained with
β = γ = 1.
For any choice of β > 0 and γ ≥ 0, the resulting loss function is convex in w, since, up to ascale factor, it is the composition of the (convex) log-sum-exp function with an affine map.2
In Section 8.4 we present a dual coordinate ascent online algorithm to handle the learning
problem in Eq. 8.1, for this family of losses.

8.3 Loss Evaluation and Differentiation
Recall that in Chapter 5 we have considered constrained factor graphs G , which assume adecomposition of the feature vector over places,f ( x, y) = ∑ f p ( x, y p ),

(8.3)

p ∈P

where the places are the variable nodes and the soft factor nodes of the factor graph G . We
considered the corresponding set of parts R = {( p, y p ) | p ∈ P, y p ∈ Y p }, and formed theD-by-|R| feature matrix F( x ), whose columns are the local feature vectors f p ( x, y p ), for
2 Some important non-convex losses can also be written as differences of losses in this family. By defining
δL β,γ = L β,γ − L β,0 , the case β = 1 yields δL β,γ (w; x, y) = log Ew exp ρ(Y, y), which is an upper bound on
Ew ρ(Y, y), used in minimum risk training (Smith and Eisner, 2006). For β = ∞, δL β,γ becomes a structured ramp
loss (Collobert et al., 2006).

CHAPTER 8. LEARNING WITH DUAL COORDINATE ASCENT

176

each ( p, y p ) ∈ R. Then, we discussed the geometry of constrained factor graphs, and in
Proposition 5.3 we established the following variational representation of the log-partition
function (cf. Eq. 5.20):
log Z (w, x ) =maxµ∈MARG(G )w > F ( x ) µ + H ( µ ),(8.4)

where H is the entropy function expressed in terms of the marginal parametrization µ =µ(w), i.e.:(
H (µ) :=

Ew [− log Pw (Y | X = x )] if there is w such that µ = µ(w)−∞otherwise.

(8.5)

We now invoke Proposition 5.3 to derive a variational expression for evaluating any loss
L β,γ (w; x, y) in (8.2), and compute its gradient as a by-product.3 This is crucial for thelearning algorithms to be introduced in Section 8.4. Our only assumption is that the costfunction ρ(y0 , y) is decomposable. Recall from Definition 3.3 that a cost function is called
decomposable if it can be written as ρ(yb, y) = ∑ p∈P ρ p (ybp , y p ), where each ρ p is a local costfunction.
Lemma 8.1 (Decomposable cost as a linear function.) Given y ∈ Y( x ), any decomposable costfunction ρ(., y), seen as a function of the first argument, can be written as a linear function of theoutput indicator vector:
ρ(yb, y) = c(y) · χ(yb),(8.6)
where c(y) ∈ R|R| .
Proof. Trivial: set [c(y)] p,y0p = ρ p (y0p , y p ).
For example, for the Hamming loss, Lemma 8.1 holds with c(y) = 1 − χ(y); Other
examples are shown in Taskar et al. (2006b). Crucially, if the cost ρ in the family (8.2) isdecomposable, then we have a variational representation for the loss functions in this family,as the next proposition asserts.
Proposition 8.2 (Loss and gradient.) Let ρ be a decomposable cost function written in the form
(8.6). The following variational representation for the loss L β,γ holds:
L β,γ (w; x, y) =maxµ∈MARG(G )w> F( x )(µ − χ(y)) +1
H (µ) + γc(y)> µ.
β

(8.7)

Furthermore, the gradient of L β,γ at w is given by:∇ L β,γ (w; x, y) = F( x )(µb − χ(y)).

(8.8)

where µb is a maximizer in Eq. 8.7.
Proof. Let θ = F( x )> w be the vector of factor log-potentials. Under the decomposable
cost assumption, L β,γ (w; x, y) becomes expressible in terms of the log-partition function of
3 Our description also applies to the (non-differentiable) hinge loss case, when β → ∞, if we replace all

instances of “the gradient” in the text by “a subgradient.”

8.4. ONLINE LEARNING WITH DUAL COORDINATE ASCENT

177

a distribution whose log-potentials are set to β(θ + γc(y)). From the first statement of
Proposition 5.3, we immediately obtain Eq. 8.7. Now, let µb be a maximizer in (8.7); from thesecond statement of Proposition 5.3 we obtain Eq. 8.8.
Two examples follow.
Sequence Labeling. Without hard constraints, the graphical model does not contain loops,
and therefore L β,γ (w; x, y) and ∇ L β,γ (w; x, y) may be easily computed by setting the logpotentials as described above and running the forward-backward algorithm.
Dependency Parsing. For the arc-factored model, L β,γ (w; x, y) and ∇ L β,γ (w; x, y) may becomputed exactly by modifying the log-potentials, and invoking the matrix-tree theorem tocompute the log-partition function and the marginals (Smith and Smith, 2007; Koo et al.,2007; McDonald and Satta, 2007), as seen in Chapter 2. For richer models where arc interactions are considered, exact inference is intractable. Both the marginal polytope and theentropy lack concise closed form expressions. We have discussed two factor graph representations in Section 7.3, which underlie two approximate approaches: the loopy BP algorithm
for computing pseudo-marginals (Smith and Eisner, 2008); and the LP-relaxation method forapproximating the most likely parse tree (Martins et al., 2009b). Both methods optimize over
outer bounds of the marginal polytope, as discussed in Section 7.3.

8.4 Online Learning with Dual Coordinate Ascent
We now proceed to the main contribution of this chapter, a dual coordinate ascent approach tolearn the model parameters w. This approach extends the primal-dual view of online algorithms put forth by Shalev-Shwartz and Singer (2006) to structured classification; it handles
any loss in the family (8.2). In the case of the hinge loss, we recover the online passiveaggressive algorithm of Crammer et al. 2006 (also known as MIRA), as well as its K-best
variants. With the logistic loss, we obtain a new passive-aggressive algorithm for CRFs.
Start by noting that the learning problem in Eq. 8.1 is not affected by scaling the objective
by N. It will also be convenient to pull the regularization constant λ out of the regularizer;
in the L2 case this is done by writing ΩλL2 = λΩ1L2 . With this in mind, consider a sequence ofprimal objectives P1 (w), . . . , PN +1 (w) to be minimized, each of the formt −1

Pt (w) = λNΩ(w) + ∑ L(w; xi , yi ).
i =1

Our goal is to minimize PN +1 (w); for simplicity we consider online algorithms with only
one pass over the data, but the analysis can be extended to the case where multiple epochs
are allowed.
Below, we let R̄ := R ∪ {+∞} be the extended reals and, given a function f : RD → R̄,
we denote by f ? : RD → R̄ its convex conjugate, f ? (v) = supu u · v − f (u) (see Appendix B
for a background of convex analysis). The next proposition, proved in (Kakade and ShalevShwartz, 2008), states a generalized form of Fenchel duality, which involves a dual vectorξ i ∈ Rd per each instance.

CHAPTER 8. LEARNING WITH DUAL COORDINATE ASCENT

178

Proposition 8.3 (Kakade and Shalev-Shwartz 2008) The Lagrange dual of minw Pt (w) is
max Dt (ξ 1 , . . . , ξ t−1 ),(8.9)

ξ 1 ,...,ξ t−1

where
Dt (ξ 1 , . . . , ξ t−1 ) = −λNΩ?

1 t −1
−
ξi
λN i∑
=1

!

t −1

− ∑ L ? ( ξ i ; x i , y i ).

(8.10)i =1

If Ω(w) = Ω1L2 (w) = 12 kwk2 , then Ω = Ω? , and strong duality holds for any convex L, i.e.,
Pt (w∗ ) = Dt (ξ 1∗ , . . . , ξ ∗t−1 ) where w∗ and ξ 1∗ , . . . , ξ ∗t−1 are respectively the primal and dual optima.
Moreover, the following primal-dual relation holds:w∗ = −

1 t −1 ∗
ξi .
λN i∑
=1

(8.11)

We can therefore transform our problem into that of maximizing D N +1 (ξ 1 , . . . , ξ N ). Dualcoordinate ascent (DCA) is an umbrella name for algorithms that manipulate a single dual
coordinate at a time. In our setting, the largest such improvement at round t is achievedby ξ t := arg maxξ Dt+1 (ξ 1 , . . . , ξ t−1 , ξ ). The next proposition characterizes the mapping of
this subproblem back into the primal space, shedding light on the connections with known
online algorithms.
−1
1
Proposition 8.4 Let wt := − λN∑it=1 ξ i . The Lagrange dual of maxξ Dt+1 ( ξ 1 , . . . , ξ t−1 , ξ ) is

min
w

λN
kw − wt k2 + L(w; x t , yt ).
2(8.12)

Proof. From (8.10),
max Dt+1 (ξ 1 , . . . , ξ t−1 , ξ )
ξ

=
=
=(i)
=(ii)
=
=(iii)

1
max −
2λN
ξ

t −1

∑ ξi + ξ

i =1

2

t −1

− L? (ξ; x t , yt ) − ∑ L? (ξ i ; xi , yi )
i =1

1
2
−λNwt + ξ − L? (ξ; x t , yt ) + constant
2λN
ξ
1
2
max −
−λNwt + ξ − max(ξ · w − L(w; x t , yt )) + constant
w
2λN
ξ
1
2
min max −−λNwt + ξ − ξ · w + L(w; x t , yt ) + constant
w
2λN
ξ


1
t 2
min max ξ · (−w) −ξ − λNw+ L(w; x t , yt ) + constant
w
2λN
ξ
λN
2
min
w − wt + L(w; x t , yt ) + constant,
w
2
max −

(8.13)

where in (i) we invoked the definition of convex conjugate; in (ii) we interchange min andmax since strong duality holds (as stated by Kakade and Shalev-Shwartz (2008), a sufficientcondition is that Ω is strongly convex, L is convex and dom L is polyhedral); and in (iii) weused the facts that Ω(w) = kwk2 /2 is conjugate of itself, and that g(u) = t f (u − u0 ) implies

8.4. ONLINE LEARNING WITH DUAL COORDINATE ASCENT

179

Algorithm 12 Dual coordinate ascent (DCA)
input: data D, regularization constant λ, number of epochs K
initialize w1 = 0; set N = |D| and T = NK
for t = 1 to T do
choose n = n(t) ∈ {1, . . . , N } and take training pair ( x n , yn )update wt+1 by solving (8.12) exactly or approximately (see Algorithm 13)
end foroutput: the last weight vector w
b ← w T +1 , or the average w
b ← T1 ∑tT=1 wt+1 .
Algorithm 13 Parameter updates in DCA
input: current model wt , instance ( x, y), regularization constant λobtain χ(y) from ysolve the variational problem in Eq. 8.7 to obtain µb and L(wt , x, y)tcompute the gradient: ∇ L β,γ (w ; x, y) := F( x )(µb − χ(y)) (Eq. 8.8)
compute stepsize:
1
L(wt ; x, y),ηt := min
λN k∇ L(wt ; x, y)k2
update the model: wt+1 := wt − ηt ∇ L(wt ; x, y)
output: wt+1

g? (v) = u0 · v + t f ? (v/t).

Assembling these pieces together yields Alg. 12, where the solution of (8.12) is carriedout by Alg. 13, as explained next. While the problem in Eq. 8.12 is easier than the batch
problem in Eq. 8.1, an exact solution may still be prohibitively expensive in large-scale settings, particularly because it has to be solved repeatedly. We thus adopt a simpler strategy
that still guarantees some improvement in the dual. Noting that L is non-negative, we may
rewrite the problem in Eq. 8.12 as

w.r.t.

λNk w − w t k2 + ξ
2w ∈ RD , ξ ≥ 0s.t.

L(w; x t , yt ) ≤ ξ.

minimize(8.14)

From the convexity of L, we may take its first-order Taylor approximation around wt to
obtain the lower bound L(w; x t , yt ) ≥ L(wt ; x t , yt ) + (w − wt ) · ∇ L(wt ; xt , yt ). Therefore thetrue minimum in (8.12) is lower bounded by the solution value of the following problem:
minimizew.r.t.
s.t.

λN
k w − w t k2 + ξ
2w ∈ RD , ξ ≥ 0
L(wt ; x t , yt ) + (w − wt ) · ∇ L(wt ; x t , yt ) ≤ ξ.

(8.15)

This is a simple Euclidean projection problem with slack, which admits the closed form

CHAPTER 8. LEARNING WITH DUAL COORDINATE ASCENT

180

solution w∗ = wt − ηt ∇ L(wt ; x t , yt ), withηt = minL(wt ; x t , yt )1,
λN k∇ L(wt ; x t , yt )k2

.

(8.16)

Example: 1-best MIRA. If L is the structured hinge loss, we obtain, from Eq. 8.8:

∇ LSSVM (w; x, y) = F( x )(χ(yb) − χ(y)) = f ( x, yb) − f ( x, y),(8.17)

where yb = arg maxy0 ∈Y(x) w · ( f ( x, y0 ) − f ( x, y)) + ρ(y0 , y). The update becomes
wt+1 = wt − ηt ( f ( x t , ybt ) − f ( x t , yt )),(8.18)with
ηt = min

1 wt · ( f ( x t , ybt ) − f ( x t , yt )) + ρ(ybt , yt ),λN
k f ( x t , ybt ) − f ( x t , yt )k2

.

(8.19)

This is precisely the max-loss variant of the 1-best MIRA algorithm (Crammer et al., 2006,
Sect. 8), which we have described in our Section 3.5.4. Hence, while MIRA was originallymotivated by a conservativeness-correctness tradeoff, it turns out that it also performs coordinate ascent in the dual.

Example: CRFs. This framework immediately allows us to extend 1-best MIRA for CRFs,
which optimizes the structured logistic loss. In that case, the exact problem in (8.14) can be
expressed as

w.r.t.

λNk w − w t k2 + ξ
2w ∈ RD , ξ ≥ 0s.t.

− log Pw (yt | x t ) ≤ ξ.

minimize(8.20)

In words: stay as close as possible to the previous parameter vector, but correct the modelso that the conditional probability Pw (yt | xt ) becomes large enough. From Eq. 8.8, we have

∇ LCRF (w; x, y) = F( x )(µb − χ(y))= Ew [ f ( x, Y )] − f ( x, y),(8.21)

where now µb is an expectation instead of a mode. The update becomes
wt+1 = wt − ηt (Ewt [ f ( x t , Y t )] − f ( x t , yt )),(8.22)

with
1 wt · (Ewt [ f ( x t , Y t )] − f ( x t , yt )) + H ( Pwt (.| xt ))
= min
,
λN
kEwt [ f ( x t , Y t )] − f ( x t , yt )k2

1− log Pw (yt | x t )
= min
,
.
λN kEwt [ f ( x t , Y t )] − f ( x t , yt )k2


ηt



(8.23)

8.5. EXPERIMENTS

181

Thus, the difference with respect to standard 1-best MIRA (8.19) consists of replacing the feature vector of the loss-augmented mode f ( x t , ybt ) by the expected feature vector Ewt [ f ( x t , Y t )],and the value of the cost function ρ(ybt , yt ) by that of the entropy function H ( Pwt (.| x t )). The difference with respect to SGD is that the stepsize ηt is automatically adjusted through Eq. 8.23.
Example: k-best MIRA. Tighter approximations to the problem in Eq. 8.12 can be built
by using the variational representation machinery; see Eq. 8.7 for losses in the family L β,γ .
Plugging this variational representation into the constraint in (8.14) we obtain the following
semi-infinite quadratic program:w.r.t.

λNk w − w t k2 + ξ
2w ∈ RD , ξ ≥ 0s.t.

w ∈ H(µ; yt , β, γ),minimize∀µ ∈ MARG(G ),(8.24)

where each H(µ; y, β, γ) := {w ∈ RD | a> w ≤ b} is a half-space, with a = F( x )(µ − χ(y))
and b = ξ − γ(c> µ) − β−1 H (µ). The constraint set in (8.24) is a convex set defined by theintersection of uncountably many half-spaces (indexed by the points in the marginal polytope).4 Our first-order Taylor approximation consisted of relaxing the problem in Eq. 8.24
by discarding all half-spaces except the one indexed by the µb which solved the variational
problem (8.7). However, tigher relaxations can be obtained by keeping some of the otherhalf-spaces. For the hinge loss, rather than just using the mode µb = χ(yb), we may well rank
the k-best outputs and add a half-space constraint for each. This procedure approximatesthe constraint set by a polyhedron and the resulting problem can be addressed using rowaction methods, such as Hildreth’s algorithm (Censor and Zenios, 1997). This corresponds
precisely to k-best MIRA.5

8.5

Experiments

We report experiments on two tasks: named entity recognition and dependency parsing.6 For
each, we compare DCA (Algorithm 12) with SGD. We report results for several values of theregularization parameter C = 1/(λN ). The learning rate schedule for SGD is set according
to the formula ηt = η/(1 + t/N ), as suggested by LeCun et al. (1998). We choose η using
dev-set validation after a single epoch, as in Collins et al. (2008).
Named Entity Recognition. We use the English data from the CoNLL 2003 shared task (Tjong
Kim Sang and De Meulder, 2003), which consist of English news articles annotated with four
entity types: person, location, organization, and miscellaneous. We used a standard set offeature templates, as in Kazama and Torisawa (2007), with token shape features (Collins,
4 Interestingly, when the hinge loss is used, only a finite (albeit exponentially many) of these half-spaces arenecessary, those indexed by vertices of the marginal polytope. In that case, the constraint set is polyhedral.
5 The prediction-based variant of 1-best MIRA (Crammer et al., 2006) is also a particular case, where one also
picks a single half-space, the one indexed by the prediction under the current model wt , rather than the mode of
LSSVM (wt , x t , yt ).
6 The experiments in named entity recognition were made by Kevin Gimpel, co-author of Martins et al. (2010c),
who also generated the corresponding plots.

CHAPTER 8. LEARNING WITH DUAL COORDINATE ASCENT
86

91
90

Test Set F1

Development Set F1

182

89

DCA, C = 10
DCA, C = 1
DCA, C = 0.1
SGD, η = 1, C = 10
SGD, η = 0.1, C = 1
SGD, η = 0.1, C = 0.1
L-BFGS, C = 1

85

84

88
87

0

10

20

30

No. Iterations

40

50

83

0

10

20

30

40

50

No. Iterations

Figure 8.1: Named entity recognition. Learning curves for DCA (Alg. 12), SGD, and LBFGS. The SGD curve for C = 10 is lower than the others because dev-set validation chosea suboptimal value of η. DCA, by contrast, does not require choosing any hyperparameters
other than C. L-BFGS ultimately converges after 121 iterations to an F1 of 90.53 on thedevelopment data and 85.31 on the test data.
2002b) and simple gazetteer features; a feature was included iff it occurs at least once in the
training set (total 1,312,255 features). The task is evaluated using the F1 measure computedat the granularity of entire entities. We set β = 1 and γ = 0 (the CRF case). In addition to
SGD, we also compare with L-BFGS (Liu and Nocedal, 1989), a common choice for optimizing conditional log-likelihood. We used {10a , a = −3, . . . , 2} for the set of values consideredfor η in SGD. Figure 8.1 shows that DCA (which only requires tuning one hyperparameter)
reaches better-performing models than the baselines.
Dependency Parsing. We trained non-projective dependency parsers for three languages(Arabic, Danish, and English), using datasets from the CoNLL-X and CoNLL-2008 sharedtasks (Buchholz and Marsi, 2006; Surdeanu et al., 2008). Performance is assessed by the unlabeled attachment score (UAS), the fraction of non-punctuation words which were assigned
the correct parent. We adapted the turbo parsers described in Chapter 7 to handle any loss
function L β,γ , via Algorithm 12; for decoding, we used the sum-product loopy BP algorithmof Smith and Eisner (2008). We used the pruning strategy described in (Martins et al., 2009b)
and tried two feature configurations: one arc-factored model, for which decoding is exact,
and another model with second-order features (siblings and grandparents) for which it isapproximate.
The comparison with SGD for the CRF case is shown in Figure 8.2. For the arc-factoredmodels, the learning curve of DCA seems to lead faster to an accurate model. Notice thatthe plots do not account for the fact that SGD requires four extra iterations to choose thelearning rate. For the second-order models of Danish and English, however, DCA did not
perform as well.7
Finally, Table 8.1 shows results obtained for different settings of β and γ.8 Interestingly,
we observe that the higher scores are obtained for loss functions that are “between” SVMs
7 Further analysis showed that for ∼15% of the training instances, loopy BP led to very poor variational
approximations of log Z (w, x ), yielding estimates Pwt (yt | x t ) > 1, thus a negative learning rate (see (8.23)),that we truncate to zero. Thus, no update occurs for those instances, explaining the slower convergence. A
possible way to fix this problem is to use techniques that guarantee upper bounds on the log-partition function(Wainwright and Jordan, 2008).
8 Observe that there are only two degrees of freedom: indeed, ( λ, β, γ ) and ( λ0 , β0 , γ0 ) lead to equivalentlearning problems if λ0 = λ/a, β0 = β/a and γ0 = aγ for any a > 0, with the solutions related via w0 = aw.

8.6. CONCLUSIONS AND FUTURE WORK

183

Figure 8.2: Learning curves for DCA (Alg. 12) and SGD, the latter with the learning rate
η = 0.01 chosen from {0.001, 0.01, 0.1, 1} using the same procedure as before. The instability
when training the second-order models might be due to the fact that inference there isapproximate.
βγ

10 (CRF)

1
1

1
3

1
5

3
1

5
1

∞
1 (SVM)

NER

best C
F1 (%)

1.0
85.48

10.0
85.54

1.0
85.65

1.0
85.72

1.0
85.55

1.0
85.48

1.0
85.41

dependency
parsing

best CUAS (%)

0.1
90.76

0.01
90.95

0.01
91.04

0.01
91.01

0.01
90.94

0.01
90.91

0.1
90.75

Table 8.1: Varying β and γ: neither the CRF nor the SVM are optimal. We report only the
results for the best C, chosen from {0.001, 0.01, 0.1, 1} with dev-set validation. For namedentity recognition, we show test set F1 after K = 50 iterations (empty cells will be filled in in
the final version). Dependency parsing experiments used the arc-factored model on English
and K = 10.
and CRFs.

8.6 Conclusions and Future Work
We presented a general framework for aggressive online learning of structured classifiers by
optimizing any loss function in a wide family. The technique does not require a learning
rate to be specified. We derived an efficient technique for evaluating the loss function andits gradient, which invokes our results in Chapter 5 regarding the variational representationof log-partition functions. The result is an algorithmic framework that puts in the sameground the training of SVMs, CRFs, and the use of loss functions in-between, shedding
some light on connections between these loss functions and learning algorithms such as
MIRA and the structured perceptron, which are widely used in NLP. Experiments in namedentity recognition and dependency parsing showed that the algorithm converges to accurate
models at least as fast as stochastic gradient descent.
There are some open problems and paths for future research in the context of the work
presented here.

184

CHAPTER 8. LEARNING WITH DUAL COORDINATE ASCENTFirst, note that the primal-dual characterization of our algorithms pretends that they only
make one pass over the data; basically, we let the number of dual variables grow with thenumber of epochs. While this makes sense in the context of online learning, it is not what
is usually regarded as dual coordinate ascent (with a fixed number of coordinates). It is
straightforward, however, to change our algorithms slightly so that the number of dual variables is N (the dataset size), independent of the number of passes. This, however, requiresthat one stores, for each instance n, the last solution µbn obtained for that instance; then, we
need to correct wt , before each parameter update, based on that solution. An evaluation of
the modified algorithm will be subject of future work.
Second, the family of loss functions that we have presented leaves open the choice of thehyperparameters β and γ. An automatic choice of these coefficients based on the training
data would be highly desirable, and we defer this to future work.
Third, the impact of approximate inference in the learning problem deserves further
study. A step forward was given by Kulesza and Pereira (2007), who have shown cases
where learning with approximate inference fails. Before that, Wainwright (2006) suggested
that it may be beneficial to consider the same approximation (e.g., a tree-reweighted entropy
approximation) at training and test times. However, our preliminary experiments in that
direction were not very successful. Later, Finley and Joachims (2008) provided an empirical study of learning with LP-relaxations. In Martins et al. (2009c), we provided conditions
that guarantee algorithmic separability under LP-relaxed inference. The conditions are that
separability in the exact setting must hold with a large enough margin, which depends ona coefficient expressing the tightness of the local polytope approximation. Those conditions,however, are too stringent, and it is possible to find counter-examples in fully connected
pairwise MRFs for which they will never hold.9 We point out, however, that for our parsing
problems, the local polytope approximation is much tighter than in fully connected pairwise
MRFs. Apart from these geometric characterizations, it appears that the theory of approximation algorithms (Vazirani, 2001) could be useful in this context. We intend to pursue thisline of research in the future.

9 David Sontag, personal communication.

Chapter 9

Online Learning of Structured
Predictors with Multiple Kernels
Training structured predictors often requires considerable time and effort from the practicioner to get the model right. Many successful approaches rely on clever feature engineering,or (in the scope of kernel-based machine learning) on the ability of designing a kernel function that suits the task at hand. In this chapter, we propose a new approach for learning
structured prediction models that are able to learn the kernel function automatically fromthe data. The following novel contributions are presented:• We adapt the framework of multiple kernel learning (MKL, Lanckriet et al. 2004) to structured prediction.
• We propose a new family of online algorithms that can tackle many variants of MKL
and group-Lasso. These algorithms can also deal with overlapping groups of features,provided a co-shrinkage property that we introduce is satisfied.
• We show regret, convergence, and generalization bounds for those algorithms, mixing
results from online convex programming (Shalev-Shwartz, 2011) and the theory of proximity operators (Bauschke and Combettes, 2011).
• We propose ways of circumventing one of the drawbacks of kernel-based training algorithms in large-scale tasks, its quadratic dependency on the training data. Througha small set of preliminary experiments on a handwriting recognition task, we show
how accurate models can be obtained rapidly by combining a sparse kernel with afeature-based model.
Most of the results presented in this chapter—with the exception of the co-shrinkage
property and a set of toy experiments on online binary classification—were originally introduced in Martins et al. (2010b,e) and further developed in Martins et al. (2011b).
Some of the key results presented here will be later reused in Chapter 10, where we
discuss structured sparsity and present a wide set of experiments in structured prediction
tasks.
185

186

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

9.1 Motivation and Related Work
Throughtout this thesis, we have dealt with models and algorithms for structured prediction (Lafferty et al., 2001; Taskar et al., 2003; Tsochantaridis et al., 2004). These models havebeen shown to excel in many tasks, ranging from natural language processing to computer
vision, robotics, and computational biology. The untold story is that a substantial effort isoften necessary to achieve success in those tasks. Obtaining (structured) predictors withgood generalization capability typically involves two steps: (i) designing features or similarity measures (i.e., kernels), (ii) minimizing a regularized empirical loss function. In practice,
one typically spends a considerable amount of time iterating over these two steps, as they arestrongly interdependent. This loop is particularly expensive when training structured predictors in large scale settings, where achieving state-of-the-art performance usually involves
extensive feature/kernel engineering.
Kernel engineering can be partially automated by using the kernel learning approachpioneered by Lanckriet et al. (2004) and Bach et al. (2004), where a combination of multiple
kernels is learned from the data. While multi-class and scalable multiple kernel learning(MKL) algorithms have been proposed (Sonnenburg et al., 2006; Zien and Ong, 2007; Rakotomamonjy et al., 2008; Chapelle and Rakotomamonjy, 2008; Xu et al., 2009; Kloft et al.,2010), none are well suited for large-scale structured prediction, for the following reason:
they all involve an inner loop in which a standard learning problem (e.g., an SVM) needs tobe repeatedly solved; in large-scale structured prediction, as we discussed in Section 3.5, thisstandard learning problem is too expensive to tackle in batch form, and online methods areusually preferred (Bottou, 1991b; Collins, 2002a; Ratliff et al., 2006; Vishwanathan et al., 2006;
Crammer et al., 2006; Collins et al., 2008). These online methods are fast in achieving low
generalization error, but converge slowly to the training objective, resulting unattractive for
repeated use in the inner loop of a MKL algorithm. Note that early stopping is inadequate,
since it can misguide the overall MKL optimization.
In this chapter, we propose a stand-alone online MKL algorithm that iterates between
subgradient and proximal steps, which overcomes the above difficulty. Our theoretical analysis and experiments show that the proposed algorithm:• is simple, flexible, and compatible with sparse and non-sparse MKL;
• can handle composite regularizers, even if they overlap;• is suitable for large-scale structured prediction;• can learn time-varying concepts;• is equipped with regret, convergence, and generalization guarantees.
When applied to structured prediction, the proposed algorithm is termed SPOM (Structured Prediction by Online MKL). Our approach extends and kernelizes the forward-backward
splitting scheme Fobos (Duchi and Singer, 2009), whose regret bound we improve. In
passing, we show how to efficiently compute the proximal projections associated with thesquared L1 -norm, despite the fact that the underlying optimization problem is not separable. Finally, we also consider composite regularizers, possibly overlapping, which haverecently been the subject of intense research (Bach, 2008b; Zhao et al., 2009; Jenatton et al.,9.2. MULTIPLE KERNEL LEARNING AND GROUP SPARSITY

187

2009; Liu and Ye, 2010a,b; Mairal et al., 2010). We show that a simple approach where theproximal steps are applied sequentially still converges if the regularizers satisfy the so-called“co-shrinkage” condition.
This chapter is organized as follows: after reviewing structured prediction and MKL(Section 9.2), we present a new class of online proximal algorithms which handle composite
regularizers with multiple proximal steps (Section 9.3). We derive convergence rates andshow how these algorithms are applicable to MKL, group-Lasso, and other structured sparsity formalisms.1 In Section 9.4, we apply this algorithm to structured prediction (yieldingSPOM), in a handwritten text recognition problem. We obtain encouraging results in termsof runtime and accuracy.

9.2 Multiple Kernel Learning and Group Sparsity
9.2.1

Inference and Learning with Kernels

We assume the setting described in the background chapters, where we have an input set X
and an output set Y. Given an input x ∈ X, let Y( x ) ⊆ Y be its set of admissible outputs;
in structured prediction, this set is assumed to be structured and exponentially large in the“size” of x. We let U := {( x, y) | x ∈ X, y ∈ Y( x )} denote the set of admissible inputoutput pairs. Given labeled data D := (( x1 , y1 ), . . . , ( x N , y N )) ∈ U N , the goal is to learn acompatibility function gw : U → R that allows to make predictions via
x 7→ ŷ( x ) := arg max gw ( x, y).
y ∈Y( x )(9.1)

Section 3.2 described linear functions, gw ( x, y) := w · f ( x, y), where w is a parameter vector
and f ( x, y) a feature vector; here, we still resort to the linear case, but assume more broadlythat these vectors live in a reproducing kernel Hilbert space (RKHS) H, with corresponding
kernel K : U × U → R. We denoteby h., .iH the inner product in H, and by k.kH thepcorresponding norm, kwkH := hw, wi. The kernel evaluation corresponds to the innerproduct of the feature vectors in H:2
K (( x, y), ( x 0 , y0 )) = h f ( x, y), f ( x 0 , y0 )iH .

(9.2)

For convenience, we often drop the subscript H when the underlying Hilbert space is clear
from the context. In the sequel, features will be used implicitly or explicitly, as convenience
determines.
As before, we formulate the learning problem as the minimization of a regularized empirical risk functional,
gbw := arg min Ω( gw ) +
gw ∈H

1 N
L ( gw ; x i , y i ),
N i∑
=1

(9.3)

1 We will elaborate more on their application to structured sparsity in Chapter 10, where we also present

additional experiments in sequence labeling and dependency parsing tasks.
2 We omit details on kernels and reproducing kernel Hilbert spaces. There is an extense literature about
kernel-based methods for machine learning. We recommend Schölkopf and Smola (2002) for a thorough description of this subject.

188

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

where Ω : H → R+ is a regularizer, and L : H × X × Y → R is a convex loss function.
We have described several loss functions in Chapter 3, such as the structured logistic lossof CRFs (Section 3.4.2) and the structured hinge loss of structured SVMs (Section 3.4.3). In
kernel-based learning, these losses are written as:
LCRF ( gw ; x, y) := log

exp gw ( x, y0 ) − gw ( x, y) ,
∑
0


(9.4)

y ∈Y( x )LSSVM ( gw ; x, y) :=max

y 0 ∈Y( x )gw ( x, y0 ) − gw ( x, y) + ρ(y0 , y) .

(9.5)

With L2 -regularization, Ω( gw ) ≡ ΩλL2 ( gw ) = λ2 kwk2 , the solution of Eq. 9.3 can be expressed
as a kernel expansion, according to the structured version of the representer theorem (Hofmann et al., 2008, Corollary 13). We next discuss alternative forms of regularization that take
into consideration another level of structure—now in the feature space.

9.2.2

Block-Norm Regularization and Kernel Learning

Selecting relevant features or picking a good kernel are ubiquitous problems in statisticallearning, both of which have been addressed with sparsity-promoting regularizers. We firstillustrate this point for the explicit feature case. Often, features exhibit a natural “block”
structure: for example, many models in NLP consider feature templates—these are binaryfeatures indexed by each word w in the vocabulary, by each part-of-speech tag t, by eachpair (w, t), etc. Each of these templates correspond to a block (also called a group) in thefeature space H. Thus, H is endowed with a block structure, where each block (indexed by
m = 1, . . . , M) is itself a “smaller” feature space Hm ; formally, H is a direct sum of HilbertLM
spaces: H = m=1 Hm .

Group-Lasso. Consider the goal of learning a model in the presence of many irrelevant
feature templates. A well-known criterion is the group-Lasso (Bakin, 1999; Yuan and Lin,M2006), which uses the following block-norm regularizer: ΩGL
λ ( f w ) = λ ∑m=1 k wm k. This canbe seen as the L1 -norm of the vector containing the L2 -norms of the groups. When ΩGLis used in (9.3), it promotes group sparsity, i.e., solutions in which only a few groups arenonzero; within the mth group, either the optimal w∗m is identically zero—in which case theentire group is discarded—or it is non-sparse. We will come back to group-Lasso regularizers
in Chapter 10.

Sparse MKL. In MKL (Lanckriet et al. 2004), the kernel function is learned as a convex
Mcombination of M prespecified base kernel functions {K1 , . . . , K M }, i.e., K = ∑m
=1 β m K m ,where the coefficients β = ( β 1 , . . . , β M ) are constrained to belonging to the simplex ∆ M :={ β ≥ 0 | k βk1 = 1}. This corresponds precisely to the direct sum of RKHS described above,LM
where the RKHS H induced by K is written as H = m=1 Hm , each Hm being the RKHS
induced by Km . Denoting by h·, ·iH and h·, ·iHm the inner products associated with K and
Km , respectively, the compatibility function associated with K is gw ( x, y) = hw, f ( x, y)iH =MM∑m=1 β m h wm , f m ( x, y )iHm = ∑m=1 β m gwm ( x,y) . The kernel learning problem is then formu-

9.2. MULTIPLE KERNEL LEARNING AND GROUP SPARSITY

189

lated as an outer minimization of (9.3) w.r.t. β ∈ ∆ M :
min

β∈∆ M

min

gw1 ∈H1 ,...,gw M ∈H M

λ M
1 N
2
L
β
k
w
k
+
m
m
2 m∑
N i∑
=1
=1

M

∑ β m gw ; x i , y i
m

!
.

(9.6)

m =1

Remarkably, as shown by Bach et al. (2004) and Rakotomamonjy et al. (2008), a simple changeof variable allows transforming this joint optimization over β and w into a single optimizaM
2tion of the form (9.3) with a block-structured regularizer ΩMKL ( gw ) = 12 (∑m=1 k wm k) . Notethat this coincides with the square of the group-Lasso regularizer; in fact, the two problems
are equivalent up to a change of λ (Bach, 2008a). Hence, this MKL formulation promotes
sparsity in the number of selected kernels (i.e., only a few nonzero entries in β).
Non-Sparse MKL. A more general MKL formulation (not necessarily sparse) was recently
proposed by Kloft et al. (2010). Define, for p ≥ 1, the set ∆ M
p : = { β ≥ 0 | k β k p = 1}. Then,M
by modifying the constraint in (9.6) to β ∈ ∆ p , the resulting problem can also be written inform (9.3), with the following block-structured regularizer:
1ΩMKL,q ( gw ) =2

 M∑ kwm k

q

2/q
:=

m =1

1
kwk22,q ,2(9.7)

where q = 2p/( p + 1). The function k.k2,q satisfies the axioms of a norm, and is called
the L2,q mixed norm. Given a solution w∗ , the optimal kernel coefficients can be computed
as β∗m ∝ kw∗m k2−q . Note that the case p = q = 1 corresponds to sparse MKL and thatp = ∞, q = 2 corresponds to standard L2 -regularization with a sum-of-kernels.

9.2.3

Learning the Kernel in Structured Prediction

Until Martins et al. (2010e), all proposed formulations of MKL were applied to classification problems with small numbers of classes. The algorithmic obstacles that prevented thestraightforward application of those formulations to structured prediction will be discussed
in Section 9.2.4; here, we formally present our MKL formulation for structured prediction.
As we saw in Chapter 3 of this thesis, in structured prediction, model factorization assumptions are needed to make the inference problem (3.5) tractable. This can be accomplished by defining a set of parts over which the model decomposes (Taskar et al., 2003).
Suppose, for example, that outputs correspond to vertex labelings in a pairwise Markov network (V, E). Then, we may let each part be either a vertex or an edge, and write the feature
vector as f ( x, y) = ( f V ( x, y), f E ( x, y)), withf V ( x, y) =f E ( x, y) =∑ ψV (x, i) ⊗ ζ V (yi ),(9.8)∑ ψE (x, ij) ⊗ ζ E (yi , y j ),

(9.9)

i ∈V

ij∈E

where ⊗ denotes the Kronecker product, ψV and ψE are input feature vectors, and ζ V andζ E are local output feature vectors which are functions, respectively, of a single vertex and asingle edge. Notice that this decomposition into parts does not concern the input: both ψVand ψE may depend on the entire x. Using the decomposition in (9.8)–(9.9), the compatibility

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

190

function defined in (3.5) takes the formf w ( x, y) = hw, f ( x, y)i=∑ hwV , ψV (x, i) ⊗ ζ V (yi )i + ∑ hwE , ψE (x, ij) ⊗ ζ E (yi , y j )i

i ∈V

=

ij∈E

∑ θi (yi ; x, wV ) + ∑ θij (yi , y j ; x, wE ),

i ∈V

ij∈E

where θi (yi ; x, wV ) and θij (yi , y j ; x, wE ) are local scores. The decoding problem (3.5) becomes
equivalent to that of finding the MAP configuration in a pairwise Markov random field,
which can be done using the methods discussed in Chapter 4.
In terms of kernels, the decomposition in (9.8–9.9) can be written as
KV (( x, i ), ( x 0 , i0 )) LV (yi , yi00 )∑0

K (u, u0 ) = K (( x, y), ( x 0 , y0 )) =i,i ∈V+

KE (( x, ij), ( x 0 , i0 j0 )) LE ((yi , y j ), (yi00 , y0j0 )),∑0 0ij,i j ∈E

where
KV (( x, i ), ( x 0 , i0 )) := hψV ( x, i ), ψV ( x 0 , i0 )i,
LV (yi , yi00 ) := hζ V (yi ), ζ V (yi00 )i,
KE (( x, ij), ( x 0 , i0 j0 )) := hψE ( x, ij), ψE ( x 0 , i0 j0 )i,
LE ((yi , y j ), (yi00 , y0j0 )) := hζ E (yi , y j ), ζ E (yi00 , y0j0 )i.

(9.10)

A common simplifying option consists in letting ζ V and ζ E be identity feature mappings(with the former scaled by β 0 > 0) and ψE ≡ 1. We can then learn the kernel KV as a
M , yielding a kernel decomposition of the form
combination of base kernels {KV,m }m=1
K (( x, y), ( x 0 , y0 )) = β 0 · |{ij, i0 j0 ∈ E : yi = yi00 , y j = y0j0 }|+∑i,i0 ∈V: yi =yi00

M∑ βm KV,m ((x, i), (x0 , i0 )).

m =1

In our sequence labeling experiments (Section 9.4), vertices and edges correspond to unigrams and bigrams of labels. We explore two strategies: learning β 1 , . . . , β M , with β 0 = 1
fixed, or also learning β 0 .

9.2.4

Existing MKL Algorithms

Early approaches to MKL (Lanckriet et al., 2004; Bach et al., 2004) considered the dual of (9.6)
in the form of a quadratically constrained quadratic program or of a second order cone program, and thus were limited to small/medium scale problems. Subsequent work focused on
scalability: Sonnenburg et al. (2006) proposed a semi-infinite linear programming formulation and a cutting plane algorithm; Rakotomamonjy et al. (2008) proposed a gradient-based
method (SimpleMKL) for optimizing the kernel coefficients β; Chapelle and Rakotomamonjy(2008) proposed to use Newton steps; Kloft et al. (2010) adopted a Gauss-Seidel scheme, al-9.3. ONLINE PROXIMAL ALGORITHMS

191

ternating between optimization w.r.t. β and the SVM instances; Xu et al. (2009) proposed an
extended level method.
The methods mentioned in the previous paragraph are all wrapper-based algorithms: they
repeatedly solve problems of the form (9.3) (or smaller chunks of it, as in Kloft et al. 2010)
in an inner loop, while adjusting the kernel coefficients β in an outer loop. Although warmstarting may offer considerable speed-ups, convergence relies on the exactness (or prescribedaccuracy in the dual) of these solutions, which constitutes a serious obstacle when using
such algorithms for structured prediction. As seen in Section 3.5, large-scale solvers forstructured SVMs lack strong convergence guarantees; the best methods require O( 1e ) rounds
to converge to e-accuracy. Sophisticated second-order methods are intractable, since thekernel matrix is exponentially large and hard to invert; furthermore, there are typicallymany support vectors, since they are indexed by elements of Y( xi ).
In contrast, we tackle (9.6) in primal form. Rather than repeatedly calling off-the-shelfsolvers for (9.3), we propose a stand-alone online algorithm with runtime comparable to thatof solving a single instance of (9.3) by fast online methods. This paradigm shift paves theway for extending MKL to structured prediction, a vast unexplored territory.

9.3 Online Proximal Algorithms
We frame our online MKL algorithm in a wider class of online proximal algorithms. The
theory of proximity operators (Moreau, 1962), which is widely known in optimization andhas recently gained prominence in the signal processing community (Combettes and Wajs,
2006; Wright et al., 2009), provides tools for analyzing these algorithms and generalizes
many known results, sometimes with remarkable simplicity. We thus start by summarizing
its important concepts in Section 9.3.1.

9.3.1

Proximity Operators and Moreau Projections

We have presented a few basic concepts of convex analysis in Appendix B; we now complement them with some important results concerning direct sums of Hilbert spaces.
Let H1 ⊕ · · · ⊕ H M be the direct sum of M Hilbert spaces. Then, if ϕ : H1 ⊕ · · · ⊕ H M →R̄ is block-separable, i.e., ϕ( x) = ∑kM=1 ϕk ( xk ), where xk ∈ Hk , it is a known fact that its
proximity operator inherits the same block-separability: [prox ϕ ( x)]k = prox ϕk ( xk ) (Wrightet al., 2009). For example, the proximity operator of the mixed L2,1 -norm, which is blockseparable, has this form. The following proposition extends this result by showing how tocompute proximity operators of functions (maybe not separable) that only depend on theL2 -norms of their blocks; e.g., the proximity operator of the squared L2,1 -norm reduces to thatof squared L1 .
Proposition 9.1 Let ϕ : H1 ⊕ · · · ⊕ H M → R̄ be of the formϕ( x1 , . . . , x M ) = ψ(k x1 kH1 , . . . , k x M kH M )(9.11)

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

192
for some ψ : R M → R̄. Then,hM ϕ ( x1 , . . . , x M ) = Mψ (k x1 kH1 , . . . , k x M kH M ),i
= [proxψ (k x1 kH1 , . . . , k x M kH M )]k ( xk /k xk kHk ).
prox ϕ ( x1 , . . . , x M )k(9.12)(9.13)

Proof. See Appendix F.1.
Finally, we recall the Moreau decomposition, relating the proximity operators of Fenchel
conjugate functions (Combettes and Wajs, 2006) and present a corollary, proved in Appendix F.2, that is the key to our regret bound in Section 9.3.4.
Proposition 9.2 (Moreau (1962)) Let ϕ : H → R̄ be a convex, lsc, and proper function. Then,x = prox ϕ ( x) + prox ϕ? ( x),k x k2 = 2 M ϕ ( x ) + M ϕ? ( x ) .

(9.14)(9.15)

Corollary 9.3 Let ϕ : H → R̄ be as in Proposition 9.2, and x̄ := prox ϕ ( x). Then, any y ∈ Hsatisfiesky − x̄k2 ky − xk2 k x − x̄k2−+
≤ ϕ(y) − ϕ( x̄).
2
2
2

(9.16)

Although the Fenchel dual ϕ? does not appear in (9.16), it plays a crucial role in proving
Corollary 9.3. Later (in Proposition 9.5), we provide a generalization of Corollary 9.3 inwhich x̄ is the outcome of sequential proximal steps (rather than a single proximal step). This
will be used constructively in Algorithm 14, to be presented next.

9.3.2

An Online Proximal Gradient Scheme

The general algorithmic scheme that we propose and analyze in this paper is presented as
Algorithm 14. It deals (in an online fashion3 ) with problems of the formmin Ω(w) +

w ∈W

1 N
L(w; xi , yi ),N i∑=1(9.17)

where W ⊆ H is a convex set and the regularizer Ω has a composite form Ω(w) =J∑ j=1 Ω j (w). This encompasses all formulations described in Section 9.2.1–9.2.2: standard
L2 -regularized SVMs and CRFs, group-Lasso, and sparse and non-sparse variants of MKL.
For all of these, we have W = H and J = 1. Moreover, using J > 1 allows new variants ofblock-norm regularization, as discussed in Section 9.3.3. Our only assumption is that eachproxΩ j is “simple” (i.e., has closed form or can be cheaply computed), whereas proxΩ may
not be.
Algorithm 14 shares with stochastic gradient descent (SGD, Bottou 1991b) the fact that it
performs “noisy” (sub)gradient steps based on a single instance ( xn , yn ) (lines 5–6), whichmakes it specially suitable for problems with large N. However, unlike SGD, each round of
3 For simplicity, we focus on the pure online setting, i.e., each parameter update uses a single observation;

analogous algorithms may be derived for the batch and mini-batch cases.

9.3. ONLINE PROXIMAL ALGORITHMS

193

Algorithm 14 Online Proximal Algorithm
1: input: data D, number of rounds T, learning rate sequence (ηt )t=1,...,T
2: initialize w1 = 0; set N = |D|
3: for t = 1 to T do
4:choose n = n(t) ∈ {1, . . . , N } and take training pair ( x n , yn )5:obtain a subgradient g ∈ ∂L(wt ; x n , yn )6:w̃t = wt − ηt g (subgradient step)
7:
for j = 1 to J dojj −1w̃t+ J = proxηt Ω j (w̃t+ J ) (proximal step)
9:
end for10:
wt+1 = projW (w̃t+1 ) (projection step)
11: end forT
12: output: the last model w T +1 or the averaged model w̄ = T1 ∑t=1 wt
8:Algorithm 14 computes subgradients only w.r.t. the loss function L, followed by J proximal
steps, one per each term Ω j (line 7). For sparsity-promoting regularizers, such as Ω(w) =λkwk1 or Ω(w) = λkwk2,1 , SGD and Algorithm 14 behave very differently. Due to the factthat those regularizers are non-differentiable at the origin, SGD is very unlikely to obtaina sparse solution, since the components that should be zero only oscillate asymptoticallytowards zero. In contrast, since proxΩ has a thresholding effect, Algorithm 14 is likely toreturn sparse solutions after a finite number of steps.
As in the Pegasos algorithm (Shalev-Shwartz et al., 2007), the projection step (line 10) is
used to accelerate convergence. Even when the original learning problem is unconstrained(i.e., (9.17), with W = H), if it is known beforehand that the optimum lies in a convex setΞ ⊆ H, then solving (9.17) with W = Ξ ensures that each iterate wt is confined to a boundedregion containing the optimum.
Before analyzing Algorithm 14, we remark that it includes, as particular cases, manywell-known online learners:• if Ω = 0 and ηt ∝ √1t , Algorithm 14 is the online projected subgradient algorithm ofZinkevich (2003);
1• if Ω = 0, L = LSSVM + λ2 kwk2 , and ηt = λt
, Algorithm 14 becomes Pegasos, a popularalgorithm for learning SVMs that has been extended to structured prediction (ShalevShwartz et al., 2007, 2010);• If Ω(w) = kwk1 , Algorithm 14 is equivalent to the truncated gradient descent methodof Langford et al. (2009);• If J = 1, Algorithm 14 coincides with Fobos (Duchi and Singer, 2009), which was used
for learning SVMs and also for group-Lasso (but not for structured prediction).
In Section 9.3.5, we show how to kernelize Algorithm 14 and apply it to sparse MKL. The
case J > 1 has applications in variants of MKL or group-Lasso with composite regularizers(Tomioka and Suzuki, 2010; Friedman et al., 2010; Bach, 2008b; Zhao et al., 2009). For thosecases, Algorithm 14 is seamlessly more suitable than Fobos since it sidesteps the (often
difficult) evaluation of proxΩ , requiring only sequential evaluations of proxΩ j for j = 1, . . . , J.

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

194

Algorithm 15 Proximity Operator of L21
1: input: vector x ∈ R M and parameter λ > 0
2: sort the entries of | x | into y (yielding y1 ≥ . . . ≥ y M )3: set ρ = maxnjj ∈ {1, ..., M} | y j − 1+λjλ ∑r=1 yr > 0oρ

4: output: z = soft(x, τ ), where τ = 1+λρλ ∑r =1 yr

9.3.3

Proximity Operators of Block-Norm Regularizers

For Algorithm 14 to handle the MKL and group-Lasso problems (described in Section 9.2.2),
it needs to compute the proximal steps for block-norm regularizers. Proposition 9.1 abover -proximity
is crucial for this purpose; as a particular instance of (9.13), computing any L2,qr
operator can be reduced to computing an Lq one, the result of which is used to scale eachblock independently. The following are some examples of block-norm regularizers.
Group-Lasso. This corresponds to q = r = 1, so we are left with the problem of computing
the L1 -proximity operator, which has a well-known closed form solution: the soft-threshold
function, as presented in Section 9.3.1.
Sparse MKL. This corresponds to q = 1, r = 2, and there are two options: one is totransform the problem back into group-Lasso, by removing the square from ΩMKL ; as shownby Bach (2008a) and pointed out in Section 9.2, these two problems are equivalent in the sensethat they have the same regularization path. The other option (that we adopt) is to tackleΩMKL directly, which makes possible the comparison with other MKL algorithms, for thesame values of λ, as reported in Section 9.4. Proposition 9.1 enables reducing the evaluationof a L22,1 -proximity operator to that of L21 . However, L21 is not separable (unlike L1 ), hence
the proximity operator cannot be evaluated coordinate-wise. This apparent difficulty has led
some authors (e.g., Suzuki and Tomioka 2009) to adopt the first option. However, despite thenon-separability of L21 , this proximal step can still be efficiently computed (with O( M log M)cost), using Algorithm 15, which only requires sorting the group weights.4 Correctness of
this algorithm is shown in Appendix F.8.
Non-Sparse MKL. For the case q ≥ 1 and r = 2, a direct evaluation of the proximity
operator of L22,q is more involved. It seems advantageous to transform this problem intoq
an equivalent one, which uses a separable L2,q regularizer instead (the two problems arealso equivalent up to a change in the regularization constant). The resulting proximal stepamounts to solving M scalar equations of the form b − b0 + τqbq−1 = 0, w.r.t. b, which is stillvalid for q ≥ 2 (unlike the method described by Kloft et al. 2010). This can be done very
efficiently using Newton’s method.
Other variants. Many other variants of MKL and group-Lasso can be handled by Algorithm 14, with J > 1. For example, the elastic net MKL (Tomioka and Suzuki, 2010) uses
a sum of two regularizers, σ2 k.k2 + 1−2 σ k.k22,1 . In hierarchical Lasso and group-Lasso with
4 A similar algorithm was proposed independently by Kowalski and Torrésani (2009) in a different context.

9.3. ONLINE PROXIMAL ALGORITHMS

195

overlaps (Bach, 2008b; Zhao et al., 2009; Jenatton et al., 2009), each feature may appear
in more than one group. Algorithm 14 handles these problems seamlessly by enabling a
proximal step for each group.5 Sparse group-Lasso (Friedman et al., 2010) simultaneously
promotes group-sparsity and sparsity within each group, using a regularizer of the form
σk.k2,1 + (1 − σ)k.k1 ; Algorithm 14 can handle this regularizer by using two proximal steps,
both involving simple soft-thresholding: one at the group level, and another within eachgroup.

9.3.4

Regret, Convergence, and Generalization Bounds

We next show that, for a convex loss L and under standard assumptions, Algorithm 14
converges up to e precision, with high confidence, in O(1/e2 ) iterations. If L or Ω aree (1/e), where O
e hides logarithmic terms. Ourstrongly convex, this bound is improved to O
proofs combine tools of online convex programming (Zinkevich, 2003; Hazan et al., 2007)
and classical results about proximity operators (Moreau, 1962). We start by introducing thefollowing important concept.
Definition 9.1 (co-shrinkage property) We say that a sequence of convex, lsc, and proper regularizers Ω1 , . . . , Ω J is co-shrinking if∀ j0 < j,Ω j0 (w) ≥ Ω j0 (proxΩ j (w)),

that is, the proximity operator of any Ω j does not increase the value of any previous Ω j0 . Moreover,Ω1 , . . . , Ω J is said to be strongly co-shrinking if, ∀σ1 , . . . , σJ > 0, the sequence σ1 Ω1 , . . . , σJ Ω J is
co-shrinking. Finally, we say that a family of regularizers R = {Ω j | j ∈ J} has the co-shrinkage
property (resp. strong co-shrinkage property) if any pair of elements in R is co-shrinking (resp.
strongly co-shrinking). This implies that any finite sequence in R is (strongly) co-shrinking.
We note that most regularizers arising in sparse and non-sparse modelling are strongly
co-shrinking, as the next proposition attests. The proof is presented in Appendix F.3.
M
Proposition 9.4 Let H = m=1 Hm be a block-structured Hilbert space. Let B ⊆ {1, . . . , M } be afamily of blocks, and, given w ∈ H, denote by wB := (wm )m∈B . The following set of regularizers
has the strong co-shrinkage property:

L

Rnorms =qw 7→ kwB k2,p

p, q ≥ 1, B ⊆ {1, . . . , M } .

(9.18)A consequence of Proposition 9.4 is that any sequence of regularizers Ω1 , . . . , Ω J , each ofqj
the form Ω j (w) = kwBj k2,p, with p j , q j ≥ 1 and B j ⊆ {1, . . . , M}, is strongly co-shrinking.
jNote that the subsets of blocks B1 , . . . , B J can overlap, i.e., we may have Bi ∩ B j 6= ∅ forsome i 6= j. An example where this arises is the overlapping group-Lasso regularizer, wheresuch overlaps occur and we have Hm = R, and p j = q j = 1 for all j.
5 Recently, a lot of effort has been placed on ways for computing the proximal step for regularizers withoverlapping groups (Liu and Ye, 2010a,b; Mairal et al., 2010). Algorithm 14 suggests an alternative approach:split the regularizer into several non-overlapping parts and apply sequential proximal steps. Although in generalproxΩ J ◦ . . . ◦ proxΩ1 6= proxΩ J ◦...◦Ω1 , Algorithm 14 is still applicable, as we will see in Section 9.3.4.

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

196

The next result (proved in Appendix F.4) extends Corollary 9.3 for compositions of proximity operators of co-shrinking functions, proxΩ J ◦ . . . ◦ proxΩ1 . It is crucial for proving thecorrectness of Algorithm 14 when composite regularizers are used.
Proposition 9.5 Let Ω1 , . . . , Ω J : H → R̄ be convex, lsc, and proper functions which are coshrinking. Denote Ω = ∑ jJ=1 Ω j , and let w̄ := proxΩ J ◦ . . . ◦ proxΩ1 (w). Then, any ξ ∈ Hsatisfieskξ − w̄k2 kξ − wk2 kw − w̄k2−+
≤ Ω(ξ ) − Ω(w̄).
22
2J(9.19)

Note that the only difference between the bounds in Corollary 9.3 and Proposition 9.5
is that the latter includes a J in the denominator of the last term of the left hand side.
This reflects the “approximation” that is being made by taking sequencial proximal steps
proxΩ J ◦ . . . ◦ proxΩ1 rather than a single proximal step proxΩ1 +...+Ω J .
The key to our regret, convergence, and generalization bounds for Algorithm 14 is thefollowing lemma (proved in Appendix F.5).
Lemma 9.6 Let L be convex and G-Lipschitz on W, and let Ω = ∑ jJ=1 Ω j satisfy the following
conditions: (i) each Ω j is convex, lsc, and proper; (ii) Ω1 , . . . , Ω J are strongly co-shrinking; (iii)Ω(w) ≥ Ω(ΠW (w)), i.e., projecting the argument onto W does not increase Ω. Then, for anywb ∈ W, at each round t of Algorithm 14,b ) + Ω(wb ) + et ,L ( w t ) + Ω ( w t +1 ) ≤ L ( w
where
et =

b − w t k2 − k w
b − w t +1 k 2
ηt 2 k w
G +
.
2
2ηt

(9.20)(9.21)

b − w t k2 .
If L is σ-strongly convex, this bound can be strengthened by replacing et with et0 = et − σ2 kw
The bound in Lemma 9.6 is tighter6 than the one derived by Duchi and Singer (2009)
for the special case of J = 1 (i.e., for Fobos). For the even more special case of J = 1 andΩ = k · k1 , the bound derived by Langford et al. (2009) matches the one in Lemma 9.6,
showing that our lemma extends their bound to a much wider class of problems. Finally,note that the conditions (i)–(iii) are not restrictive: they hold whenever the regularizers
belong to the wide family in Proposition 9.4.
Let w∗ be the best fixed hypothesis, i.e., the one obtained in batch mode from the sameobservations,T w∗ = arg min ∑ Ω(w) + L(w; x n(t) , yn(t) ) ,(9.22)w ∈Wt =1

and the cumulative regret of an online algorithm be defined as


T 
T 
RegT := ∑ Ω(wt ) + L(wt ; x n(t) , yn(t) ) − ∑ Ω(w∗ ) + L(w∗ ; x n(t) , yn(t) ) .
t =1(9.23)

t =1

We next characterize Algorithm 14 in terms of its cumulative regret.
6 Instead of the term ηt G2 in (9.21), their bound has 7 ηt G2 , as can be seen from their Eq. 9 with A = 0 and22

ηt = ηt + 1 .
2

9.3. ONLINE PROXIMAL ALGORITHMS

197

Proposition 9.7 (regret bounds) Let the conditions of Lemma 9.6 hold, Ω ≥ 0, and Ω(0) = 0.
Then:
1. Running Algorithm 14 with fixed learning rate η yields
RegT ≤

ηT 2 kw∗ k2
G +
.
2
2η

(9.24)√√
Setting7 η = kw∗ k/( G T ), yields a sublinear regret of kw∗ k G T.
2. If W is bounded with diameter F (i.e., ∀w, w0 ∈ W, kw − w0 k ≤ F) and the learning rate is√ηt = η0 / t, with arbitrary η0 > 0, then,
RegT ≤

√
F2
2+ G η0T.
2η0(9.25)√√
Setting η0 = F/( 2G ), yields RegT ≤ FG 2T.
3. If L is σ-strongly convex, and ηt = 1/(σt), then
RegT ≤ G2 (1 + log T )/(2σ).

(9.26)

Proof. See Appendix F.6.
Once an online-to-batch conversion is specified, regret bounds allow obtaining PAC
bounds on optimization and generalization errors. The following proposition can be proved
using the techniques of Cesa-Bianchi et al. (2004) and Shalev-Shwartz et al. (2007).
Proposition 9.8 (optimization and estimation error) If the assumptions of Proposition 9.7 hold√and ηt = η0 / t as in item 2 in Proposition 9.7, then the version of Algorithm 14 that returns
the averaged model solves (9.17) with e-accuracy8 in T = O(( F2 G2 + log(1/δ))/e2 ) iterations
with probability at least 1 − δ. If L is also σ-strongly convex and ηt = 1/(σt) as in item 3 ofe ( G2 /(σδe)).
Proposition 9.7, then, for the version of Algorithm 14 that returns w T +1 , we get T = O
The generalization bounds are of the same orders.
We now pause to examine some concrete cases. The requirement that the loss is GLipschitz holds for the hinge and logistic losses, where G = 2 maxu∈U k f (u)k (see Appendix F.7), as well as the family of loss functions presented in Chapter 8. These losses
are not strongly convex, and therefore Algorithm 14 has only O(1/e2 ) convergence. If thee (1/e) convergence isregularizer Ω is σ-strongly convex, a possible workaround to obtain Ott
to let L “absorb” that strong convexity by redefining L̃(w; x , y ) = L(w; x t , yt ) + σkwk2 /2.
Since neither the L2,1 -norm nor its square are strongly convex, we cannot use this trick for
the MKL case, but it does apply for non-sparse MKL (L22,q -norms are strongly convex forq > 1) and for elastic MKL. Still, the O(1/e2 ) rate for MKL is competitive with the best batchalgorithms that tackle the dual; e.g., the method of Xu et al. (2009) achieves e primal-dual
gap in O(1/e2 ) iterations.9 Some losses of interest (e.g., the squared loss, or the modified
7 Note that this requires knowing both k w∗ k and the number of rounds T in advance.
8 I.e., it returns a feasible solution whose objective value is less than e apart from the optimum.
9 On the other hand,batch proximal gradient methods for smooth losses can be accelerated to achieve√O(1/ e) convergence in the primal objective (Beck and Teboulle, 2009).

198

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

Algorithm 16 SPOM
1: input: data D, regularizarion constant λ, number of rounds T, radius γ, learning rate
sequence (ηt )tT=1
2: initialize w1 ← 0
3: for t = 1 to T do
4:choose n = n(t) ∈ {1, . . . , N } and take training pair ( x n , yn )5:compute scores for m = 1, . . . , M: f m ( x n , y) = hwtm , f m ( x n , y)iMnn6:decode: ybt ∈ arg maxy∈Y(x) ∑m=1 f m ( x , y ) + ρ ( y, y )7:
Gradient step for m = 1, . . . , M: w̃tm = wtm − ηt ( f m ( x n , ybn ) − f m ( x n , yn ))t = k w̃t k
8:compute weights for m = 1, . . . , M: b̃mmt
9:shrink weights bt = proxηt λk.k2 (b̃ ) with Algorithm 15
2,1

t / b̃t · w̃t
Proximal step for m = 1, . . . , M: w̃tm+1 = bm
m
m
t
+
1
t
+
1
11:
Projection step: w= w̃ · min{1, γ/kw̃t+1 k}
12: end forT +1 k for m = 1, . . . , M
13: compute β m ∝ k wm
14: return β, and the last model w T +1

10:loss L̃ above) are G-Lipschitz in any compact subset of H but not in H. However, if the
optimal solution is known to lie in some compact set W, we can run Algorithm 14 with the
projection step, making the analysis still applicable.

9.3.5

SPOM: Structured Prediction with Online MKL

The instantiation of Algorithm 14 for structured prediction and ΩMKL (w) = λ2 kwk22,1 yields
the SPOM algorithm (Algorithm 16). We consider L = LSSVM ; adapting to any generalized
linear model (e.g., L = LCRF ) is straightforward. As discussed in the last paragraph of
Section 9.3.4, the inclusion of an apparently vacuous projection may accelerate convergence.
Hence, an optional upper bound γ on kwk is accepted as input. Suitable values of γ for the
SVM and CRF cases are given in Appendix F.7.
In line 5, the scores of candidate outputs are computed blockwise; as described in Section 9.2.3, a factorization over parts is assumed and the scores are for partial output assignments. Line 6 gathers all these scores and performs decoding (loss-augmented inference for
the SVM case, or marginal inference for the CRF case). Line 10 is where the block structure
is taken into account, by applying a proximity operator which corresponds to a blockwise
shrinkage/thresolding, with some blocks possibly being set to zero.
Although Algorithm 16 is described with explicit features, it can be kernelized, as shown
next. Observe that the parameters of the mth block after round t can be written as wtm+1 =
t+1 ( f ( x n(s) , yn(s) ) − f ( x n(s) , ŷn(s) )), where∑ts=1 αms
m
m

t 
r
r
+1
αtms
= η s ∏ ( bm/b̃m
) min{1, γ/kw̃r+1 k}r =s(t / b̃t ) min{1, γ/ k w̃t+1 k}
η t ( bm
if s = tm=tt / b̃t ) min{1, γ/ k w̃t+1 k} if s < t.
αms (bmm
Therefore, the inner products in line 5 can be kernelized. The cost of this step is O(min{ N, t}),instead of the O(dm ) (where dm is the dimension of the mth block) for the explicit feature9.4. EXPERIMENTS

199

case. After the decoding step (line 6), the supporting pair ( x t , ybt ) is stored. Lines 9, 11 and
13 require the norm of each group, which can be manipulated using kernels: indeed, after
each gradient step (line 7), we have (denoting ut = ( x n(t) , yn(t) ) and ubt = ( x n(t) , ybn(t) ))kw̃tm k2 = kwtm k2 − 2ηt hwtm , f m (ut )i + ηt2 k f m (ubt ) − f m (ut )k2= kwtm k2 − 2ηt f m (ubt ) + ηt2 (Km (ut , ut ) + Km (ubt , ubt ) − 2Km (ut , ubt ));(9.27)and the proximal and projection steps merely scale these norms. When the algorithm termiT +1
nates, it returns the kernel coefficients β and the sequence (αmt).
One can also use explicit features in some blocks and implicit features in others. Blocks
with explicit features can have their scores computed directly by evaluating the inner product between the feature and the weight vector; this is advantageous when dm  N, whichhappens often when some kernels in the combination are linear. In case of sparse explicit
features, an implementation trick analogous to the one used by Shalev-Shwartz et al. (2007)(where each wm is represented by its norm and an unnormalized vector) can substantiallyreduce the amount of computation. In the case of implicit features with a sparse kernel
matrix, sparse storage of this matrix (which we adopt in our implementation) can also significantly speed up the algorithm, eliminating its dependency on N in line 5. Note that all
steps involving block-specific computation can be carried out in parallel using multi-coremachines, making Algorithm 16 capable of handling many kernels (large M).

9.4

Experiments

We evaluate SPOM (Algorithm 16) on a structured prediction task, handwriting recognition,
formulated as a sequence labeling problem (Section 9.4.1). We then illustrate the robustness of online MKL for handling a particular kind of concept drift in a binary classification
problem (Section 9.4.2).

9.4.1

Handwriting Recognition

We formulate the task of recognizing sequences of handwritten characters as a sequencelabeling problem, which clearly falls in the structured prediction category. We use the OCR
dataset of Taskar et al. (2003), which has a total of 6,877 words and 52,152 characters.10
Each character (the input) is a 16 × 8 binary image, with one of 26 labels (the charactersa to z, the output to predict). As Taskar et al. (2003), we address this sequence labelingproblem with a structural SVM; however, we use the SPOM algorithm to learn the kernel
from the data. We use an indicator basis function to represent the correlation between
consecutive outputs, i.e., we made the common simplification described in Section 9.2.3
where L E ((yi , yi+1 ), (yi00 , yi00 +1 )) = I(yi = yi00 ∧ yi+1 = yi00 +1 ) in (9.10).
MKL versus average kernel. Our first experiment (upper part of Table 9.1; solid lines
in Figure 9.1) compares linear, quadratic, and Gaussian kernels, either used individually,
combined via a simple average, or with MKL (via SPOM). The results show that MKL outperforms the others by ≥ 2%, and that learning the bigram weight β 0 (Section 9.2.3) did not
10 Available at www.cis.upenn.edu/

˜taskar/ocr.

200

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELSTable 9.1: Results for handwriting recognition. Averages over 10 runs (same folds as Taskar
et al. (2003), training on one and testing on the others). The linear and quadratic kernels
are normalized to unit diagonal. In all cases, 20 epochs were used, with η0 in (9.25) pickedfrom {0.01, 0.1, 1, 10} by selecting the one that most decreases the objective after 5 epochs. In
all cases, the regularization coefficient C = 1/(λN ) was chosen with 5-fold cross-validationfrom {0.1, 1, 10, 102 , 103 , 104 }.
KernelLinear (L)Quadratic (Q)Gaussian (G) (σ2 = 5)
Average ( L + Q + G )/3
MKL β 1 L + β 2 Q + β 3 G
MKL β 0 , β 1 L + β 2 Q + β 3 G
B1 -Spline (B1 )Average ( L + B1 )/2
MKL β 1 L + β 2 B1
MKL β 0 , β 1 L + β 2 B1Training Runtimes
6 sec.
116 sec.
123 sec.
118 sec.
279 sec.
282 sec.
8 sec.
15 sec.
15 sec.
16 sec.

Test Acc. (per char.)
71.8 ± 3.9%
85.5 ± 0.3%
84.1 ± 0.4%
84.3 ± 0.3%
87.5 ± 0.3%
87.5 ± 0.4%
75.4 ± 0.9%
83.0 ± 0.3%
85.2 ± 0.3%
85.2 ± 0.3%

make any difference. We note, however, that each epoch of SPOM (without parallelization)is more costly than the the single kernel algorithms. To provide a fair analysis, we show in
Figure 9.1 the test set accuracies as a function of the training runtime. We can see that, even
without parallelization, the MKL approach achieves an accurate model sooner (with a slightadvantage over the quadratic and Gaussian kernels).
Feature and kernel sparsity. The second experiment aims at showing SPOM’s ability to
exploit both feature and kernel sparsity. A disadvantage of the Gaussian and quadratic kernels
used above is that they yield dense kernel matrices, rendering a quadratic runtime withrespect to the sample size. However, we have seen in Section 9.3.5 that SPOM can have amuch faster runtime when some kernels are linear or when some kernel matrices are sparse.
To illustrate this, we learn a combination of the linear kernel above (explicit features) with ageneralized B1 -spline kernel, given by K ( x, x0 ) = max{0, 1 − k x − x0 k/h}, with h chosen sothat the kernel matrix has approximately ∼ 95% of zeros. The rationale is to combine thestrength of a simple feature-based kernel with that of one depending only on a few nearest
neighbors. The results (bottom part of Tab. 9.1) show that MKL outperforms by ∼ 10% theindividual kernels, and by more than 2% the averaged kernel. Perhaps more importantly,
the accuracy is not much worse than the best one obtained in the previous experiment, while
the runtime is much faster (15 versus 279 seconds). Figure 9.1 (dashed lines) is striking inshowing the ability of producing a reasonable model very fast.
SPOM versus wrapper-based methods. To assess the effectiveness of SPOM as a kernel
learning algorithm, we compare it with two wrapper-based MKL algorithms: a Gauss-Seidel
method alternating between optimizing the SVM and the kernel coefficients (see, e.g., Kloft
et al. 2010), and a gradient method (SimpleMKL, Rakotomamonjy et al. 2008)11 . In both
11 Code available in http://asi.insa-rouen.fr/enseignants/

˜arakotom/code/mklindex.html.

9.4. EXPERIMENTS

201

0.88

(48 epochs) (20 epochs)(133 epochs)(133 epochs)

0.86(47 epochs)

0.84
0.82

(45 epochs)(249 epochs)

Linear (L)Quadratic (Q)Gaussian (G)Average ((L+Q+G)/3))
MKL (β1L+β2Q+β3G)
B1−Spline (B1)

0.8

Average ((B1+L)/2)

0.78(750 epochs)

MKL (β1B1+β2L)

0.76
50

100

150

200

250

Figure 9.1: Test set accuracies of single kernel and multiple kernel methods as a function of
the training stopping times (in seconds).

Figure 9.2: Comparison between SPOM (Algorithm 16) and two wrapper based methods
in the OCR dataset, with C = 100. The wrapper-based methods run 20 epochs of pegasos
in their first SVM call; subsequent calls run 3 epochs with warm-starting. With only 20–30
passes over the data, SPOM approaches a region very close to the optimum; the wrapperbased methods need about 100 epochs.

cases, the SVMs were optimized with structured pegasos. Despite the fact that each SVM
is strongly convex and has O( 1e ) convergence, its combination with an outer loop becomestime-consuming, even if we warm-start each SVM optimization. This is worse when regularization is weak (small λ). In contrast, SPOM, with its overall O( e12 ) convergence, is stableand very fast to converge to a near-optimal region, as attested in Figure 9.2, suggesting its
usefulness in settings where each epoch is costly.

9.4.2

Online Binary Classification

In our last experiment, we show how the online nature of Algorithm 14 provides an adaptive
behaviour that is useful in handling a certain kind of drift, by combining the online learning paradigm with the ability of learning the kernel function. For the scenario we will next
describe, online MKL succeeds where standard online learners (i.e., with a fixed kernel) are
doomed to fail.
We revisit the “Christmas stars” synthetic problem posed in Sonnenburg et al. (2006): abinary classification task where two concentric star-like shapes are to be distinguished fromeach other. If the diameters of the two stars are different enough, a Gaussian kernel with
an appropriate width performs well on this task. The kernel learning problem consists of

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

202

Figure 9.3: The “expanding stars” synthetic dataset, with N = 5000 data points, evenly split
into two classes; shown are the first 1000, 2000, 3000, 4000, and 5000 points. The two classes
are star-shaped and concentric. The diameter of the outermost class increases linearly from4 to 14; the innermost class has a diameter 1.5 times smaller. The noise is Gaussian for bothclasses, with standard deviation of 10% of the corresponding star diameters.

MKL
Average
2σ2 = 0.01
2σ2 = 0.1

0.9
Accuracy

0.85
0.8

2σ2 = 10.75

2σ2 = 102σ2 = 100

0.7
0.65
1000

2000
3000
Number of Rounds

4000

5000

Figure 9.4: Performance of single kernel online learners and a multiple kernel online learneron the “expanding stars” dataset, for Gaussian kernels with different widths. Shown is thefraction of correct predictions up to each round. The regularization parameter C = 1/(λN )was set to 10.

9.5. RELATED WORK

203

selecting the appropriate width, which is addressed by Sonnenburg et al. (2006) using abatch MKL formulation where several Gaussian kernels of different widths are combined.
Here, we present a variant of this problem, in which data is not i.i.d. Instead, both stars
expand as time evolves: their diameters increase linearly with the round number t, while theratio is kept constant. Figure 9.3 shows the generated data points.
The reason why this problem is challenging is that there is no single Gaussian kernel(with a fixed width) that is suitable for distinguishing between the two stars: the “best”
width should increase with t. This is illustrated in Figure 9.4, which shows that there is no
single kernel which when plugged into the online algorithm yields good performance overthe entire dataset. In contrast, the online MKL algorithm commits fewer mistakes and is able
to keep improving its performance as the rounds proceed. This is because it is able to adapt
its kernel in addition to its decision boundary.

9.5 Related work
Discriminative learning of structured predictors has been an active area of research since theseminal works of Lafferty et al. (2001); Collins (2002a); Altun et al. (2003); Taskar et al. (2003);
Tsochantaridis et al. (2004).
Following the introduction of MKL by Lanckriet et al. (2004), a string of increasingly
efficient algorithms were proposed (Sonnenburg et al., 2006; Zien and Ong, 2007; Rakotomamonjy et al., 2008; Chapelle and Rakotomamonjy, 2008; Xu et al., 2009; Suzuki and Tomioka,
2009; Kloft et al., 2010), although none was applied to structured prediction. Group-Lassois due to Bakin (1999); Yuan and Lin (2006), after which many variants and algorithms appeared, all working in batch form: Bach (2008b); Zhao et al. (2009); Jenatton et al. (2009);
Friedman et al. (2010).
Independently from us, Jie et al. (2010) recently proposed an online algorithm for multiclass MKL (called OM-2), which differs from ours in that, rather than subgradient and proximal steps, online updates perform coordinate descent in the dual. Our algorithm is more
flexible: while OM-2 is limited to L22,q -regularization, with q > 1, and becomes slow whenq → 1, we efficiently handle the L22,1 case as well as arbitrary composite regularizers. Jie et al.
(2010) also have not addressed structured prediction.
Proximity operators are well known in convex analysis and optimization (Moreau, 1962;
Lions and Mercier, 1979) and have recently seen wide use in signal processing; see Combettes and Wajs (2006), Wright et al. (2009), and references therein. Specifically, the theory ofproximity operators (see Appendix F.1) underlies the proofs of our regret bounds (Proposition 9.7).

9.6 Conclusions and Future Work
We proposed a new method for multiple kernel learning of structured predictors. To accomplish this, we introduced a class of online proximal algorithms applicable to many variantsof MKL and group-Lasso. We provided a theoretical analysis of its convergence rate, in theform of regret, optimization, and generalization bounds.
Experiments on a structured prediction task (character sequence recognition) have shown

204

CHAPTER 9. ONLINE LEARNING WITH MULTIPLE KERNELS

that the algorithm achieves state-of-the-art performance. Moreover, we have illustrated experimentally how the ability to learn the kernel online offers a new paradigm for problems
in which the underlying geometry (induced by the similarities between objects) evolves overtime.
Our work may impact other problems. In structured prediction, the ability to promote
structured sparsity might be useful for learning simultaneously the structure and parametersof graphical models. The online proximal gradient algorithm that we proposed will be usedin the next chapter for structured sparsity applications.
In a different direction, the use of sparse kernels redeems one of the disadvantages ofkernel methods in large scale problems: the quadratic dependency on the dataset size. Previous approaches to obviate this, such as budgeted learning methods, look inadequate in
NLP problems with lexical features, since the feature space grows with the data, and throwing away support vectors implies throwing away features that can be useful. It would beinteresting to try combining sparse kernels with linear features in other problems, such as
named entity recognition or parsing. The sparse kernel could be used to recognize longmatches of input text. This line of research could bring together ideas from memory-basedlearning and structured linear models.

Chapter 10

Structured Sparsity for Structured
Prediction
While a lot of progress has been made in efficient training of linear models with various loss
functions, the problem of endowing learners with a mechanism for feature selection is still
unsolved. Common approaches employ ad hoc filtering or L1 -regularization; both ignore
the structure of the feature space, preventing practicioners from encoding structural priorknowledge. In this chapter, we fill this gap by adopting regularizers that promote structured
sparsity, along with efficient algorithms to handle them. The following novel contributions
are presented:• We apply group-Lasso regularization to structured prediction problems. We consider
several possible choices of groups of features, with or without overlaps.
• We propose using those regularizers for the problem of learning feature templates. We
show that this framework is able to model the structure of the feature space throughcoarse-to-fine regularization.
• For learning using those regularizers, we use the class of online proximal gradientalgorithms developed in Chapter 9, which under the perceptron loss become a newalgorithm we call sparseptron. We enhance the algorithms with delayed proximal steps
and a novel budget-driven shrinkage technique, in order to achieve memory and computational efficiency.
Experiments on three tasks (chunking, entity recognition, and dependency parsing) show
gains in performance, compactness, and model interpretability.
The results presented in this chapter were originally introduced in Martins et al. (2011d).
The online algorithm was introduced in Martins et al. (2011b), along with a thorough convergence analysis, which we have presented in Chapter 9 of this thesis.

10.1 Motivation and Previous WorkAs seen in previous chapters, models for structured outputs are in demand across natural language processing, with applications in information extraction, parsing, and machinetranslation. State-of-the-art models usually involve linear combinations of features and are
205

206

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

trained discriminatively; examples are conditional random fields (Lafferty et al., 2001), structured support vector machines (Altun et al., 2003; Taskar et al., 2003; Tsochantaridis et al.,2004), and the structured perceptron (Collins, 2002a); all of them were reviewed in Sections 3.4–3.5. In all these cases, the underlying optimization problems differ only in the
choice of loss function; choosing among them has usually a small impact on predictive performance.
In this chapter, we are concerned with model selection: which features should be used todefine the prediction score? The fact that models with few features (“sparse” models) aredesirable for several reasons (compactness, interpretability, good generalization) has stimulated much research work which has produced a wide variety of methods (Della Pietra
et al., 1997; Guyon and Elisseeff, 2003; McCallum, 2003). Our focus is on methods whichembed this selection into the learning problem via the regularization term. We depart fromprevious approaches in that we seek to make decisions jointly about all candidate features,and we want to promote sparsity patterns that go beyond the mere cardinality of the setof features. For example, we want to be able to select entire feature templates (rather thanfeatures individually), or to make the inclusion of some features depend on the inclusion ofother features.
We achieve the goal stated above by employing regularizers which promote structured
sparsity. Such regularizers are able to encode prior knowledge and guide the selection offeatures by modeling the structure of the feature space. Lately, this type of regularizers hasreceived a lot of attention in computer vision, signal processing, and computational biology(Zhao et al., 2009; Kim and Xing, 2010; Jenatton et al., 2009; Obozinski et al., 2010; Jenatton et al., 2010; Bach et al., 2011). Eisenstein et al. (2011) employed structured sparsity incomputational sociolinguistics. However, none of these works have addressed structured
prediction. Here, we combine these two levels of structure: structure in the output space,and structure in the feature space. The result is a framework that allows building structured predictors with high predictive power, while reducing manual feature engineering.
We obtain models that are interpretable, accurate, and often much more compact than L2 regularized ones. Compared with L1 -regularized models, ours are often more accurate andyield faster runtime.

10.2 Sparse Modeling in Structured Prediction
Consider again the learning problem associated with structured linear classifiers, that wehave described in Chapter 3:w
b = arg min Ω(w) +w ∈R D

1 N
L(w, x n , yn ),N n∑=1(10.1)

where Ω is a regularizer and L is a loss function.
In practice, it has been observed that the choice of loss (which we have addressed in
Chapter 8) has far less impact than the model design and choice of features. Hence, in thischapter, we focus our attention on the regularization term in Eq. 10.1. We specifically address ways in which this term can be used to help design the model by promoting structured
sparsity. While this has been a topic of intense research in signal processing and compu-

10.3. STRUCTURED SPARSITY: GROUP-LASSO REGULARIZATION

207

tational biology (Jenatton et al., 2009; Liu and Ye, 2010b; Bach et al., 2011), it has not yet
received much attention in the NLP community, where the choice of regularization for supervised learning has essentially been limited to the following (cf. Section 3.2.1 for details
and pointers to the literature):• L2 -regularization: ΩλL2 (w) := λ2 kwk22 ;• L1 -regularization: ΩτL1 (w) := τ kwk1 ,The latter is known as “Lasso,” as popularized by Tibshirani (1996) in the context of sparse
regression. In the two cases above, λ and τ are nonnegative coefficients controlling theintensity of the regularization. Each of the regularizers above is appealing for different
reasons: ΩλL2 usually leads to easier optimization and robust performance; ΩτL1 encourages
sparser models, where only a few features receive nonzero weights; see Gao et al. (2007) foran empirical comparison. More recently, Petrov and Klein (2008b) applied L1 regularization
for structure learning in phrase-based parsing; a comparison with L2 appears in Petrov andKlein (2008a). Elastic nets interpolate between L1 and L2 , having been proposed by Zou andHastie (2005) and used by Lavergne et al. (2010) to regularize CRFs.
Despite their differences, the regularizers mentioned above share a common property:
all of them “ignore” the structure of the feature space, since they all treat each dimension
independently—we call them unstructured regularizers, as opposed to the structured ones
that we next describe.

10.3 Structured Sparsity: Group-Lasso Regularization
We are interested in regularizers that share with ΩτL1 the ability to promote sparsity, so thatthey can be used for selecting features. In addition, we want to endow the feature space
RD with additional structure, so that features are not penalized individually (as in the L1 case) but collectively, encouraging entire groups of features to be discarded. The choice ofgroups will allow encoding prior knowledge regarding the kind of sparsity patterns that are
intended in the model. This can be achieved with group-Lasso regularization, which we next
describe.

10.3.1The Group Lasso

To capture the structure of the feature space, we group our D features into M groups
G1 , . . . , GM , where each Gm ⊆ {1, . . . , D }. Ahead, we discuss meaningful ways of choosing group decompositions; for now, let us assume a sensible choice is obvious to the model
designer. Denote by wm = (wd )d∈Gm the subvector of those weights that correspond to thefeatures in the m-th group, and let d1 , . . . , d M be nonnegative scalars (one per group). We
consider the following group-Lasso regularizers:MΩGL
d = ∑ d m k w m k2 .

(10.2)

m =1

These regularizers were first proposed by Bakin (1999) and Yuan and Lin (2006) in the context
of regression. If d1 = . . . = d M , ΩGLd becomes the “L1 norm of the L2 norms.” Interestingly,

208

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

this is also a norm, called the mixed L2,1 -norm.1 These regularizers subsume the L1 and L2
cases, which correspond to trivial choices of groups:• If each group is a singleton, i.e., M = D and Gd = {wd }, and d1 = . . . = d M = τ, werecover L1 -regularization.
• If there is a single group spanning all the features, i.e., M = 1 and G1 = {1, . . . , D }, thenthe right hand side of Eq. 10.2 becomes d1 kwk2 . This is equivalent to L2 regularization.2
We next present some non-trivial examples concerning different topologies of B ={ G1 , . . . , GM }.
Non-overlapping groups. Let us first consider the case where B is a partition of the featureS
space: the groups cover all the features ( m Gm = {1, . . . , D }), and they do not overlap(Ga ∩ Gb = ∅, ∀ a 6= b). Then, ΩGLd is termed a non-overlapping group-Lasso regularizer. It
encourages sparsity patterns in which entire groups are discarded. A judicious choice ofgroups can lead to very compact models and pinpoint relevant groups of features. The
following examples lie in this category:• The two cases above (L1 and L2 regularization).
• Label-based groups. In multi-label classification, where Y = {1, . . . , K }, features are
typically designed as conjunctions of input features with label indicators, i.e., theytake the form f ( x, y) = ψ( x ) ⊗ ey , where ψ( x ) ∈ RDX , ey ∈ RK has all entries zero
except the y-th entry, which is 1, and ⊗ denotes the Kronecker product. Hence f ( x, y)
can be reshaped as a DX -by-K matrix, and we can let each group correspond to a row.
In this case, all groups have the same size and we typically set d1 = . . . = d M . A similar
design can be made for sequence labeling problems, by considering a similar grouping
for the unigram features.3• Template-based groups. In NLP, features are commonly designed via templates. For example, a template such as w0 ∧ p0 ∧ p−1 denotes the word in the current position (w0 )conjoined with its part-of-speech (p0 ) and that of the previous word (p−1 ). This template encloses many features corresponding to different instantiantions of w0 , p0 , and
p−1 . In Section 10.5, we learn feature templates from the data, by associating each group
to a feature template, and letting that group contain all features that are instantiationsof this template. Since groups have different sizes, it is a good idea to let dm increase
with the group size, so that larger groups pay a larger penalty for being included.
1 In the statistics literature, such mixed-norm regularizers, which group features and then apply a separate
norm for each group, are called composite absolute penalties (Zhao et al., 2009); other norms besides L2,1 can beused, such as L∞,1 (Quattoni et al., 2009; Wright et al., 2009; Eisenstein et al., 2011).
2 To be precise, this is not exactly the same as the L regularization scenario we have described so far, where2
the L2 norm is squared. However it can be shown that both regularizers lead to identical learning problems(Eq. 10.1) up to a transformation of the regularization constant.
3 The same idea is also used in multitask learning, where labels correspond to tasks (Caruana, 1997); and in
multiple kernel learning, where they correspond to kernels—see Chapter 9.

10.3. STRUCTURED SPARSITY: GROUP-LASSO REGULARIZATION

209

Tree-structured groups. More generally, we may let the groups in B overlap but be nested,i.e., we may want them to form a hierarchy (two distinct groups either have empty intersection
or one is contained in the other). This induces a partial order on B (the set inclusion relation⊇), endowing it with the structure of a partially ordered set (poset).
A convenient graphical representation of the poset (B, ⊇) is its Hasse diagram. Eachgroup is a node in the diagram, and an arc is drawn from group Ga to group Gb if Gb ⊂ Ga
and there is no b0 s.t. Gb ⊂ Gb0 ⊂ Ga . When the groups are nested, this diagram is a forest(a union of directed trees). The corresponding regularizer enforces sparsity patterns wherea group of features is only selected if all its ancestors are also selected.4 Hence, entire subtrees
in the diagram can be pruned away. Examples are:
• The elastic net. The diagram of B has a root node for G1 = {1, . . . , D } and D leaf nodes,
one per each singleton group (see Figure 10.1).
• The sparse group-Lasso. This regularizer was proposed by Friedman et al. (2010):ΩSGLd,τ ( w ) =M0∑ (dm kwm k2 + τm kwm k1 ) ,(10.3)

m =1

where the total number of groups is M = M0 + D, and the components w1 , . . . , w M0 are
non-overlapping. This regularizer promotes sparsity at both group and feature levels(i.e., it eliminates entire groups and sparsifies within each group).

Graph-structured groups. In general, the groups in B may overlap without being nested.
In this case, the Hasse diagram of B is a directed acyclic graph (DAG). As in the treestructured case, a group of features is only selected if all its ancestors are also selected.
Based on this property, Jenatton et al. (2009) suggested a way of reverse engineering thegroups from the desired sparsity pattern. We next describe a strategy for coarse-to-fine featuretemplate selection that directly builds on that idea.
Suppose that we are given M feature templates T = { T1 , . . . , TM } which are partiallyordered according to some criterion, such that if Ta  Tb we would like to include Tb in
our model only if Ta is also included. This criterion could be a measure of coarseness: wemay want to let coarser part-of-speech features precede finer lexical features, e.g., p0 ∧ p1 
w0 ∧ w1 , or conjoined features come after their elementary parts, e.g., p0  p0 ∧ p1 . The
order does not need to be total, so some templates may not be comparable (e.g., we may
want p0 ∧ p−1 and p0 ∧ p1 not to be comparable). To achieve the sparsity pattern encodedin (T , ), we choose B = ( G1 , . . . , GM ) as follows: let I ( Ta ) be the set of features that areS
instantiations of template Ta ; then define Ga = b:ab I ( Tb ), for a = 1, . . . , M. It is easy to
see that (B, ⊇) and (T , ) are isomorph posets (their Hasse diagrams have the same shape;
see Figure 10.1). The result is a “coarse-to-fine” regularizer, which prefers to select featuretemplates that are coarser before zooming into finer features.
4 We say that a group of features G is selected if some feature in G (but not necessarily all) has a nonzerommweight.

210

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

Figure 10.1: Hasse diagrams of several group-based regularizers. For all tree-structured
cases, we use the same plate notation that is traditionally used in probabilistic graphicalmodels. The rightmost diagram represents a coarse-to-fine regularizer: each node is a template involving contiguous sequences of words (w) and POS tags (p); the symbol order∅  p  w induces a template order (Ta  Tb iff at each position i [ Ta ]i  [ Tb ]i ). Digits
below each node are the group indices where each template belongs.

10.3.2

Bayesian Interpretation

The prior knowledge encoded in the group-Lasso regularizer (Eq. 10.2) comes with a Bayesianinterpretation, as we next describe. In a probabilistic model (e.g., in the CRF case, where
L = LCRF ), the optimization problem in Eq. 3.6 can be seen as maximum a posteriori estimation of w, where the regularization term Ω(w) corresponds to the negative log of a priordistribution (call it p(w)). It is well-known that L2 -regularization corresponds to choosing
independent zero-mean Gaussian priors, wd ∼ N(0, λ−1 ), and that L1 -regularization resultsfrom adopting zero-mean Laplacian priors, p(wd ) ∝ exp(τ |wd |).
Figueiredo (2002) provided an alternative interpretation of L1 -regularization in terms ofa two-level hierarchical Bayes model, which happens to generalize to the non-overlappinggroup-Lasso case, where Ω = ΩGL
d . As in the L2 -case, we also assume that each parameterreceives a zero-mean Gaussian prior, but now with a group-specific variance τm , i.e., wm ∼N(0, τm I ) for m = 1, . . . , M. This reflects the fact that some groups should have their feature
weights shrunk more towards zero than others. The variances τm ≥ 0 are not pre-specified
but rather generated by a one-sided exponential hyperprior p(τm |dm ) ∝ exp(−d2m τm /2). It
can be shown that after marginalizing out τm , we obtainp(wm |dm ) =Z ∞0p(wm |τm ) p(τm |dm )dτm

∝ exp (−dm kwm k) .

(10.4)

Hence, the non-overlapping group-Lasso corresponds to the following two-level hierachical
Bayes model: independently for each m = 1, . . . , M,τm ∼ Exp(d2m /2),

wm ∼ N(0, τm I ).

(10.5)

10.4. ONLINE PROXIMAL GRADIENT ALGORITHMS

10.3.3211

Prox-operators

Before introducing our learning algorithm for handling group-Lasso regularization, we recall
the concept of a Ω-proximity operator, defined in Appendix B. This is the function proxΩ :RD → RD defined as follows:1
k w 0 − w k2 + Ω ( w 0 ).
proxΩ (w) = arg min
w0 2(10.6)

Proximity operators generalize Euclidean projections and have many interesting properties;
see Bach et al. (2011) for an overview. By requiring zero to be a subgradient of the objectivefunction in Eq. 10.6, we obtain the following closed expression (called soft-thresholding) forthe ΩτL1 -proximity operator: wd − τ if wd > τ(10.7)[proxΩ L1 (w)]d =0
if |wd | ≤ ττ w + τ if w < −τ.
ddFor the non-overlapping group Lasso case, the proximity operator is given by([proxΩGL (w)]m =d0k w m k2 − d mwm
k w m k2

if kwm k2 ≤ dmotherwise.

(10.8)

which can be seen as a generalization of Eq. 10.7: if the L2 -norm of the m-th group is lessthan dm , the entire group is discarded; otherwise it is scaled so that its L2 -norm decreases by
an amount of dm .
When groups overlap, the proximity operator lacks a closed form. When B is treestructured, it can still be efficiently computed by a recursive procedure (Jenatton et al., 2010).
When B is not tree-structured, no specialized procedure is known, and a convex optimizer
is necessary to solve Eq. 10.6.

10.4 Online Proximal Gradient Algorithms
We now turn our attention to efficient ways of handling group-Lasso regularizers. Several
fast and scalable algorithms having been proposed for training L1 -regularized CRFs, basedon quasi-Newton optimization (Andrew and Gao, 2007), coordinate descent (Sokolovska
et al., 2010; Lavergne et al., 2010), and stochastic gradients (Carpenter, 2008; Langford et al.,2009; Tsuruoka et al., 2009). The algorithm that we use in this chapter (Algorithm 17) is avariation of the one introduced in Chapter 9 for multiple kernel learning. It extends thestochastic gradient methods for group-Lasso regularization.
Algorithm 17 addresses the learning problem in Eq. 10.1 by alternating between online(sub-)gradient steps with respect to the loss term, and proximal steps with respect to theregularizer. Proximal-gradient methods are very popular in sparse modeling, both in batch(Liu and Ye, 2010b; Bach et al., 2011) and online (Duchi and Singer, 2009; Xiao, 2009) settings.
The reason we have chosen the algorithm proposed in Chapter 9 is that it effectively handles
overlapping groups, without the need of evaluating proxΩ (which, as seen in Section 10.3.3,

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

212

Algorithm 17 Online Sparse Prox-Grad Algorithm
data D, (Ω j ) jJ=1 , number of rounds T, gravity sequence ((σjt ) jJ=1 )tT=1 , stepsizesequence (ηt )tT=1
2: initialize w = 0
3: for t = 1 to T do
4:choose n = n(t) ∈ {1, . . . , N } and take training pair ( x n , yn )5:w ← w − ηt ∇ L(θ; x n , yn ) (gradient step)
6:
for j = 1 to J do
7:
w = proxηt σjt Ω j (w) (proximal step)
8:
end for
9: end for
10: output: w
1: input:can be costly if B is not tree-structured). To do so, we decompose Ω asJΩ(w) = ∑ σj Ω j (w)(10.9)

j =1

for some J ≥ 1, and nonnegative σ1 , . . . , σJ ; each Ω j -proximal operator is assumed easy tocompute. Such a decomposition always exists: if B does not have overlapping groups, takeS
J = 1. Otherwise, find J ≤ M disjoint sets B1 , . . . , B J such that jJ=1 B j = B and the groups
on each B j are non-overlapping. The proximal steps are then applied sequentially, one pereach Ω j . Overall, Algorithm 17 satisfies the following important requirements:• Computational efficiency. Each gradient step at round t is linear in the number of featuresthat fire for that instance and independent of the total number of features D. Eachproximal step is linear in the number of groups M, and does not need be to performed
every round (as we will see later).
• Memory efficiency. Only a small active set of features (those that have nonzero weights)
need to be maintained. Entire groups of features can be deleted after each proximal
step. Furthermore, only the features which correspond to nonzero entries in the gradient vector need to be inserted in the active set; for some losses (the structured hinge
loss LSSVM and the structured perceptron loss LSP ) many irrelevant features are neverinstantianted.
• Convergence. With high probability, Algorithm 17 produces an e-accurate solution afterT ≤ O(1/e2 ) rounds, for a suitable choice of stepsizes and holding σjt constant, σjt = σj
(Proposition 9.8 in Chapter 9). This result can be generalized to any sequence (σjt )tT=1
such that σj = T1 ∑tT=1 σjt .
We next describe several algorithmic ingredients that make Algorithm 17 effective insparse modeling.
Budget-Driven Shrinkage. Algorithm 17 requires the choice of a “gravity sequence.” We
follow Langford et al. (2009) and set (σjt ) jJ=1 to zero for all t which is not a multiple of someprespecified integer K; this way, proximal steps need only be performed each K rounds,

10.4. ONLINE PROXIMAL GRADIENT ALGORITHMS

213

yielding a significant speed-up when the number of groups M is large. A direct adoptionof the method of Langford et al. (2009) would set σjt = Kσj for those rounds; however, wehave observed that such a strategy makes the number of groups vary substantially in early
epochs. We use a different strategy: for each B j , we specify a budget of Bj ≥ 0 groups (thismay take into consideration practical limitations, such as the available memory). If t is amultiple of K, we set σjt as follows:
1. If B j does not have more than Bj nonzero groups, set σjt = 0 and do nothing.
2. Otherwise, sort the groups in B j by decreasing order of their L2 -norms. Check theL2 -norms of the Bj -th and Bj+1 -th entries in the list and set σjt as the mean of these two
divided by ηt .
3. Apply a ηt σjt Ω j -proximal step using Eq. 10.8. At the end of this step, no more than Bj
groups will remain nonzero.5
If the average of the gravity steps converge, limT →∞ T1 ∑tT=1 σjt → σj , then the limit points
σj implicitly define the regularizer, via Ω = ∑ jJ=1 σj Ω j .6 Hence, we have shifted the controlof the amount of regularization to the budget constants Bj , which unlike the σj have a clear
meaning and can be chosen under practical considerations.
Space and Time Efficiency. The proximal steps in Algorithm 17 have a scaling effect oneach group, which affects all features belonging to that group (see Eq. 10.8). We want toavoid explicitly updating each feature in the active set, which could be time consuming. We
mention two strategies that can be used for the non-overlapping group Lasso case.
• The first strategy is suitable when M is large and only a few groups ( M) have features that fire in each round; this is the case, e.g., of label-based groups (see Section 10.3.1).
It consists of making lazy updates (Carpenter, 2008), i.e., to delay the update of all features in a group until at least one of them fires; then apply a cumulative penalty. The
amount of the penalty can be computed if one assigns a timestamp to each group.
• The second strategy is suitable when M is small and some groups are very populated;
this is the typical case of template-based groups (Section 10.3.1). Two operations need
to be performed: updating each feature weight (in the gradient steps), and scaling
entire groups (in the proximal steps). We adapt a trick due to Shalev-Shwartz et al.
(2007): represent the weight vector of the m-th group, wm , by a triple (ξ m , cm , ρm ) ∈R|Gm | × R+ × R+ , such that wm = cm ξ m and kwm k2 = ρm . This representation allows
performing the two operations above in constant time, and it keeps track of the group
L2 -norms, necessary in the proximal updates.
For sufficient amounts of regularization, our algorithm has a low memory footprint.
Only features that, at some point, intervene in the gradient computed in line 5 need to
be instantiated; and all features that receive zero weights after some proximal step can bedeleted from the model (cf. Figure 10.2).
5 When overlaps exist (e.g. the coarse-to-fine case), we specify a total pseudo-budget B ignoring the overlaps,
which induces budgets B1 , . . . , B J which sum to B. The number of actually selected groups may be less than B,
however, since in this case some groups can be shrunk more than once. Other heuristics are possible.
6 The convergence assumption can be sidestepped by freezing the σ after a fixed number of iterations.
jCHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

214

F1 (%)
model size (# features)

MIRA
93.10
5,300,396

92.99
71,075

Group Lasso (template-based)
93.28
93.59
93.42
93.40
158,844 389,065 662,018 891,378

Table 10.1: Results for text chunking.
MIRA
Spa. dev/test
Dut. dev/test
Eng. dev/test

70.38/74.09
8,598,246
69.15/71.54
5,727,004
83.95/79.81
8,376,901

C = 0.1
69.19/71.9
68,56564.07/66.35
164,960
80.92/76.95
232,865

Lasso
C = 0.5
70.75/72.38
1,017,769
66.82/69.42
565,704
82.58/78.84
870,587C=1
71.7/74.03
1,555,683
70.43/71.89
953,668
83.38/79.35
1,114,016

Group Lasso (template-based)
B = 100
B = 200
B = 300
71.79/73.62
72.08/75.05
71.48/73.3
83,036354,872
600,646
69.48/72.83
71.03/73.33
71.2/72.59
128,320
447,193889,660
85.62/80.26
85.86/81.47
85.03/80.91
255,165
953,178
1,719,229

Table 10.2: Results for named entity recognition. Each cell shows F1 (%) and the number offeatures.
Sparseptron and Debiasing. Although Algorithm 17 allows to simultaneously select features and learn the model parameters, it has been observed in the sparse modeling literaturethat Lasso-like regularizers usually have a strong bias which may harm predictive performance. A post-processing stage is usually taken (called debiasing), in which the model isrefitted without any regularization and using only the selected features (Wright et al., 2009).
If a final debiasing stage is to be performed, Algorithm 17 only needs to worry about feature
selection, hence it is appealing to choose a loss function that makes this procedure as simple
as possible. Examining the input of Algorithm 17, we see that both a gravity and a stepsize sequence need to be specified. The former can be taken care of by usingpbudget-driven
shrinkage, as described above. The stepsize sequence can be set as ηt = η0 / dt/N e, which
ensures convergence, however η0 requires tuning. Fortunately, for the structured perceptron
loss LSP (Eq. 3.31), Algorithm 17 is independent of η0 , up to a scaling of w, which does notaffect predictions.7 We call the instantiation of Algorithm 17 with a group-Lasso regularizer
and the loss LSP the sparseptron. Overall, we propose the following two-stage approach:
1. Run the sparsepton for a few epochs and discard the features with zero weights.
2. Refit the model without any regularization and using the loss L which one wants tooptimize.

10.5 Experiments
We present experiments in three structured prediction tasks for several group choices: text
chunking, named entity recognition, and arc-factored dependency parsing (see Chapter 2
for a detailed description of these tasks).
Text Chunking. We use the English dataset provided in the CoNLL 2000 shared task (Sangand Buchholz, 2000), which consists of 8,936 training and 2,012 testing sentences (sections
15–18 and 20 of the WSJ.) The input observations are the token words and their POS tags;
7 To see why this is the case, note that both gradient and proximal updates come scaled by η ; and that the0
gradient of the loss is ∇ LSP (w, x, y) = f ( x, yb) − f ( x, y), where yb is the prediction under the current model,
which is insensitive to the scaling of w. This independence on η0 does not hold when the loss is LSSVM or LCRF .

10.5. EXPERIMENTS

215

we want to predict the sequences of IOB tags representing phrase chunks. We built 96
contextual feature templates as follows:• Up to 5-grams of POS tags, in windows of 5 tokens on the left and 5 tokens on theright;• Up to 3-grams of words, in windows of 3 tokens on the left and 3 tokens on the right;• Up to 2-grams of word shapes, in windows of 2 tokens on the left and 2 tokens on
the right. Each shape replaces characters by their types (case sensitive letters, digits,
and punctuation), and deletes repeated types—e.g., Confidence and 2,664,098 arerespectively mapped to Aa and 0,0+,0+ (Collins, 2002b).
We defined unigram features by conjoining these templates with each of the 22 output labels.
An additional template was defined to account for label bigrams—features in this template
do not look at the input string, but only at consecutive pairs of labels.8
We evaluate the ability of group-Lasso regularization to perform feature template selection.
To do that, we ran 5 epochs of the sparseptron algorithm with template-based groups andbudget-driven shrinkage (budgets of 10, 20, 30, 40, and 50 templates were tried). For eachgroup Bm , we set dm = log2 | Gm |, which is the average number of bits necessary to encodea feature in that group, if all features were equiprobable. We set K = 1000 (the number of
instances between consecutive proximal steps). Then, we refit the model with 10 iterationsof the max-loss 1-best MIRA algorithm (Crammer et al., 2006).9 Table 10.1 compares theF1 scores and the model sizes obtained with the several budgets against those obtained byrunning 15 iterations of MIRA with the original set of features. Note that the total numberof iterations is the same; yet, the group-Lasso approach has a much smaller memory footprint (see Figure 10.2) and yields much more compact models. The small memory footprint
comes from the fact that Algorithm 17 may entertain a large number of features without
ever instantiating all of them. The predictive power is comparable (although some choicesof budget yield slightly better scores for the group-Lasso approach).10
Named Entity Recognition. We experiment with the Spanish, Dutch, and English datasets
provided in the CoNLL 2002/2003 shared tasks (Sang, 2002; Sang and De Meulder, 2003).
For Spanish, we use the POS tags provided by Carreras (http://www.lsi.upc.es/˜nlp/
tools/nerc/nerc.html); for English, we ignore the syntactic chunk tags provided with thedataset. Hence, all datasets have the same sort of input observations (words and POS) andall have 9 output labels. We use the feature templates described above plus some additional
ones (yielding a total of 452 templates):• Up to 3-grams of shapes, in windows of size 3;• For prefix/suffix sizes of 1, 2, 3, up to 3-grams of word prefixes/suffixes, in windowsof size 3;
8 State-of-the-art models use larger output contexts, such as label trigrams and 4-grams. We resort to bigram

labels as we are mostly interested in identifying relevant unigram templates.
9 This variant optimizes the L
SSVM loss, as described in Chapter 8. For the refitting, we used unregularized
MIRA. For the baseline (described next), we used L2 -regularized MIRA and tuned the regularization constantwith cross-validation.
10 We also tried label-based group-Lasso and sparse group-Lasso (Section 10.3.1), with less impressive results.

216

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION
6# Features

6

x 10

4
MIRA
Sparceptron + MIRA (B=30)

2
0
0

5

10

15

# Epochs

Figure 10.2: Memory footprints of the MIRA and sparseptron algorithms in text chunking.
The oscillation in the first 5 epochs (bottom line) comes from the proximal steps each K =1000 rounds. The features are then frozen and 10 epochs of unregularized MIRA follow.
Overall, the sparseptron requires < 7.5% of the memory as the MIRA baseline.
• Up to 5-grams of case, punctuation, and digit indicators, in windows of size 5.
As before, an additional feature template was defined to account for label bigrams. We dofeature template selection (same setting as before) for budget sizes of 100, 200, and 300. We
compare with both MIRA (using all the features) and the sparseptron with a standard Lasso
regularizer ΩτL1 , for several values of C = 1/(τN ). Table 10.2 shows the results. We observethat template-based group-Lasso wins both in terms of accuracy and compactness. Note alsothat the ability to discard feature templates (rather than individual features) yields faster test
runtime than models regularized with the standard Lasso: fewer templates will need to beinstantiated, with a speed-up in score computation.
Multilingual Dependency Parsing. We trained non-projective dependency parsers for 6
languages using the CoNLL-X shared task datasets (Buchholz and Marsi, 2006): Arabic,
Danish, Dutch, Japanese, Slovene, and Spanish. We chose the languages with the smallest
datasets, because regularization is more important when data is scarce. The output to bepredicted from each input sentence is the set of dependency links, which jointly define aspanning tree. We use arc-factored models, for which exact inference is tractable (McDonald
et al., 2005b). We defined M = 684 feature templates for each candidate arc by conjoining thewords, shapes, lemmas, and POS of the head and the modifier, as well as the contextual POS,
and the distance and direction of attachment. We followed the same two-stage approachas before, and compared with a baseline which selects feature templates by ranking themaccording to the information gain criterion. This baseline assigns a score to each template Tm
which reflects an empirical estimate of the mutual information between Tm and the binary
variable A that indicates the presence/absence of a dependency link:
IGm ,

∑

∑

f ∈ Tm a∈{0,1}P( f , a) log2

P( f , a),P( f ) P( a)(10.10)where P( f , a) is the joint probability of feature f firing and an arc being active (a = 1) or
innactive (a = 0), and P( f ) and P( a) are the corresponding marginals. All probabilities are
estimated from the empirical counts of events observed in the data.
The results are plotted in Figure 10.3, for budget sizes of 200, 300, and 400. We observethat for all but one language (Spanish is the exception), non-overlapping group-Lasso regularization is more effective at selecting feature templates than the information gain criterion,

10.5. EXPERIMENTS

217

DanishArabic

93.5

89.8

78
UAS (%)

Japanese

90

78.5

93

89.6
77.5

76.5
2

89.4

Group−Lasso
Group−Lasso (C2F)
Lasso
Filter−based (IG)

77

4

6
8
10
Number of Features

92.5

89.2
12

89
0

5

10

15

6

92
0

2

4

8
6

x 10

Slovenex 10

Spanish

84

6

6

x 10

Turkish

84

76

83.5

75.5

83

75

82.5

74.5

83

82

81
0

2

4

6

8

10
6

x 10

82
0

0.5

1

1.5

2
x 10

7

74
0

5

10

15
6

x 10

Figure 10.3: Comparison between non-overlapping group-Lasso, coarse-to-fine group-Lasso(C2F), and a filter-based method based on information gain for selecting feature templates
in multilingual dependency parsing. The x-axis is the total number of features at different
regularization levels, and the y-axis is the unlabeled attachment score. The plots illustrate
how accurate the parsers are as a function of the model sparsity achieved, for each method.
The standard Lasso (which does not select templates, but individual features) is also shownfor comparison.

218

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

Bilexical
Lex. → POS
POS → Lex.
POS → POS
Middle POS
Shape
Direction
Distance

Ara.
++
+
++
++
++
++

Dan.
+
+
++
++
+
+

Jap.

Slo.

+
+
++
++
++
+
+

+
++
++
+
+

Spa.
+

Tur.

+

+

++

++

+
+

+
+

Table 10.3: Variation of feature templates that were selected accross languages. Each line
groups together similar templates, involving lexical, contextual POS, word shape information, as well as attachment direction and length. Empty cells denote that very few or none of
the templates in that category was selected; + denotes that some were selected; ++ denotesthat most or all were selected.

and slightly better than coarse-to-fine group-Lasso. For completeness, we also display the
results obtained with a standard Lasso regularizer. Table 10.3 shows what kind of featuretemplates were most selected for each language. Some interesting patterns can be observed:
morphologically-rich languages with small datasets (such as Turkish and Slovene) seem toavoid lexical features, arguably due to potential for overfitting; in Japanese, contextual POS
appear to be specially relevant. It should be noted, however, that some of these patterns may
be properties of the datasets rather than of the languages themselves.

10.6 Related Work
The online proximal gradient algorithm used in this chapter was proposed for multiple
kernel learning in Martins et al. (2011b), along with a theoretical analysis, which we include
in Chapter 9. Budget-driven shrinkage and the sparseptron are novel techniques, to thebest of our knowledge. Apart from Martins et al. (2011b), the only work we are awareof which combines structured sparsity with structured prediction is Schmidt and Murphy(2010); however, their goal is to predict the structure of graphical models, while we aremostly interested in the structure of the feature space. Schmidt and Murphy (2010) usedgenerative models, while our approach emphasizes discriminative learning.
Mixed norm regularization has been used for a while in statistics as a means to promote
structured sparsity. Group Lasso is due to Bakin (1999) and Yuan and Lin (2006), after
which a string of variants and algorithms appeared (Bach, 2008b; Zhao et al., 2009; Jenatton
et al., 2009; Friedman et al., 2010; Obozinski et al., 2010). The flat (non-overlapping) casehas tight links with learning formalisms such as multiple kernel learning (Lanckriet et al.,2004) and multi-task learning (Caruana, 1997). The tree-structured case has been addressedby Kim and Xing (2010), Liu and Ye (2010b) and Mairal et al. (2010), along with L∞,1 and L2,1
regularization. Graph-structured groups are discussed in Jenatton et al. (2010), along with a
DAG representation. In NLP, mixed norms have been used recently by Graça et al. (2009) inposterior regularization, and by Eisenstein et al. (2011) in a multi-task regression problem.

10.7. CONCLUSIONS AND FUTURE WORK

219

10.7 Conclusions and Future WorkIn this chapter, we have explored two levels of structure in NLP problems: structure on
the outputs, and structure on the feature space. We have shown how the latter can be
useful in model design, through the use of regularizers which promote structured sparsity.
We propose an online algorithm with minimal memory requirements for exploring large
feature spaces. Our algorithm, which specializes into the sparseptron, yields a mechanism forselecting entire groups of features. We apply the sparseptron for selecting feature templates
in three structured prediction tasks, with advantages over filter-based methods, L1 , and L2
regularization in terms of performance, compactness, and model interpretability.
The work described in this chapter is just a first incursion in promoting structured sparsity in NLP; there are plenty of avenues for future research. The results that we have obtainedfor coarse-to-fine regularization are a little bit disappointing, since this strategy is outperformed by non-overlapping groups. However, we cannot draw strong conclusions from this.
Our online algorithms, despite their scalabilility and memory efficiency advantages, haveslow convergence, and it may happen that the final solution we obtain is not the optimal
one—this is worse in the coarse-to-fine regularization case, since the groups overlap, and theproximal step is not exact.
M . These weights play a very imAnother important issue are the prior weights {dm }m=1
portant role, and it is not obvious how to set them when groups are very unbalanced. A
principled strategy for choosing the weights would be highly desirable. A related topic isthat of learning the groups automatically from the data, a problem which has been addressedin some previous work (Huang et al., 2009; Lorbert et al., 2010; Grave et al., 2011), but whichis still far from solved. This would be an alternative to the approach taken here, in which
the choice of groups is done manually as a way of expressing prior knowledge about theintended sparsity patterns.
Finally, it is worth mentioning that if the problem at hand is not very large, other online or batch algorithms may be a better choice for achieving structured sparsity. For nonoverlapping groups, we have observed in preliminary experiments that the regularized dual
averaging algorithm proposed by Xiao (2010) is very effective at getting sparse iterates, andit could be an alternative option. Other batch algorithms may also be a good alternative,
as they have have excelled in signal processing applications (Bioucas-Dias and Figueiredo,
2007; Wright et al., 2009; Beck and Teboulle, 2009). For overlapping groups, the ADMM algorithm (see Chapter 6 for the use of such algorithm in inference, rather than learning) has
also been used with great success (Afonso et al., 2010).

220

CHAPTER 10. STRUCTURED SPARSITY FOR STRUCTURED PREDICTION

Part IV

Conclusions

221Chapter 11

Conclusions
In this chapter, we summarize our contributions and highlight some open problems, suggesting possible directions for future research. Fine-grained discussions about each topic
can be found at the end of the corresponding chapters.

11.1 Summary of Contributions
In this thesis, we have advanced the state of the art in structured prediction. Generally
speaking, our contributions fall into two realms:• We have proposed a new methodology for inference in structured prediction, in which
the design of models is decoupled from algorithmic considerations. This allows constructing rich models that incorporate complex and global features and constraints.
We have shown how to handle these models with principled approximate algorithms
based on relaxations. We have applied the proposed methodology to an important
problem in NLP, dependency parsing, with substantial gains in predictive power.
• We have contributed new methods for learning structured predictors in an online manner. We have made progress on two fronts: by proposing regularizers that are capableof selecting features by promoting structured sparsity; and by presenting algorithms
that deal with various of loss functions under a unified framework.
Regarding the inference problem, we have contributed a better theoretical understandingof constrained structured prediction (Chapter 5). We provided a formal characterization of constrained graphical models and their geometry, extending known results for unconstrainedmodels. We made these models capable of handling declarative constraints by introducing a simple set of logic factors, and we derived analytical expressions for their messages,
marginals, entropies, and marginal polytopes. Our contributions are relevant for several
frameworks that have been considered in NLP, such as constrained conditional models (Rothand Yih, 2004) and Markov logic networks (Richardson and Domingos, 2006).
Given a unified treatment of constrained models, we introduced a new dual decomposition
inference algorithm, called AD3 (Chapter 6), whose convergence properties we analyzed. The
algorithm has the same modular architecture of previous dual decomposition algorithms,
such as the ones proposed by Komodakis et al. (2007); Rush et al. (2010): a centralized
controller iteratively broadcasts subproblems to workers, and gathers their local solutions to
223

224

CHAPTER 11. CONCLUSIONS

make a global update. AD3 is particularly suitable for handling declarative constraints, andwe have derived procedures for solving the subproblems associated with all logic factors
above. In addition, we have provided an active set method for handling arbitrary factors,
requiring only an oracle for computing the local MAP at each factor. Our experiments withbenchmark datasets (Ising and Potts models, protein design, and frame semantic parsing)give encouraging results.
We applied the machinery above to an important problem in NLP, dependency parsing(Chapter 7). We started by providing a concise ILP formulation based on multi-commodity
flows, improving on previous formulations which require an exponential number of constraints (Riedel and Clarke, 2006). Having done this, we opened the door for incorporatingrich global features and constraints into the model, which led to a substantial impact inperformance. We then relaxed this ILP and showed that the resulting parser is an instanceof a turbo parser: it performs approximate inference in a loopy graphical model, ignoring
global effects caused by the loops. We explicitly derived the underlying graphical modeland showed that other parsers Smith and Eisner (2008); Koo et al. (2010) are also turbo
parsers.
Regarding the learning problem, we started by considering a wide family of loss functions, which generalizes CRFs, SVMs, and the loss underlying the structured perceptron
algorithm (Chapter 8). We presented a new family of online algorithms that can be seen
as dual coordinate ascent algorithms, and that generalize the MIRA algorithm (Crammer andSinger, 2003; Crammer et al., 2006) to other loss functions, including CRFs. The resulting
algorithm is similar to online and stochastic gradient descent, but it does not require specifying a learning rate hyperparameter.
We then turned to the regularizer in the learning problem (Chapters 9–10). We proposed
new online proximal-gradient algorithms that can gracefully handle block structured regularizers, providing their regret and convergence properties. We use those regularizers in the context of structured prediction, to learn combinations of multiple kernels (Chapter 9), and to
identify relevant feature templates (Chapter 10), in which we take into account the structureof the feature space and the kind of sparsity patterns that are desired. Our algorithms areable to explore large feature spaces with fast runtime and minimal memory requirements.

11.2 Future Work
There are many possible avenues for future research in the scope of this thesis. We outline
just a few.

11.2.1 Broader Constrained Formalisms and Better Entropy Approximations
We linked constrained structured prediction to constrained factor graphs, but in fact it could
be useful to consider broader formalisms, such as case-factor diagrams (McAllester et al.,2008), AND/OR search spaces (Dechter and Mateescu, 2007), and sum-product networks(Poon and Domingos, 2011). Not much is known about the geometry underlying these formalisms, and it is an open problem to obtain duality and variational characterizations of thepartition functions and entropies in these broader classes. In the same line, our “negative”
result in Chapter 5 concerning the poor quality of Bethe entropy approximations in some11.2. FUTURE WORK

225

constrained graphical models puts a demand for better entropy approximations, anotheropen problem whose study has mostly been focused on the unconstrained case (Wiegerinck
and Heskes, 2003; Wainwright et al., 2005b; Weiss et al., 2007; Hazan and Shashua, 2010). The
connection mentioned in Section 5.6 with the posterior regularization framework of Ganchev
et al. (2010) is also potentially useful.

11.2.2

Applying AD3 to Other Combinatorial Problems

So far, the AD3 algorithm has been applied to two combinatorial NLP problems, dependency
parsing (Chapter 7) and frame-semantic parsing (Section 6.7; see more details in Das 2012).
However, there is a wide universe of NLP problems involving logical constraints for which
AD3 looks useful, such as compositional semantics (Carpenter, 1997; Liang et al., 2011),
coreference resolution (Denis and Baldridge, 2007), syntax-based and phrase-based machinetranslation (Rush and Collins, 2011; Chang and Collins, 2011) and summarization (Clarkeand Lapata, 2008; Martins and Smith, 2009; Berg-Kirkpatrick et al., 2011), to name just a few.
The last also involves budget constraints, which AD3 never addressed before.
A careful empirical analysis of AD3 on a wide set of problems is desirable. Many addons have been incorporated in AD3 since the algorithm was first proposed in Martins et al.
(2011a), and it is likely that new ones might be incorporated in the future. Initially, dense,
large and combinatorial factors were seen as a bottleneck, when compared with other dual
decomposition and message-passing algorithms. The active set method described in Section 6.5 addresses this issue, enabling the application of AD3 to any decomposable problem
as long as the components have local MAP oracles. It would be interesting to compare AD3
with the projected subgradient algorithm of Komodakis et al. (2007) and Rush et al. (2010)
regarding the number of oracle calls necessary to achieve a desired level of precision.
Other engineering tricks can be used to speed up AD3 , for example through parallelization, as discussed in Section 6.8. Another interesting direction concerns the branch-andbound search procedure described in Section 6.6. While the successful results in framesemantic parsing are promising, additional experiments need to be made to ensure therobustness of the method for general problems. It is likely that further engineering can
speed up the search, through careful caching and warm-starting of the branch-and-bounditerations. An alternative approach to branch-and-bound is a cutting-plane tightening procedure similar to that in Sontag et al. (2008). It would be interesting to compare these two
approaches.

11.2.3Sparse Structured Prediction

In the context of the AD3 algorithm, we have devised a procedure for maximizing a linear
score regularized by an Euclidean penalty, which pushes for a sparse solution, expressed as a
combination of a few outputs—see, e.g., Proposition 6.6, which puts a bound on the supportof the solution. This quadratic problem is interesting on its own, since the solution implicitly
defines a sparse distribution over the output space. This can be useful for developing pruning
models, as an alternative to typical approaches based on hard thresholding or K-best lists(Charniak et al., 2006; Weiss and Taskar, 2010). The AD3 algorithm can be adapted for thiskind of problem with minimal changes, since all subproblems are still quadratic.

CHAPTER 11. CONCLUSIONS

226

11.2.4

Turbo Parsers and Other Syntactic Formalisms

Given the ability of AD3 to deal with global features and constraints, a lot of feature engineering could be done for incorporating better global features and constraints into ourparsing models, using expert linguistic knowledge. We suggested some in Section 7.6. It
would also be useful to approach turbo parsing using the variant of AD3 with the active set
method, comparing its speed with the dual decomposition parsers of Koo et al. (2010).
It is also possible to consider tighter relaxations for parsing, either polyhedral or semidefinite. Examples are the Sherali-Adams, Lovász-Schrijver and Lasserre hierarchies studied
by Sontag (2010). Our branch-and-bound procedure can also be applied. However, ourexperiments suggest that simple linear relaxations are already quite effective and often tight,
so search error does not seem to be an issue, at least for dependency parsing.
Other syntactic formalisms, such as phrase-structure grammars, tree-adjoining grammars, or lexical-functional grammars may also be addressed with similar approaches to theones described in Chapter 6. The AD3 algorithm looks particularly suitable for dealing withthe kind of logical constraints induced by unification-based grammars.

11.2.5

Learning With Approximate Inference

The impact of approximate inference on the learning problem deserves further study. Whilesome previous work has addressed this issue (Wainwright, 2006; Kulesza and Pereira, 2007;
Finley and Joachims, 2008; Martins et al., 2009c), there is still a huge gap between theory
and practice. In Martins et al. (2009c), we provided conditions that guarantee algorithmic
separability under LP-relaxed inference, however the bounds seem too loose to be usefulin practice. On the other hand, most theoretical counter-examples seem to rarely happen
in practice. It would be useful to have a theory of learning with approximate inference,
establishing necessary conditions for learnability and providing a better theoretical understanding of this problem. It is likely that polyhedral characterizations (Grünbaum, 2003;
Schrijver, 2003) and the theory of approximation algorithms (Vazirani, 2001) can be useful inthis context.

11.2.6Structured Sparsity in NLP

Our work on structured sparsity is a first step in modeling the structure of the feature space;however, much is left to be done. Broadly speaking, discovering structure in language has
been a long-term goal since the early days of computational linguistics, and there is hopethat the current advances in sparse modeling can be the key to accomplish that goal.
In our dissertation, we have focused on online algorithms, and we were particularly concerned about memory efficiency. For small and medium-scale problems, there are batchalgorithms with better optimization guarantees that might be more suitable for the task.
Examples are SpaRSA (Wright et al., 2009) and FISTA (Beck and Teboulle, 2009), which canbe easily extended to tree-structured group-Lasso regularization. These algorithms have alarger memory footprint, but asymptotically have much faster convergence rates—for example, for differentiable loss functions with Lipschitz continuous gradients (such as CRFs),√FISTA has an iteration bound of O(1/ e), which is much better than the O(1/e2 ) boundof online algorithms. For non-hierarchical overlapping groups, the ADMM algorithm has11.2. FUTURE WORK

227

also been used with great success (Afonso et al., 2010). There is also the online regularized
dual averaging algorithm by Xiao (2010), which is very effective in practice at getting sparse
iterates. It would be interesting to evaluate the performance of such algorithms on NLP
tasks.
Another important issue are the prior weights in the penalty term. Currently, settingthese weights is a black art, and it would be highly desirable to have a principled strategy
that works when groups are unbalanced.

228

CHAPTER 11. CONCLUSIONSPart V

Appendices229

Appendix A

Background Proofs
A.1

Proof of Proposition 2.1

To prove Proposition 2.1, we need a couple of lemmata.
Lemma A.1 Let y be a dependency tree. If an arc (h, m) ∈ y is non-projective, then it must be“crossed inward” by some other arc (h0 , m0 ) ∈ y, i.e., m0 6= h is in the span of (h, m) and h0 is
outside that span (see Figure A.1).
Proof. If (h, m) is non-projective, then by definition there must be some k ∈ span(h, m) suchthat h  k. Let {(0, k1 ), (k1 , k2 ), . . . (k J −1 , k J )} be the path of arcs from the root to k J := k; thenh is not touched by any arc in this path, and since 0 lies outside span(h, m) there must be
some k j with 1 ≤ j ≤ J such that k j is inside the span and k j−1 is not. Hence the arc (k j−1 , k j )crosses (h, m) inward.
Note that the converse of Lemma A.1 does not hold: in the tree {(0, 1), (4, 2), (1, 3), (1, 4)}
the arc (1, 3) is projective, despite the fact that it is crossed inward by (4, 2).
Lemma A.2 Let y be a dependency tree. If two arcs cross each other, then at least one is nonprojective.
Proof. Let (h, m) and (h0 , m0 ) be the crossing arcs. There are three scenarios that must
be considered (see Figure A.1): (i) both arcs cross outward; (ii) one crosses inward andthe other outward; (iii) both cross inward. Let us start with (i) and assume, w.l.o.g., thatm0 < h < h0 < m. Suppose that (h, m) is projective; then we must have h  h0 . Similarly,
if we assume that (h0 , m0 ) is projective; we must have h0  h. Thus, if the two arcs areprojective we must have h = h0 , which leads to a contradiction. For case (ii) assume w.l.o.g.
that h < h0 < m < m0 . If (h, m) is projective, we must have h  h0 ; if (h0 , m0 ) is projective, wemust have h0  m. But since h is the immediate predecessor of m and h0 6= m, we must haveh0  h as well, which again leads to h0 = h. Finally, assume for case (iii), w.l.o.g., that thath < m0 < m < h0 . If (h, m) is projective, we must have h  m0 and similarly to the previous
case this implies h  h0 . By symmetry, if (h0 , m0 ) is projective we must have h0  h and weare lead to yet another contradiction.
We now prove Proposition 2.1, i.e., that a dependency tree is projective if and only if noarcs cross. From Lemma A.1 we have that any tree which is not projective must have crossing
arcs. This proves the direction ⇐. To prove the direction ⇒, invoke Lemma A.2: any pair of231

Figure A.1: The three ways arcs can cross each other.

crossing arcs implies the existence of at least one non-projective arc, which renders the tree
not projective.

A.2

Derivation of the MPLP Algorithm

Consider the LP-MAP problem of Eq. 4.50. By writing the local polytope inequalities explicitly, and introducing new variables ζ αi (yα ) := µα (yα ), which are copies of the alreadyexisting factor marginals µα (.) (one copy per variable i ∈ N(α)), we can rewrite the LP-MAP
problem as:
maximize∑ ∑ θi ( yi ) µi ( yi ) + ∑ ∑ θ α ( y α ) µ α ( y α )i ∈ V y i ∈ Yi

w.r.t.
s.t.

(A.1)

α ∈F y α ∈Yαµ, ζ∀α ∈ F, i ∈ N(α), yi ∈ Yi ,µi (yi ) = ∑yα ∼yi ζ αi (yα ),(A.2)

ζ αi (yα ) = µα (yα ),∀α ∈ F, i ∈ N(α), yα ∈ Yα ,(A.3)

∑yi ∈Yi µi (yi ) = 1,∀i ∈ V,(A.4)µi (yi ) ≥ 0,∀i ∈ V, ∀yi ∈ Yi ,(A.5)

ζ αi (yα ) ≥ 0,∀α ∈ F, i ∈ N(α), yα ∈ Yα .

(A.6)

We then introduce Lagrange multipliers γα→i (yi ) and δαi (yα ) for each of the constraints (A.2)
and (A.3). The Lagrangian function becomes:
L(µ, ζ, γ, δ) =∑ ∑i ∈ V y i ∈ Yi+

∑ ∑α ∈F y α ∈Yα−θi ( yi ) +∑ γα → i ( y i )!
µi ( yi )α ∈N ( i )θα (yα ) −∑ ∑ ∑α ∈F y α ∈Yα i ∈N ( α )∑i ∈N ( α )!
δαi (yα )µα (yα )
γα→i (yi ) − δαi (yα ) ζ αi (yα ).

(A.7)

We want to maximize this function with respect to µ and ζ, subject to the simplex constraints(A.4–A.5) and the non-negativity constraint A.6, and to minimize it with respect to γ and δ.
Let us take into consideration the following facts: (i) maximizing a linear function over thesimplex amounts to picking the largest coordinate; (ii) a maximization of L with respect to ζ
would explode to +∞, unless γα→i (yi ) − δαi (yα ) ≥ 0 holds for every factor α, variable i, andassignment yα ; (iii) at optimality, those constraints must be active, so the inequalities can bereplaced by equalities. This originates the constraint
γα→i (yi ) = max δαi (yα ).

(A.8)

y α ∼ yiPutting these ingredients together, we obtain the following expression for the dual problem:
minimize

∑ max
y ∈Y

i ∈V

w.r.t.
s.t.

i

θi ( yi ) µi ( yi ) +δi ( y )∑ ymax∼y α αα ∈N ( i )

i

α

!
(A.9)

i

δ
θα (yα ) = ∑i∈N(α) δαi (yα ),∀α ∈ F, yα ∈ Yα .

This dual is then optimized by a coordinate descent method: at each time, one considers
a single factor α and optimizes with respect to δα := (δαi (.))i∈N(α) , helding the remaining
components of δ fixed. The contribution of those components to the dual objective is thesum ∑ β∈N(i)\{α} maxyβ ∼yi δβi (y β ), which by Eq. A.8 equals ∑ β∈N(i)\{α} γβ→i (yi ). Introducing
the variable
δi→α (yi ) := θi (yi ) +∑γ β →i ( y i ),(A.10)β∈N(i )\{α}

it can be shown (Globerson and Jaakkola, 2008) that the solution bδα of that problem is:δbαi (yα ) = −δi→α (yi ) +1 θ α ( y ) + ∑ δj → α ( y j )  .
αdeg(α)j ∈N ( α )(A.11)

Hence, a coordinate descent optimization that goes sequentially through each factor α ∈ Fseeking to optimize δα yields a message-passing scheme in which, for such factor, one first
updates the incoming messages through Eq. A.10, and then updates the outgoing messages
as follows:
1γα→i (yi ) := −δi→α (yi ) +max θα (yα ) + ∑ δj→α (y j ) .
(A.12)
deg(α) yα ∼yij ∈N ( α )Appendix B

Convex Analysis and Optimization
In this section, we briefly review some notions of convex analysis that are used throughout
the thesis. For more details on this subject, see e.g. Rockafellar (1970); Bertsekas et al. (1999);
Boyd and Vandenberghe (2004); Bauschke and Combettes (2011).

B.1 Convex Sets, Hulls, and Polytopes
Let V be a vector space over the field of the real numbers R. We will be mostly concernedwith the case where V is the Euclidean space RD or, more generally, a Hilbert space H.
A set C ⊆ V is convex if, for all x, y ∈ C and λ ∈ [0, 1], we have λx + (1 − λ)y ∈ C. Thisimplies that for any finite set of points { x1 , . . . , x N } ⊆ C and any choice of non-negative
coefficients λ1 , . . . , λ N such that ∑iN=1 λi = 1, we have ∑iN=1 λi xi ∈ C. The vector ∑iN=1 λi xi iscalled a convex combination of { x1 , . . . , x N }.
The convex hull of a set X ⊆ V is the set of all convex combinations of the elements of X:()
conv X :=N∑ λi xi

i =1

N

N ∈ N, { x1 , . . . , x N } ⊆ X, ∑ λi = 1, λi ≥ 0, ∀i

;

(B.1)

i =1

it is also the smallest convex set that contains X.
The affine hull of a set X ⊆ V is the set of all affine combinations of the elements of X,(
aff X :=

N

∑ λi xi

i =1

N

)

N ∈ N, { x1 , . . . , x N } ⊆ X, ∑ λi = 1 ;

(B.2)

i =1

it is also the smallest affine set that contains X.
Let H
p be a Hilbert space with inner product h., .i and corresponding norm k.k given byk xk := h x, xi. Given x ∈ H, we denote by Bγ ( x) := {y ∈ H | k x − yk ≤ γ} the ball withradius γ centered in x. The relative interior of a set X ⊆ H is its interior relative to the affine
hull X,
relint X := { x ∈ X | ∃γ > 0 : Bγ ( x) ∩ aff X ⊆ X} .
(B.3)
A convex polytope Z ⊆ RD is a set which can be expressed as a convex hull of a finite set ofpoints { x1 , . . . , x N } ∈ RD . We say that this finite set is minimal is no point xi can be written
as a convex combination of the others. If { x1 , . . . , x N } is a minimal set, then its elements235

are the vertices of Z. A representation of this form, Z = conv{ x1 , . . . , x N }, is called a vertex
representation of the convex polytope Z.
A convex polyhedron is a set P ⊆ RD which can be written as P = { x ∈ RD | Ax ≤ b}, forsome N-by-D matrix A and vector b ∈ R N . Minkowsky-Weyl theorem (see, e.g., Rockafellar1970) states that a set P is a polytope if and only if it is a bounded polyhedron.
Given a convex polytope Z ⊆ RD , we say that a convex polyhedron P ⊆ RP , with P ≥ D,
is a lifting or a lifted version of Z if we have Z = {z ∈ RD | (z, y) ∈ P}.

B.2 Convex Functions, Subdifferentials, Proximity Operators, andMoreau Projections
Let H be a Hilbert space with inner product h., .i and induced norm k.k. Let R̄ := R ∪ {+∞}
be the set of extended reals. Throughout, we let ϕ : H → R̄ be a convex, lower semicontinuous(lsc) and proper function—this means, respectively, that:• The effective domain of ϕ, domϕ := { x ∈ H | ϕ( x) < +∞} and the epigraph of ϕ,
epiϕ := {( x, t) ∈ H × R | ϕ( x) ≤ t}, are both convex sets;• epiϕ is closed in H × R;
• domϕ 6= ∅.
The subdifferential of ϕ at x0 is the set∂ϕ( x0 ) := { g ∈ H | ∀ x ∈ H, ϕ( x) − ϕ( x0 ) ≥ h g, x − x0 i},(B.4)

the elements of which are the subgradients. We say that ϕ is G-Lipschitz in S ⊆ H if ∀ x ∈S, ∀ g ∈ ∂ϕ( x), k g k ≤ G. We say that ϕ is σ-strongly convex in S if

∀ x0 ∈ S,∀ g ∈ ∂ϕ( x0 ),∀ x ∈ H,ϕ( x) ≥ ϕ( x0 ) + h g, x − x0 i + (σ/2)k x − x0 k2 .

(B.5)

The Fenchel conjugate of ϕ, denoted ϕ? : H → R̄, is defined byϕ? (y) := suphy, xi − ϕ( x).

(B.6)

x

ϕ? is always convex, since it is the supremum of a family of affine functions. Some examples
follow:• If ϕ is an affine function, ϕ( x) = h a, xi + b, then ϕ? (y) = −b if y = a and −∞otherwise.
• If H = RD and ϕ is the ` p -norm, ϕ( x) = k xk p , then ϕ? is the indicator of the unit ballinduced by the dual norm, ϕ? (y) = 0 if kykq ≤ 1 and +∞ otherwise, with p−1 + q−1 =1.
• If H = RD and ϕ is half of the squared ` p -norm, ϕ( x) = k xk2p /2, then ϕ? is half of thesquared dual norm, ϕ? (y) = kyk2q /2, with p−1 + q−1 = 1.
• If ϕ is convex, lsc, and proper, then ϕ?? = ϕ.

• If ψ( x) = tϕ( x − x0 ), with t ∈ R+ and x0 ∈ H, then ψ? (y) = h x0 , yi + tϕ? (y/t).
The Moreau envelope of ϕ, denoted M ϕ : H → R̄, and the proximity operator of ϕ, denotedprox ϕ : H → H, are defined, respectively, as
1
M ϕ (y) := inf k x − yk2 + ϕ( x),x 2
1prox ϕ (y) := arg inf k x − yk2 + ϕ( x).
x 2

(B.7)
(B.8)

Proximity operators generalize Euclidean projectors in the following sense: consider theindicator function ιC of a convex set C ⊆ H, i.e., ιC ( x) = 0 if x ∈ C, and +∞ otherwise. Then,proxιC is the Euclidean projector onto C and MιC is the residual of the projection. Two otherimportant examples of proximity operators are:• if ϕ( x) = (λ/2)k xk2 (with λ ≥ 0) then prox ϕ (y) = y/(1 + λ);• if H = Rd and ϕ( x) = τ k xk1 , then prox ϕ (y) = soft(y, τ ) is the soft-threshold function(Donoho and Johnstone, 1994), defined as:[soft(y, τ )]k = sgn(yk ) · max{0, |yk | − τ }.

(B.9)

We terminate by stating Danskin’s theorem, which allows us to compute subgradients offunctions that have a variational representation. A proof can be found in Bertsekas et al.
(1999, p.717).
Theorem B.1 (Danskin (1966)) Let C ⊆ RD be a compact set and ϕ : RK × C → R be continuousand such that ϕ(., x) : RK → R is convex for every x ∈ C. Then:
1. The function ψ : RK → R given by
ψ(y) := max ϕ(y, x)(B.10)x ∈C

is convex.
2. Let C(y) be the set of maximizing points in Eq. B.10. If ϕ(y, x) is continuously differentiablewith respect to y for each x ∈ C, then the subdifferential of ψ is given by:∂ψ(y) = conv

∂ϕ(y, x)∂yx ∈ C( y ) .

(B.11)

Appendix C

Derivation of Messages for Logic
Factors
C.1 Sum-Product Messages
The One-hot XOR Factor. For the partition function, we need to sum over all the allowed
configurations (i.e., those in the acceptance set SXOR ). This can be done explicitly for the XOR
factor, since the cardinality of this set is linear in K. We obtain:K

ZXOR (ω ) =∑ ∏ Mk→XOR (yk )y∈SXOR k =1

K

K

=

∑ Mi→XOR (1) ∏ Mk→XOR (0)k=1,k 6=i

i =1

K

=
=

Mi→XOR (1)∑ Mi→XOR (0)
i =1
K

K

i =1

i =1

!

K

× ∏ Mk→XOR (0)k =1∑ mi→XOR × ∏(1 + mi→XOR )−1 .

(C.1)

Expressions for the marginals can be derived similarly. We obtain:µi (1; ω ) = ZXOR (ω )−1 ×K∑ ∏ Mk→XOR (yk )y∈SXOR k =1
y i =1= ZXOR (ω )−1 × Mi→XOR (1)K∏ Mk→XOR (0)k =1,k6=iK= ZXOR (ω )−1 × mi→XOR ∏ (1 + mk→XOR )−1
k =1=

mi→XORK∑k=1 mk→XOR
239

(C.2)

and
µi (0; ω ) = 1 − µi (1; ω )∑kK=1,k6=i mk→XOR=∑kK=1 mk→XOR

,

(C.3)

so the marginal ratio is:µi (1; ω )m= K i→XOR.
µi (0; ω )∑k=1,k6=i mk→XOR(C.4)

From Proposition 5.4, we have
MXOR→i (1)µ (1; ω )
1
= m−
× i
,
k
→
XOR
MXOR→i (0)µi (0; ω )mXOR→i =

(C.5)

hence
! −1

K

∑

mXOR→i =mk→XOR(C.6).

k =1,k6=i

Finally, the entropy is given by:K

HXOR (bXOR (.)) = log ZXOR (ω ) − ∑∑µi (yi ; ω ) log Mi→XOR (yi )i =1 yi ∈{0,1}KK= log ZXOR (ω ) − ∑ µi (1; ω ) log mi→XOR + ∑ log(1 + mi→XOR )i =1

K∑ mi→XOR

= log

!

i =1
K

∑ mi→XOR

= log

i =1
K

= −∑

i =1

i =1

K

− ∑ µi (1; ω ) log mi→XOR
i =1

!

K

−∑

mi→XORK∑k=1 mk→XOR

i =1

mi→XORK∑k=1 mk→XOR

!
log

!
log mi→XOR

mi→XORK∑k=1 mk→XOR

!
,

(C.7)

that is,K

HXOR (bXOR (.)) = − ∑ bi (1) log bi (1).

(C.8)

i =1

The OR Factor. We next turn to the OR factor. For the partition function, note that (unlike the XOR case) we cannot efficiently enumerate all the allowed configurations in SOR ,since there are 2K − 1 of them. However, we can write the sum as if it was over all 2K
configurations—which will yield a simple expression— and then discount the all-zeros configuration, which is the one which does not belong to SOR . More concretely, note that we
have:

∑

K

K

∏ Mk→OR (yk ) = ∏y∈{0,1}K k =1∑

k =1 yk ∈{0,1}

Mk→OR (yk ),(C.9)

which equals 1 if we assume that the incoming messages are normalized—which we canwithout loss of generality.1 We thus obtain:K

ZOR (ω ) =∑ ∏ Mk→OR (yk )y∈SOR k=1

KK

y∈{0,1}K k=1

k =1

=

∏ Mk→OR (yk ) − ∏ Mk→OR (0)∑K= 1 − ∏(1 + mi→OR )−1 .

(C.10)

i =1

For the marginals, we can follow the same procedure:µi (1; ω ) = ZOR (ω )−1 ×K∑ ∏ Mk→OR (yk )y∈SOR k =1
y i =1= ZOR (ω )−1 × Mi→OR (1) ×K∑ ∏ Mk→OR (yk )y1 ,...,yi−1 k=1
yi+1 ,...,yK= ZOR (ω )−1 × Mi→OR (1),(C.11)and µi (0; ω ) = 1 − µi (1; ω ), which yields the following ratio of marginals:µi (1; ω )µi (0; ω )=ZOR (ω )−1 × Mi→OR (1)1 − ZOR (ω )−1 × Mi→OR (1)= Mi→OR (1) × ( ZOR (ω ) − Mi→OR (1))−1K= mi→OR (1 + mi→OR )−1 1 − ∏ (1 + mk→OR )−1 − mi→OR (1 + mi→OR )−1

! −1

k =1

 −1


K



= mi→OR 1 − ∏ (1 + mk→OR )−1 .

(C.12)

k =1
k 6 =i

From Proposition 5.4, we now have that the message updates are given by:
K

mOR→i =1−

∏ (1 + mk→OR )

! −1
−1

.

(C.13)

k =1,k6=i

1 Recall that the message update equations (Eqs. 4.7 and 4.8) and the belief equations (Eqs. 4.9 and 4.10) are

unaffected by multiplying each Mi→α or Mα→i by a positive constant.

Finally, the entropy is given by:K

HOR (bOR (.)) = log ZOR (ω ) − ∑∑µi (yi ; ω ) log Mi→OR (yi )i =1 yi ∈{0,1}KK= log ZOR (ω ) − ∑ µi (1; ω ) log mi→OR + ∑ log(1 + mi→OR )
i =1

i =1

K

= log 1 − ∏(1 + mi→OR )−1

!

i =1

K

−∑

K

∏ (1 + mk→OR )−1

1 + mi→OR −! −1

× mi→OR log mi→OR

k =1,k 6=i

i =1K+ ∑ log(1 + mi→OR ).

(C.14)i =1

The OR-with-output Factor. We next derive expressions for the log-partition function andmarginals associated with the OR-with-output factor. As in the case of the OR-factor, wecannot efficiently enumerate all the allowed configurations in SOR-out , since there are 2K−1 ofthem. However, we can decouple the sum in two parts: one which includes all assignments
for which yK+1 = 1, which can be reduced to the computation of the partition function for
the OR factor; and other which considers the case where yK+1 = 0, whose only allowed
configuration is the all-zeros one. Thus, we obtain:
K +1

ZOR-out (ω ) =∑ ∏ Mk→OR-out (yk )y∈SOr-out k =1KK

y1 ,...,yK k =1

k =1

∑ ∏ Mk→OR-out (yk ) − ∏ Mk→OR-out (0)= MK+1→OR-out (1)

!

+

K +1

∏ Mk→OR-out (0)

k =1

!

K

K +1

= MK+1→OR-out (1) 1 − ∏ Mk→OR-out (0) + ∏ Mk→OR-out (0).
k =1

(C.15)

k =1

The marginals can be computed similarly. We consider first the case where i ≤ K:
µi (1; ω ) = ZOR-out (ω )−1 ×

K +1∑ ∏ Mk→OR-out (yk )y∈Sor-out k =1
y i =1= ZOR-out (ω )−1 × Mi→OR-out (1) × MK+1→OR-out (1),(C.16)

where we have used the fact that∑K

∏ Mk→OR-out (yk ) = 1.

y1 ,...,yi−1 k =1,k6=iyi+1 ,...,yK(C.17)

and µi (0; ω ) = 1 − µi (1; ω ). After some algebra, the ratio of marginals becomes:µi (1; ω )µi (0; ω )Mi→OR-out (1) × MK+1→OR-out (1)ZOR-out (ω ) − Mi→OR-out (1) × MK+1→OR-out (1) −1ZOR-out (ω )=−1
Mi→OR-out (1) × MK+1→OR-out (1)


 −1

MK+1→OR-out (1) 1 − ∏kK=1 Mk→OR-out (0) + ∏kK=+11 Mk→OR-out (0)− 1= 
Mi→OR-out (1) × MK+1→OR-out (1)



 −1
K
1
1 − 1 − m−
×
M
(
0
)
∏
k
→
OR-outk=1
K +1→OR-out= − 1Mi→OR-out (1)
! −1

=

1
= mi→OR-out 1 − (1 − m−
K +1→OR-out )K∏ (1 + mk→OR-out )−1.

(C.18)

k =1,k 6=i

We next consider the case where i = K + 1; it is easier to derive µK+1 (0; ω ) instead ofµK+1 (1; ω ):µK+1 (0; ω ) = ZOR-out (ω )−1 ×

K +1∑ ∏ Mk→OR-out (yk )y∈SOR-out k=1
y K +1 = 0
K +1= ZOR-out (ω )−1 × ∏ Mk→OR-out (0),(C.19)

k =1

and µK+1 (1; ω ) = 1 − µK+1 (0; ω ). After some algebra, the ratio of marginals becomes:µK+1 (1; ω )µK+1 (0; ω )

=
=

=

=

ZOR-out (ω ) − ∏kK=+11 Mk→OR-out (0)∏kK=+11 Mk→OR-out (0)ZOR-out (ω )−1
K +1
∏k=1 Mk→OR-out (0)
MK+1→OR-out (1) 1 − ∏kK=1 Mk→OR-out (0) + ∏kK=+11 Mk→OR-out (0)∏kK=+11 Mk→OR-out (0)
MK+1→OR-out (1) 1 − ∏kK=1 Mk→OR-out (0)−1∏kK=+11 Mk→OR-out (0)= mK+1→OR-out ×

1 − ∏kK=1 Mk→OR-out (0)∏kK=1 Mk→OR-out (0)K= mK+1→OR-out ×∏ (1 + mk→OR-out ) − 1

k =1

!
.

(C.20)

From Proposition 5.4, we then obtain the following message updates:

mOR-out→i =1
1 − (1 − m −K +1→OR-out )K∏ (1 + mk→OR-out )

! −1
−1

,

(C.21)

k =1,k 6=i

for i ≤ K, andK

mOR-out→K+1 = ∏ (1 + mk→OR-out ) − 1.

(C.22)

k =1

Like in the previous factors, the entropy can be computed via:
K +1

HOR-out (bOR-out (.)) = log ZOR-out (ω ) − ∑∑µi (yi ; ω ) log Mi→OR-out (yi )i =1 yi ∈{0,1}

K +1= log ZOR-out (ω ) − ∑ µi (1; ω ) log mi→OR-out
i =1

K +1

+ ∑ log(1 + mi→OR-out ).

(C.23)i =1

C.2 Max-Product Messages

The One-hot XOR Factor. We are now going to compute expressions for the mode andmax-marginals associated with the one-hot XOR factor. Then, invoking Proposition 5.5, we
can provide closed-form expressions for the messages and beliefs.
Let us start with the mode, which requires us to maximize over all the allowed configurations (i.e., those in the acceptance set SXOR ). Like in the sum-product case, this can be done
explicitly, since the cardinality of SXOR is linear in K. We obtain:∗
PXOR(ω ) =

Kmax ∏ Mk→XOR (yk )y∈SXOR

k =1




K



= max  Mi→XOR (1) ∏ Mk→XOR (0)i ∈[K ]

=k =1
k 6 =iMi→XOR (1)maxi ∈[K ] Mi →XOR (0)K

× ∏ Mk→XOR (0)k =1K= max mi→XOR × ∏(1 + mi→XOR )−1 .
i ∈[K ]

i =1

(C.24)

For the max-marginals, we obtain:Kνi (1; ω ) =max ∏ Mk→XOR (yk )y∈SXORk =1
y i =1K= Mi→XOR (1) ∏ Mk→XOR (0)k =1
k 6 =iK= mi→XOR ∏ (1 + mk→XOR )−1(C.25)

k =1

and
K

νi (0; ω ) =max ∏ Mk→XOR (yk )y∈SXORk =1
y i =0K= Mi→XOR (0) × max  M j→XOR (1) ∏ Mk→XOR (0)j∈[K ]\{i }k =1k∈/ {i,j}K= Mi→XOR (0) × max m j→XOR × ∏ (1 + mi→XOR )−1
j∈[K ]\{i }k =1
k 6 =iK=

max m j→XOR × ∏ (1 + mi→XOR )−1 ,(C.26)

νi (1; ω )mi→XOR.
=νi (0; ω )maxk∈[K]\{i} mk→XOR(C.27)

j∈[K ]\{i }k =1

so the marginal ratio is:

From Proposition 5.5, we now have that the message updates are given by:
MXOR→i (1) ∝ Mi→XOR (1)−1 × νi (1; ω )(C.28)

MXOR→i (0) ∝ Mi→XOR (0)−1 × νi (0; ω ),(C.29)

andwhence, since we have
mXOR→i =Mi→XOR (1)ν (1; ω )
1
= m−
× i
,
k
→
XOR
Mi→XOR (0)νi (0; ω )(C.30)

we obtain
 −1
mXOR→i =max mk→XOR
k 6 =i.

(C.31)

The OR Factor. We next derive expressions for the mode and max-marginals associatedwith the OR factor. Let us start with the mode. The acceptance set SOR accepts all configurations except the all-zeros one, where y1 = . . . = yK = 0. Assuming for the moment thatthe all-zeros configuration would not receive the largest score, we could easily compute themode as the componentwise product of the maxima,
K

Mk→OR (yk ).
∏ max
y

k =1

(C.32)k

The only scenario that would get us in trouble would be if for every k ∈ [K ] we have
Mk→OR (1) < Mk→OR (0), since in that case the unique MAP would be the all-zeros configuration, which is forbidden. This condition can be written equivalently as
Mk→OR (1)< 1.
k ∈[K ] Mk→OR (0)max(C.33)

If (C.33) holds, then the true mode of the OR factor is instead the second best configuration; inM(1)other words, we would need to look at the largest ratio Mk→OR (0) and flip the correspondingk→OR
yk to 1; the mode would then be the value in Eq. C.32 times this largest ratio. This can all be
written compactly via the following expression:
K

Mk→OR (yk )
∏ max
y

∗
POR
(ω ) =

k

k =1

!



M
(1)
× min 1, max k→ORk ∈[K ] Mk→OR (0).

(C.34)The minimum with one in Eq. C.34 is taken to ensure that the multiplication just mentioned
takes place if and only if condition (C.33) holds.

We next turn to the max-marginals. By definition, we haveKνi (1; ω ) =max ∏ Mk→OR (yk )

y∈SOR
k =1
y i =1K= Mi→OR (1) ∏ max Mk→OR (yk ),k =1
k 6 =i(C.35)

yk

since any assignment in which yi = 1 will necessarily be in the acceptance set. The computation of bi (0; ω ) is less trivial but may be carried out with a similar reasoning as thecomputation of the mode:Kνi (0; ω ) =max ∏ Mk→OR (yk )

y∈SOR
k =1
y i =0




K




= Mi→OR (0)  ∏ max Mk→OR (yk ) × min 1, max
k =1
k 6 =iyk

Mk→OR (1)k ∈[K ]\{i } Mk→OR (0). (C.36)

The ratio of the max-marginals is:
K

Mi→OR (1) ∏ max Mk→OR (yk )νi (1; ω )νi (0; ω )k =1
k 6 =i

=



yk


K




Mi→OR (0)  ∏ max Mk→OR (yk ) × min 1, max
k =1
k 6 =i

Mk→OR (1)k ∈[K ]\{i } Mk →OR (0)

yk



1

= mi→OR ×

min 1, max mk→ORk∈[K ]\{i }−1= mi→OR × max 1, min mk→OR .

(C.37)k ∈[K ]\{i }As a consequence, we obtain the following simple expression for the max-product message
updates, invoking Proposition 5.5:mOR→i = max

1
1, min m−
k→OR
k 6 =i.

(C.38)The OR-with-output Factor. We now turn to the problem of computing the mode in the
OR-with-output factor. Two things can happen:• Either the MAP assignment has yK+1 = 0, in which case everything must be zero;• Or the MAP assignment has yK+1 = 1, in which case the problem can be reduced to
the OR case, as at least one yk (with k ∈ [K ]) must be one.

Hence, the mode will be the maximum of these two possibilities. The first possibility corresponds to the score
K +1∏ Mk→OR-out (0);(C.39)

k =1

the second corresponds to MK+1→OR-out (1) times the score in Eq. C.34. Putting the pieces
together, we obtain(∗
POR-out(ω ) = max

K +1∏ Mk→OR-out (0),MK+1→OR-out (1)×k =1
K

Mk→OR-out (yk )
∏ max
y

k =1

k

!



M
(1)
× min 1, max k→OR-out
k ∈[K ] Mk →OR-out (0)). (C.40)

We now turn to the max-marginals. Let us first handle the case where i ≤ K. By
definition, we have
K +1

νi (1; ω ) =∏ Mk→OR-out (yk )maxy∈SOR-outk =1
y i =1K= Mi→OR-out (1) MK+1→OR-out (1) ∏ max Mk→OR-out (yk ),k =1
k 6 =i(C.41)

yk

since any assignment in which yi = 1 will necessarily be in the acceptance set, provided
yK+1 = 1. As in the OR case, the computation of νi (0; ω ) is less trivial but may be carriedout with a similar reasoning as the computation of the mode:
K +1

νi (0; ω ) =max∏ Mk→OR-out (yk )y∈SOR-outk =1
y i =0 K +1∏ Mk→OR-out (0), MK+1→OR-out (1)×
 k =1k 6 =i






K
Mk→OR-out (1) 


max
M
(
y
)
×
min
1,
max
. (C.42)∏k→OR-out k ykk∈[K ]\{i } Mk →OR-out (0) 
k =1= Mi→OR-out (0) × max

k 6 =iLet us now address the case where i = K + 1. We have that the computation of νK+1 (1; ω )can be reduced to computing a mode of the OR-factor:
K +1

νK+1 (1; ω ) =max∏ Mk→OR-out (yk )y∈SOR-outk =1
y K +1 = 1

K= MK+1→OR-out (1) ×max

y1 ,...,yK ∈SOR

∏ Mk→OR-out (yk )k =1= MK+1→OR-out (1) ×
K

Mk→OR-out (yk )
∏ max
y

k =1

k

!



M
(1)
× min 1, max k→OR-out
k ∈[K ] Mk →OR-out (0).

(C.43)

As for νK+1 (0; ω ), it is immediate from the definition:
K +1

νK+1 (0; ω ) =max∏ Mk→OR-out (yk )y∈SOR-outk =1
y K +1 = 0
K +1=∏ Mk→OR-out (0).

k =1(C.44)Given the max-marginals, we can now compute their ratios. As before, we start with thecase i ≤ K. After some algebra, one obtains:
K

mi→OR-out × ∏ max Mk→OR-out (yk )νi (1; ω )νi (0; ω )

=

yk

k =1
k 6 =i



K




−
1


m
×
M
(
0
)
,


k
→
OR-out∏K+
1→
OR-out




k
=
1




k
6
=
i


max
 


K



 




 ∏ max Mk→OR-out (yk ) × min 1, max mk→OR-out 


yk


k
∈[
K
]\{
i
}
 k =1k 6 =i=max

mi→OR-out


K


M
(
0
)
k →OR-out

1
 m−

, 


 K+1→OR-out × ∏ min
yk Mk →OR-out ( yk ) 
k =1

k 6 =i










 min 1, max mk→OR-outk ∈[K ]\{i }


K







 mK+1→OR-out × ∏ max{1, mk→OR-out }, 
k =1
= mi→OR-out × min
,
k
6
=
i








1


 max 1, min m−
k→OR-out(C.45)k ∈[K ]\{i }where in the last equality we have used the fact thatM(0)
min k→OR-out
yk Mk →OR-out ( yk )

 −1

= max
yk

Mk→OR-out (yk )= max{1, mk→OR-out }.
Mk→OR-out (0)(C.46)For i = K + 1, we have:νK+1 (1; ω )νK+1 (0; ω )= mK+1→OR-out ×K∏ max Mk→OR-out (yk )

k =1

!

yk



M
(1)
× min 1, max k→OR-out
k ∈[K ] Mk →OR-out (0)K∏ Mk→OR-out (0)k =1= mK+1→OR-out ×
K

Mk→OR-out (yk )∏ max
ykMk→OR-out (0)
k =1

!



M
(1)
× min 1, max k→OR-out
k ∈[K ] Mk →OR-out (0)= mK+1→OR-out ×K∏ max{1, mk→OR-out }

k =1

!





× min 1, max mk→OR-out
k∈[K ](C.47)

As a consequence of the facts above, we obtain the following simple expression for the
max-product message updates:

mOR-out→i = min







K



1
mK+1→OR-out ∏ max{1, mk→OR-out }, max 1, min m−
k →OR-outk ∈[K ]\{i }k =1
k 6 =i(C.48)

for i ≤ K, andK

mOR-out→K+1 =∏ max{1, mk→OR-out }

k =1

!



× min 1, max mk→OR-out .
k ∈[K ]

(C.49)

Appendix D

Proof of Convergence Rate of ADMM
and AD3
In this appendix, we prove the O(1/e) convergence bound of the ADMM algorithm, first in
the sense of a variational inequality involving the primal and dual variables, using resultsrecently established by He and Yuan (2011) and Wang and Banerjee (2012), and then in termsof the dual objective value, which follows easily from the aforementioned results. We thenconsider the special case of AD3 , where we interpret the constants in the bound in terms ofproperties of the graphical model.
We start with the following proposition, which states the variational inequality associatedwith the Lagrangian function of the problem in Eq. 6.1. Recall that (6.1) is equivalent tomax min L(u, v, λ),λ∈Λ u∈U,v∈V(D.1)

where L = L0 is the standard Lagrangian (the expression in Eq. 6.3 with η = 0), and Λ :={λ | minu∈U,v∈V L(u, v, λ) > −∞}.
Proposition D.1 (Variational inequality.) Let W := U × V × Λ. Given w = (u, v, λ) ∈ W, define h(w) := f (u) + g(v) and F (w) := (A> λ, B> λ, −(Au + Bv − c)). Then, w∗ := (u∗ , v∗ , λ∗ ) ∈W is a primal-dual solution of Eq. D.1 if and only if:∀w ∈ W,h(w) − h(w∗ ) + (w − w∗ ) · F (w∗ ) ≥ 0.

(D.2)

Proof.
Assume w∗ is a primal-dual solution of Eq. D.1. Then, from the saddle pointconditions, we have for every w := (u, v, λ) ∈ W:
L(u∗ , v∗ , λ) ≤ L(u∗ , v∗ , λ∗ ) ≤ L(u, v, λ∗ ).

(D.3)

Hence:
0 ≤ L(u, v, λ∗ ) − L(u∗ , v∗ , λ)=f (u) + g(v) + λ∗ · (Au + Bv − c) − f (u∗ ) − g(v∗ ) − λ · (Au∗ + Bv∗ − c)= h(w) − h(w∗ ) + u · A> λ∗ + v · B> λ∗ − (λ − λ∗ ) · (Au∗ + Bv∗ − c) − λ∗ · (Au∗ + Bv∗ )= h ( w ) − h ( w ∗ ) + ( w − w ∗ ) · F ( w ∗ ).

(D.4)
251

Conversely, let w∗ satisfy Eq. D.2. Taking w = (u∗ , v∗ , λ), we obtain L(u∗ , v∗ , λ) ≤ L(u∗ , v∗ , λ∗ ).
Taking w = (u, v, λ∗ ), we obtain L(u∗ , v∗ , λ∗ ) ≤ L(u, v, λ∗ ). Hence (u∗ , v∗ , λ∗ ) is a saddle
point, and therefore it is a primal-dual solution.

We next reproduce the result established by Wang and Banerjee (2012) (which is related
to previous work by He and Yuan (2011)) concerning the convergence rate of the general
ADMM algorithm in terms of the variational inequality stated in Proposition D.1. The proof
relies itself on variational inequalities associated with the updates in the ADMM algorithm(see Facchinei and Pang 2003 for a comprehensive overview of variational inequalities andcomplementarity problems).

Proposition D.2 (Variational convergence rate.) Assume the conditions stated in Proposition 6.2,
τ = 1, and λ0 = 0. Let w̄ T = T1 ∑tT=1 wt , where the wt := (ut , vt , λt ) are the ADMM iterates.
Then we have after T iterations:∀w ∈ W,h(w̄ T ) − h(w) + (w̄ T − w) · F (w̄ T ) ≤

C
,
T

(D.5)

where
C=

η
1
kAu + Bv0 − ck2 + kλk2 .
2
2η

(D.6)

Proof. From the variational inequality associated with the u-update in Eq. 6.6 we have, forevery u ∈ U,
0≤

∇ u L η ( u t +1 , v t , λ t ) · ( u − u t +1 )=∇ f (ut+1 ) · (u − ut+1 ) + (u − ut+1 ) · A> (λt + η (Aut+1 + Bvt − c))≤ (i )

f (u) − f (ut+1 ) + (u − ut+1 ) · A> (λt + η (Aut+1 + Bvt − c))≤(ii)f (u) − f (ut+1 ) + (u − ut+1 ) · A> λt+1 + ηA(u − ut+1 ) · B(vt − vt+1 ),(D.7)

where in (i) we have used the convexity of f , and in (ii) we used Eq. 6.8 for the λ-updates.
Similarly, we have the following from the variational inequality associated with the v-updatein Eq. 6.7, for every v ∈ V:0≤

∇ v L η ( u t +1 , v t +1 , λ t ) · ( v − v t +1 )=∇ g(vt+1 ) · (v − vt+1 ) + (v − vt+1 ) · B> (λt + η (Aut+1 + Bvt+1 − c))≤ ( i ) g ( v ) − g ( v t +1 ) + ( v − v t +1 ) · B > λ t +1 ,(D.8)

where in (i) we have used the convexity of g. Summing (D.7) and (D.8) we obtain, for everyw ∈ W,h ( w t +1 ) − h ( w ) + ( w t +1 − w ) · F ( w t +1 )≤ ηA(u − ut+1 ) · B(vt − vt+1 ) + η −1 (λ − λt+1 ) · (λt+1 − λt ).

(D.9)

We next rewrite the two terms in the right hand side. We haveηA(u − ut+1 ) · B(vt − vt+1 ) =+η
kAu + Bvt − ck2 − kAu + Bvt+1 − ck2
2

kAu

t +1

+ Bv

t +1

2

− ck − kAu

t +1

t

+ Bv − ck2(D.10)

andη −1 ( λ − λ t +1 ) · ( λ t +1 − λ t ) =1 
kλ − λt k2 − kλ − λt+1 k2 − kλt − λt+1 k2 .(D.11)
2η

Summing (D.9) over t and noting that η −1 kλt − λt+1 k2 = η kAut+1 + Bvt+1 − ck2 , we obtainby the telescoping sum property:
T −1 

∑h ( w t +1 ) − h ( w ) + ( w t +1 − w ) · F ( w t +1 )



η
2

2

t =0

≤

T −1

kAu + Bv − ck − kAu + Bv − ck − ∑ kAu
0

2

T

!
t +1

t

+ Bv − ck

2

t =0


1 
k λ − λ0 k2 − k λ − λ T k2
2ηη1
kAu + Bv0 − ck2 + kλk2 .
2
2η

+
≤

Now, using the convexity of h, we have that h(w̄ T ) = h(D.12)
T −1 t +1
1
T ∑ t =0 w



≤ T1 ∑tT=−01 h(wt+1 ).

Note also that, for every w̃, the function w 7→ (w − w̃) · F (w) is affine:(w − w̃) · F (w) = (u − ũ) · A> λ + (v − ṽ) · B> λ − (λ − λ̃) · (Au + Bv − c)= −ũ · A> λ − ṽ · B> λ + λ̃ · (Au + Bv − c) + λ · c= −(Aũ + Bṽ − c) · λ + λ̃ · (Au + Bv − c)
= F (w̃) · w − c · λ̃.

(D.13)As a consequence,
1 T −1 h(wt+1 ) + (wt+1 − w) · F (wt+1 ) ≥ h(w̄ T ) + (w̄ T − w) · F (w̄ T ),∑T t =0(D.14)

and from (D.12):h(w̄ T ) − h(w) + (w̄ T − w) · F (w̄ T ) ≤ C/T,

(D.15)

with C as in Eq. D.6. Note also that, since Λ is convex and for every t we have λt ∈ Λ, we
also have λ̄ T ∈ Λ.
Our next step is to reuse the bound stated in Proposition D.2 to derive a convergence rate
for the dual problem.
Proposition D.3 (Dual convergence rate.) Assume the conditions stated in Proposition D.2, with

w̄ T defined analogously. Let ϕ : Λ → R denote the dual objective function:ϕ(λ) := min L(u, v, λ),u∈U,v∈V(D.16)and let λ∗ = arg maxλ∈Λ ϕ(λ) be a dual solution. Then, after T iterations of ADMM, we achievean O( T1 )-accurate solution λ̄ T :ϕ(λ∗ ) −C≤ ϕ(λ̄T ) ≤ ϕ(λ∗ ),T(D.17)

where the constant C is given by
C =

≤



η
1
kAūT + Bv0 − ck2 + kAb
u T + Bv0 − ck2 +kλ∗ k2 + kλ̄T k2
2
2η


5η
5
max kAu + Bv0 − ck2 +k λ ∗ k2 .
2
2η
u ∈U

(D.18)

Proof. By applying Proposition D.2 to w = (ū T , v̄ T , λ) we obtain for arbitrary λ ∈ Λ:− (λ̄T − λ) · (AūT + Bv̄T − c) ≤ O(1/T ).

(D.19)

By applying Proposition D.2 to w = (u, v, λ̄ T ) we obtain for arbitrary u ∈ U and v ∈ V:
f (ū T ) + g(v̄ T ) + (Aū T + Bv̄ T − c) · λ̄ T≤

f (u) + g(v) + (Au + Bv − c) · λ̄ T + O(1/T ).

(D.20)

In particular, let ϕ(λ̄ T ) := minu∈U,v∈V L(u, v, λ̄ T ) = L(ubT , vb T , λ̄ T ) be the value of the dual
objective at λ̄ T , where (ubT , vb T ) are the corresponding minimizers. We then have:f (ū T ) + g(v̄ T ) + (Aū T + Bv̄ T − c) · λ̄ T ≤ ϕ(λ̄ T ) + O(1/T ).

(D.21)

Finally we have (letting w∗ = (u∗ , v∗ , λ∗ ) be the optimal primal-dual solution):f (u∗ ) + g(v∗ )==ϕ(λ∗ )
min f (u) + g(v) + λ∗ · (Au + Bv − c)u∈U,v∈V≤f (ū T ) + g(v̄ T ) + λ∗ · (Aū T + Bv̄ T − c)≤ (i )f (ū T ) + g(v̄ T ) + λ̄ T · (Aū T + Bv̄ T − c) + O(1/T )≤(ii)

ϕ(λ̄ T ) + O(1/T ),(D.22)

where in (i ) we have used Eq. D.19 and in (ii ) we have used Eq. D.21. Note that, since any
dual objective value is a lower bound, we also have ϕ(λ̄ T ) ≤ f (u∗ ) + g(v∗ ). Since we applied
Proposition D.2 twice, the constant hidden in the O-notation becomes
C=



1
η
kAūT + Bv0 − ck2 + kAb
u T + Bv0 − ck2 +kλ∗ k2 + kλ̄T k2 .
22η(D.23)

Even though C depends on ū T , ub T , and λ̄ T , it is easy to obtain an upper bound on C when U
is a bounded set, using the fact that the sequence (λt )t∈N is bounded by a constant, whichimplies that the average λ̄ T is also bounded. Indeed, from Boyd et al. (2011, p.107), we havethat
V t := η −1 kλ∗ − λt k2 + η kB(v∗ − vt )k2(D.24)
is a Lyapunov function, i.e., 0 ≤ V t+1 ≤ V t for every t ∈ N. This implies that V t ≤ V 0 =η −1 kλ∗ k2 + η kB(v∗ − v0 )k2 ; since V t ≥ η −1 kλ∗ − λt k2 , we can replace above and write:
0 ≥ kλ∗ − λt k2 − kλ∗ k2 − η 2 kB(v∗ − v0 )k2= kλt k2 − 2λ∗ · λt − η 2 kB(v∗ − v0 )k2≥ kλt k2 − 2kλ∗ kkλt k − η 2 kB(v∗ − v0 )k2 ,(D.25)

where in the last line we invoked the Cauchy-Schwarz inequality. By solving a quadratic
equation, it is easy to see that we must havet∗

kλ k ≤ kλ k +qkλ∗ k2 + η 2 kB(v0 − v∗ )k2 ,(D.26)

which in turn implies

kλt k2 ≤ 2kλ∗ k2 + η 2 kB(v0 − v∗ )k2 + 2kλ∗ kq

kλ∗ k2 + η 2 kB(v0 − v∗ )k2≤ 2kλ∗ k2 + η 2 kB(v0 − v∗ )k2 + 2(kλ∗ k2 + η 2 kB(v0 − v∗ )k2 )= 4kλ∗ k2 + 3η 2 kB(v0 − v∗ )k2= 4kλ∗ k2 + 3η 2 kAu∗ + Bv0 − ck2 ,(D.27)

where the last line follows from the fact that Au∗ + Bv∗ = c. Replacing (D.27) in (D.23)
yields Eq. D.18.
We are finally ready to see how the bounds above apply to the AD3 algorithm, and inparticular, how the constants in the bound relate to the structure of the graphical model.

Proposition D.4 (Dual convergence rate of AD3 .) After T iterations of AD3 , we achieve an O( T1 )accurate solution λ̄ T := ∑tT=−01 λ(t) :C≤ ϕ(λ̄T ) ≤ ϕ(λ∗ ),T(D.28)5η5deg(i )(1 − |Yi |−1 ) +k λ ∗ k2 .
∑
2 i
2η

(D.29)ϕ(λ∗ ) −where the constant C is given by
C =Hence, as expected, the constant in the bound increases with the number of overlapping variables, thesize of the sets Yi , and the magnitude of the optimal dual vector λ∗ .
Proof. With the uniform initialization of the ζ-variables in AD3 , the first term in the second line of Eq. D.18 is maximized by a choice of µ-variables that puts all mass in a single

configuration, for each factor α ∈ F ∪ H. That is, we have for each i ∈ N(α):
maxkµiα − |Yi |−1 1k2 =αµi(1 − |Yi |−1 )2 + (|Yi | − 1)|Yi |−2= 1 − | Yi | − 1 .

(D.30)

Hence, the constant C becomes:
C =

=

5η
5
max
kµiα − |Yi |−1 1k2 + kλ∗ k2∑∑
α2 α i µi2η5η
5deg(i )(1 − |Yi |−1 ) +k λ ∗ k2 .
∑
2 i
2η

(D.31)A consequence of Proposition D.4 is that, after T = O(1/e) iterations of AD3 , we have adual solution which yields an objective value e-close to the optimal value. Asymptotically,
this is a better bound than the O(1/e2 ) iteration bound of the subgradient-based dual decomposition algorithm of Komodakis et al. (2007), and is equivalent to the O(1/e) boundachieved by the accelerated method of Jojic et al. (2010), despite the fact that the latter requires specifying e beforehand to set a “temperature” parameter. The bounds derived so far
for all these algorithms are with respect to the dual problem—an open problem is to obtainbounds related to primal convergence.

Appendix E

Derivation of Solutions for Quadratic
Problems in AD3
E.1

Binary pairwise factors

In this section, we derive in detail the closed form solution of problem Eq. 6.26 for binary
pairwise factors. Recall that, if α is a pairwise factor (i.e., if N (α) = 2), then the marginal
polytope MARG(G |α ) is given by:

MARG(G |α ) =









∑ y 1 ∈ Y1 µ 1 ( y 1 ) = 1∑ y 2 ∈ Y2 µ 2 ( y 2 ) = 1µ1 (y1 ) = ∑y2 ∈Y2 µ12 (y1 , y2 ),µ2 (y2 ) = ∑y1 ∈Y1 µ12 (y1 , y2 ),µ12 (y1 , y2 ) ≥ 0,(µ1 (.), µ2 (.), µ12 (.))
















.
∀ y 1 ∈ Y1


∀ y 2 ∈ Y2

∀ y 1 ∈ Y1 , y 2 ∈ Y1

(E.1)

If besides being pairwise, factor α is also binary (i.e., if Y1 = Y2 = {0, 1}), then we may
reparameterize our problem by introducing new variables1 z1 := µ1 (1), z2 := µ2 (1), andz12 := µ12 (1, 1). Noting that µ1 = (1 − z1 , z1 ), µ2 = (1 − z2 , z2 ), and µ12 = (1 − z1 − z2 +z12 , z1 − z12 , z2 − z12 , z12 ), the problem in Eq. 6.26 becomes:
minimize

1
2

"

ω (0)
1 − z1 − 1
η

2



ω (1)
+ z1 − 1
η

2



ω (0)
+ 1 − z2 − 2
η

2



ω (1)
+ z2 − 2η2 #−η −1 [θ12 (00)(1 − z1 − z2 + z12 ) + θ12 (10)(z1 − z12 ) + θ12 (01)(z2 − z12 ) + θ12 (11)z12 ]w.r.t.
s.t.

z1 , z2 , z12 ∈ [0, 1]3
z12 ≤ z1 ,z12 ≤ z2 ,z12 ≥ z1 + z2 − 1,

(z1 , z2 , z12 ).

(E.2)

or, multiplying the objective by the constant 12 :minimizew.r.t.
s.t.

11(z1 − c1 )2 + (z2 − c2 )2 − c12 z12
22
z1 , z2 , z12 ∈ [0, 1]3
z12 ≤ z1 ,z12 ≤ z2 ,z12 ≥ z1 + z2 − 1,

1 Cf. Example 5.1 and Eq. 5.14, where the same reparametrization was made.

257

(E.3)

where we have substituted
c1 = (η −1 ω1 (1) + 1 − η −1 ω1 (0) − η −1 θ12 (00) + η −1 θ12 (10))/2
c2 = ( η−1ω2 ( 1 ) + 1 − η−1ω2 ( 0 ) − η−1θ12 (00) + η

−1

(E.4)

θ12 (01))/2(E.5)

c12 = (η −1 θ12 (00) − η −1 θ12 (10) − η −1 θ12 (01) + η −1 θ12 (11))/2.

(E.6)

Now, notice that in (E.3) we can assume c12 ≥ 0 without loss of generality—indeed, if
0 = −c , z0 = 1 − z ,c12 < 0, we recover this case by redefining c10 = c1 + c12 , c20 = 1 − c2 , c122
12 2
0 = z − z . Thus, assuming that c
≥
0,
the
lower
bound
constraints
z
≥
z
+
z
−
1
z12
2
1
12
12
12
1
and z12 ≥ 0 are always inactive and can be ignored. Hence, (E.3) can be simplified to:minimizew.r.t.
s.t.

11(z1 − c1 )2 + (z2 − c2 )2 − c12 z12
22
z1 , z2 , z12z12 ≤ z1 ,z12 ≤ z2 ,z1 ∈ [0, 1],z2 ∈ [0, 1].

(E.7)

We next establish a closed form solution for this problem, proving Proposition 6.5. Let[ x ]U := min{max{ x, 0}, 1} denote projection (clipping) onto the unit interval U := [0, 1].
Note that if c12 = 0, the problem becomes separable, and a solution isz1∗ = [c1 ]U ,z2∗ = [c2 ]U ,∗z12
= min{z1∗ , z2∗ },(E.8)

which complies with Eq. 6.33. We next analyze the case where c12 > 0. The Lagrangianfunction of (E.7) is:L(z, µ, λ, ν) =11
(z1 − c1 )2 + (z2 − c2 )2 − c12 z12 + µ1 (z12 − z1 ) + µ2 (z12 − z2 )22
−λ1 z1 − λ2 z2 + ν1 (z1 − 1) + ν2 (z2 − 1).
(E.9)

At optimality, the following KKT conditions need to be satisfied:∇z1 L(z∗ , µ∗ , λ∗ , ν∗ ) = 0 ⇒ z1∗ = c1 + µ1∗ + λ1∗ − ν1∗∗∇z2 L(z∗ , µ∗ , λ , ν∗ ) = 0 ⇒ z2∗ = c2 + µ2∗ + λ2∗ − ν2∗
∗

∗

∗

∗

∇z12 L(z , µ , λ , ν ) = 0 ⇒λ1∗ z1∗ = 0
λ2∗ z2∗ = 0∗µ1∗ (z12− z1∗ ) = 0∗µ2∗ (z12− z2∗ ) = 0
ν1∗ (z1∗ − 1) = 0
ν2∗ (z2∗ − 1) = 0∗ ∗ ∗µ ,λ ,ν ≥ 0∗
z12≤ z1∗ ,∗z12
≤ z2∗ ,

z1∗ ∈ [0, 1],z2∗ ∈ [0, 1]

We are going to consider three cases separately:
1. z1∗ > z2∗

c12 = µ1∗ + µ2∗(E.10)(E.11)(E.12)(E.13)(E.14)(E.15)(E.16)(E.17)(E.18)(E.19)(E.20)

∗ < z∗ .
From the primal feasibility conditions (E.20), this implies z1∗ > 0, z2∗ < 1, and z12
1Complementary slackness (E.13,E.18,E.15) implies in turn λ1∗ = 0, ν2∗ = 0, and µ1∗ = 0.
From (E.12) we have µ2∗ = c12 . Since we are assuming c12 > 0, we then have µ2∗ > 0,
∗ = z∗ .
and complementary slackness (E.16) implies z122

Plugging the above into (E.10)–(E.11) we obtainz1∗ = c1 − ν1∗ ≤ c1 ,z2∗ = c2 + λ2∗ + c12 ≥ c2 + c12 .

(E.21)

Now we have the following:• Either z1∗ = 1 or z1∗ < 1. In the latter case, ν1∗ = 0 by complementary slackness(E.17), hence z1∗ = c1 . Since in any case we must have z1∗ ≤ c1 , we conclude thatz1∗ = min{c1 , 1}.
• Either z2∗ = 0 or z2∗ > 0. In the latter case, λ2∗ = 0 by complementary slackness(E.14), hence z2∗ = c2 + c12 . Since in any case we must have z2∗ ≥ λ2 , we conclude
that z2∗ = max{0, c2 + c12 }.
In sum:z1∗ = min{c1 , 1},∗z12= z2∗ = max{0, c2 + c12 },(E.22)and our assumption z1∗ > z2∗ can only be valid if c1 > c2 + c12 .
2. z1∗ < z2∗
By symmetry, we havez2∗ = min{c2 , 1},∗z12= z1∗ = max{0, c1 + c12 },(E.23)and our assumption z1∗ < z2∗ can only be valid if c2 > c1 + c12 .
3. z1∗ = z2∗∗ = z∗ = z∗ , and we can rewrite ourIn this case, it is easy to verify that we must have z1221
optimization problem in terms of one variable only (call it z). The problem becomesthat of minimizing 12 (z − c1 )2 + 21 (z − c2 )2 − c12 z, which equals a constant plus (z −c1 +c2 +c12 2) , subject to z ∈ U , [0, 1]. Hence:2∗
z12= z1∗ = z2∗ =

c1 + c2 + c122

.

(E.24)

U

Putting all the pieces together, we obtain the solution displayed in Eq. 6.33.

E.2

Hard constraint factors

E.2.1

Sifting Lemma

Some of the procedures that we will derive in the sequel make use of a simple sifting technique, in which a projection is computed by evaluating one or more projections onto larger
sets. The following lemma is the key to those procedures.

Algorithm 18 Dykstra’s algorithm for projecting onto

TJ

j =1 C j

Input: Point x0 ∈ RD , convex sets C1 , . . . , C J(0)Initialize x(0) = x0 , u j = 0 for all j = 1, . . . , Jt←1
repeatfor j = 1 to J do
Set s = j + (t − 1) J
( t −1)
Set x̃0 = x(s−1) − u j(t)

Set x(s) = projCj ( x̃0 ), and u j = x(s) − x̃0
end fort ← t+1
until convergence.
Output: x
Lemma E.1 (Sifting Lemma.) Consider a problem of the formP:

min f ( x)
x ∈X

s.t.

g( x) ≤ 0,

(E.25)

where X is nonempty convex subset of RD and f : X → R and g : X → R are convex functions.
Suppose that the problem (E.25) is feasible and bounded below, and let A be the set of solutions of therelaxed problem minx∈X f ( x), i.e. A = { x ∈ X | f ( x) ≤ f ( x0 ), ∀ x0 ∈ X}. Then:
1. if for some x̃ ∈ A we have g( x̃) ≤ 0, then x̃ is also a solution of the original problem P;
2. otherwise (if for all x̃ ∈ A we have g( x̃) > 0), then the inequality constraint is necessarily
active in P, i.e., problem P is equivalent to minx∈X f ( x) s.t. g( x) = 0.
Proof. Let f ∗ be the optimal value of P. The first statement is obvious: since x̃ is a solutionof a relaxed problem we have f ( x̃) ≤ f ∗ ; hence if x̃ is feasible this becomes an equality. For
the second statement, assume that ∃ x ∈ X s.t. g( x) < 0 (otherwise, the statement holds
trivially). The nonlinear Farkas’ lemma (Bertsekas et al., 2003, Prop. 3.5.4, p. 204) implies
that there exists some λ∗ ≥ 0 s.t. f ( x) − f ∗ + λ∗ g( x) ≥ 0 holds for all x ∈ X. In particular,
this also holds for an optimal x∗ (i.e., such that f ∗ = f ( x∗ )), which implies that λ∗ g( x∗ ) ≥ 0.
However, since λ∗ ≥ 0 and g( x∗ ) ≤ 0 (since x∗ has to be feasible), we also have λ∗ g( x∗ ) ≤ 0,i.e., λ∗ g( x∗ ) = 0. Now suppose that λ∗ = 0. Then we have f ( x) − f ∗ ≥ 0, ∀ x ∈ X, which
implies that x∗ ∈ A and contradicts the assumption that g( x̃) > 0, ∀ x̃ ∈ A. Hence we musthave g( x∗ ) = 0.

E.2.2

OR-with-output Factor

We have presented an algorithm for solving the local subproblem for the OR-with-output
factor which requires, as an intermediate step, projecting onto the set A0 . For the last projection, we suggested Procedure 6.1. We now prove its correctness.
Proposition E.2 Procedure 6.1 is correct.
Proof. The proof is divided into the following parts:

1. We show that Procedure 6.1 corresponds to the first iteration of Dykstra’s projection
algorithm (Boyle and Dykstra, 1986) applied to sets A1 and [0, 1]K+1 ;2. We show that Dykstra’s converges in one iteration if a specific condition is met;
3. We show that with the two sets above that condition is met.
The first part is rather trivial. Dykstra’s algorithm is shown as Algorithm 18; when J = 2,
(1)
(1)
C1 = A1 and C2 = [0, 1]K+1 , and noting that u1 = u2 = 0, we have that the first iteration
becomes Procedure 6.1.
We turn to the second part, to show that, when J = 2, the fact that x(3) = x(2) implies that(s)
x = x(2) , ∀s > 3. In words, if at the second iteration t of Dijkstra’s, the value of x does notchange after computing the first projection, then it will never change, so the algorithm hasconverged and x is the desired projection. To see that, consider the moment in Algorithm 18(2)(1)
when t = 2 and j = 1. After the projection, we update u1 = x(3) − ( x(2) − u1 ), which(1)

when x(3) = x(2) equals u1 , i.e., u1 keeps unchanged. Then, when t = 2 and j = 2, one first(1)

computes x̃0 = x(3) − u2 = x(3) − ( x(2) − x0 ) = x0 , i.e., the projection is the same as the one
already computed at t = 1, j = 2. Hence the result is the same, i.e., x(4) = x(2) , and similarly(2)(1)
u2 = u2 . Since neither x, u1 and u2 changed in the second iteration, and subsequent
iterations only depend on these values, we have that x will never change afterwards.
Finally, we are going to see that, regardless of the choice of z0 in Procedure 6.1 (x0 in
Algorithm 18) we always have x(3) = x(2) . Looking at Algorithm 10, we see that after t = 1:(
τ − x0k , if k = K + 1 or x0k ≥ τ
τ,
if
k
=
K
+
1
or
x
≥
τ
(1)
(1)
0k
u1k =
xk =
0,
otherwise,
x0k , otherwise,([ τ ]U ,if k = K + 1 or x0k ≥ τ(1)(2)(E.26)
x k = [ x k ]U =[ x0k ]U , otherwise.
(

Hence in the beginning of the second iteration (t = 2, j = 1), we have
(
(2)
(1)
x̃0k = xk − u1k =[τ ]U − τ + x0k , if k = K + 1 or x0k ≥ τotherwise.
[ x0k ]U ,(E.27)

Now two things should be noted about Algorithm 10:• If a constant is added to all entries in z0 , the set I(zK+1 ) remains the same, and τ andz are affected by the same constant;
0 = z if k = K + 1 or z ≥ τ, and z0 ≤ τ otherwise. Let z0 be• Let z00 be such that z0k0k0k0k
the projected point when such z00 is given as input. Then I(z0K+1 ) = I(zK+1 ), τ 0 = τ,0 otherwise.
z0k = zk if k = K + 1 or z0k ≥ τ, and z0k = z0k

The two facts above allow to relate the projection of x̃0 (in the second iteration) with that of
x0 (in the first iteration). Using [τ ]U − τ as the constant, and noting that, for k 6= K + 1 andx0k < τ, we have [ x0k ]U − [τ ]U + τ ≥ τ if x0k < τ, the two facts imply that:
(
(3)
xk =

(1)

xk + [τ ]U − τ = [τ ]U , if k = K + 1 or x0k ≥ τotherwise;[ x0k ]U ,(E.28)

hence x(3) = x(2) , which concludes the proof.

E.3 Proof of Proposition 6.6
We first show that the rank of the matrix M is at most ∑i∈N(α) |Yi | − N(α) + 1. For eachi ∈ N(α), let us consider the |Yi | rows of M. By definition of M, the set of entries on theserows which have the value 1 form a partition of Yα , hence, summing these rows yields the allones row vector, and this happens for each i ∈ N(α). Hence we have at least N(α) − 1 rowsthat are linearly dependent. This shows that the rank of M is at most ∑i∈N(α) |Yi | − N(α) + 1.
Let us now rewrite (6.44) as
minimizewith respect to1
k u − a k2 + g ( u )2u ∈ R ∑ i | Yi | ,(E.29)

where g(u) is the solution value of the following linear problem:
minimizewith respect tosubject to− b> v(E.30)

|Yα |

v∈R


 Mv = u
1> v = 1 v ≥ 0.

From the simplex constraints (last two lines), we have that problem (E.30) is bounded below(i.e., g(u) > −∞). Furthermore, problem (E.30) is feasible (i.e., g(u) < +∞) if and only if usatisfies the non-negativity and normalization constraints for every i ∈ N(α):∑ ui (yi ) = 1,

ui (yi ) ≥ 0, ∀yi .

(E.31)

yi

Those constraints imply 1> v = 1. Hence we can add the constraints (E.31) to the problemin (E.29), discard the constraint 1> v = 1 in (E.30), and assume that the resulting problem(which we reproduce below) is feasible and bounded below:
minimizewith respect tosubject to− b> v(E.32)v ∈ R |Yα |(Mv = uv ≥ 0.

Problem (E.32) is a linear program in standard form. Since it is feasible and bounded, it
admits a solution at a vertex of the constraint set. We have that a point v
b is a vertex ifand only if the columns of M indexed by {yα | vα (yα ) 6= 0} are linearly independent. We
cannot have more than ∑i∈N(α) |Yi | − N(α) + 1 of these columns, since this is the rank of M.
It follows that (E.32) (and hence (6.44)) has a solution v∗ with at most ∑i∈N(α) |Yi | − N(α) + 1
nonzeros.

Appendix F

Proofs for Online Proximal Gradient
Algorithms
F.1 Proof of Proposition 9.1
Let H = H1 ⊕ · · · ⊕ H M . We have respectively:
1
M ϕ ( x1 , . . . , x M ) = min ky − xk2H + ϕ(y)
y 2

=
=
=
=

1 M
∑ kyk − xk k2Hk + ψ(ky1 kH1 , . . . , ky M kHM )y1 ,...,y M 2
k =1
min

min

M
u ∈R+

min

M
u ∈R+

min

M
u ∈R+

!
1 M
ψ ( u1 , . . . , u M ) + ∑minky − xk k2Hk
2 k =1 y k : k y k k Hk = u k k
!
2
uk yk
1 M
ψ(u1 , . . . , u M ) + ∑ min
− xk
(∗ )
2 k =1 y k k y k k Hk
Hk!
M1
ψ ( u 1 , . . . , u M ) + ∑ ( u k − k x k k Hk ) 2
2 k =1= Mψ (k x1 kH1 , . . . , k x M kH M ),(F.1)

where the solution of the innermost minimization problem in (∗ ) is yk = kux k k xk . This can bekverified by noting that
2
uk yk
− xk
k y k k Hk
H

= u2k + k xk k2Hk −k2ukhx , y ik y k k Hk k k Hk

≤ u2k + k xk k2Hk − 2uk k xk kHk= ( u k − k x k k Hk ) 2 ,(F.2)

due to Cauchy-Schwarz inequality, with equality iff yk = uk xk /k xk k. Therefore:[prox ϕ ( x1 , . . . , x M )]k = [proxψ (k x1 kH1 , . . . , k x M kH M )]k
263

xk
.
k x k k Hk

(F.3)

F.2 Proof of Corollary 9.3
Let x̌ = prox ϕ∗ ( x). From (9.14)–(9.15), we have that1k x k2 =
2

=
=
≥
=

1
1
k x̄ − xk2 + ϕ( x̄) + k x̌ − xk2 + ϕ∗ ( x̌)22
11
k x̄ − xk2 + ϕ( x̄) + k x̄k2 + ϕ∗ ( x − x̄)22
11
k x̄ − xk2 + ϕ( x̄) + k x̄k2 + sup (hu, x − x̄i − ϕ(u))
2
2
u ∈H
1
1
k x̄ − xk2 + ϕ( x̄) + k x̄k2 + hy, x − x̄i − ϕ(y)221
k xk2 + h x̄ − y, x̄ − xi − ϕ(y) + ϕ( x̄),
2

(F.4)

whenceh x̄ − y, x̄ − xi ≤ ϕ(y) − ϕ( x̄).

(F.5)

Now note that the left hand side of (F.5) can be written as:

h x̄ − y, x̄ − xi =111
ky − x̄k2 − ky − xk2 + k x̄ − xk2 .
2
22This concludes the proof.

F.3 Proof of Proposition 9.4
q0
We first show that, when H = R M , the family Rnorms= {k.k p | p, q ≥ 1} is strongly coq
shrinking. We know that, for any σ > 0 and p, q ≥ 1, the σk.k p -proximity operator shrinks
each component, i.e., given any b = (b1 , . . . , bm ) ∈ R M , we have |bm | ≥ |[proxσk.kqp (b)]m | form = 1, . . . , M. Let p0 , q0 ≥ 1. We use the fact that the function x 7→ x α is increasing in R+ forany α > 0. Summing the M inequalities above, after raising each of them to the power of p0 ,
M
M
p0
p0
0
0
we obtain ∑m=1 | bm | ≥ ∑m=1 |[proxσk.kqp ( b )]m | . Raising both sides to the power of q /p
q0

q0

(which is > 0) and scaling by σ0 > 0, we obtain σ0 kbk p0 ≥ σ0 kproxσk.kqp (b)k p0 , which proves
the desired fact.
Now, invoking Proposition 9.1, we can generalize the above result for the structured blockqq00
case, i.e., for Rnorms= {k.k2,p | p, q ≥ 1}. We also have that the σk.k2,p -proximity operator
shrinks each block norm, i.e., given any w ∈ H, we have kwm k ≥ k[proxσk.kq (w)]m k for m =2,p1, . . . , M (here the norms are those induced by each RKHS Hm ). Hence, we can anagously
00derive the fact that Rnorms
is co-shrinking.
00
Finally, we consider the broader case of the family Rnorms ⊇ Rnorms, which involves
the subsets B ⊆ {1, . . . , M }. Clearly, the proximity operator associated with the functionq
w 7→ σkwB k2,p is still shrinking for any B: it can be easily verified that [proxσk.B kq (w)]m =2,p[proxσk.kq (w)]m if m ∈ B, and [proxσk.B kq (w)]m = wm otherwise. Letting B0 ⊆ {1, . . . , M}
2,p

2,p

0

0

and summing |B0 | inequalities, we obtain ∑m∈B0 kwm k p ≥ ∑m∈B0 k[proxσk.G kq (w)]m k p ,2,pwhich implies analagously that Rnorms also has the strong co-shrinkage property.

F.4 Proof of Proposition 9.5
Let w̄1 , . . . , w̄ J the sequence of points obtained after applying each proximity operator, i.e.,w̄ j = proxΩ j (w̄ j−1 ) for j = 1, . . . , J, where we define w̄0 , w and w̄ J , w̄. From Corollary 9.3, we havekξ − w̄ j k2 kξ − w̄ j−1 k2 kw̄ j−1 − w̄ j k2
−
+
2
2
2

≤ Ω j (ξ ) − Ω j (w̄ j ).

(F.6)

Summing these inequalities for j = 1, . . . , J and applying the telescopic property, we obtainJkw̄ j−1 − w̄ j k2kξ − w̄k2 kξ − wk2
−
+∑
2
2
2
j =1

J

≤ Ω(ξ ) − ∑ Ω j (w̄ j )j =1≤ Ω(ξ ) − Ω(w̄),(F.7)

where the last inequality follows from the co-shrinkage property of Ω1 , . . . , Ω J . Finally, fromthe convexity of the norm, we have that
J J 1
kw̄ j−1 − w̄ j k2 ≥
2 j∑
J
=1

J
2

J

1
∑ J (w̄ j−1 − w̄ j )
j =1

2

=

1
kw − w̄k2 ,
2J

(F.8)

which concludes the proof.

F.5 Proof of Lemma 9.6
Let u(w,b w) , Ω(wb ) − Ω(w). We have successively:kw
b − w t +1 k 2

≤(i)

kw
b − w̃t+1 k2

≤(ii)

kw
b − w̃t k2 + 2ηt u(w,b w̃t+1 )≤(iii) kw
b − w̃t k2 + 2ηt u(w,b w t +1 )=kw
b − wt k2 + kwt − w̃t k2 + 2(wb − wt )> (wt − w̃t ) + 2ηt u(w,b w t +1 )=kw
b − wt k2 + ηt2 k g k2 + 2ηt (w̄ − wt )> g + 2ηt u(w,b w t +1 )≤(iv) kw
b − wt k2 + ηt2 k g k2 + 2ηt ( L(wb ) − L(wt )) + 2ηt u(w,b w t +1 )≤kw
b − wt k2 + ηt2 G2 + 2ηt ( L(wb ) − L(wt )) + 2ηt u(w,b w t +1 ),(F.9)

where the inequality (i) is due to the nonexpansiveness of the projection operator, (ii) follows
from Proposition 9.5, (iii) results from the fact that Ω(w̃t+1 ) ≥ Ω(ΠW (w̃t+1 )), and (iv) resultsfrom the subgradient inequality of convex functions, which has an extra term σ2 kw
b − wt k2 if
L is σ-strongly convex.

F.6 Proof of Proposition 9.7
Invoking Lemma 9.6 and summing for t = 1, . . . , T gives
T

RegT =

∑



t =1
T 



T 
L(wt ; x n(t) , yn(t) ) + λΩ(wt ) − ∑ L(w∗ ; x n(t) , yn(t) ) + λΩ(w∗ )
t =1


T 
≤(i) ∑ L(wt ; x n(t) , yn(t) ) + λΩ(wt+1 ) − ∑ L(w∗ ; x n(t) , yn(t) ) + λΩ(w∗ )


t =1

≤

G2 T

2 t∑
=1

T

ηt + ∑

t =1
∗
t
2
∗
k w − w k − k w − w t +1 k 2

t =1
T 

=

G2 T
1
ηt + ∑
2 t∑
2
t =2
=1

2ηt

1
1
k w ∗ − w 1 k 2 k w ∗ − w t +1 k 2−
k w ∗ − w t k2 +−.
ηtη t −1
2η1
2ηT(F.10)

where the inequality (i) is due to the fact that w1 = 0, whenceT∑ L ( wt ; x n(t) , yn(t) ) + Ω ( wt )



T

=

t =1

≤





∑ L(wt ; xn(t) , yn(t) ) + Ω(wt+1 ) − (Ω(wt+1 ) − Ω(w1 ))

t =1
T 

∑


L ( w t ; x n ( t ) , y n ( t ) ) + Ω ( w t +1 ) .

(F.11)

t =1

Noting that the second term vanishes for a constant learning rate and that the last term is
non-positive suffices to prove the first part. For the second part, we continue as:
RegT

F2 T
G2 T
ηt +
∑
2 t =1
2 t∑
=2

≤
≤(ii)


F2
1
1
F2
G2 T
ηt +
−
+
=
∑
ηt
η t −12η1
2 t =1
2ηT
√


√
F2 T
F2 √
G2 η0 ( T − 1/2) +T,≤ G 2 η0 +
2η0
2η0


(F.12)√
where equality (ii) is due to the fact that ∑tT=1 √1t ≤ 2 T − 1. For the third part, continue
after inequality (i) as:T
G2 T
RegT ≤
ηt + ∑∑
2 t =1
t =2

=



1
1
−
−σ
ηt
η t −1

k w ∗ − w t k2
+
2



1
−σ
η1



k w ∗ − w 1 k 2 k w ∗ − w t +1 k 2−
22ηT

G2 T 1
G2
G2 T 1 σT∗
t +1 2
−
·
k
w
−
w
k
≤
≤(iii)(1 + log T ),∑∑2σ t=1 t
2
2σ t=1 t
2σ(F.13)

where the equality (iii) is due to the fact that ∑tT=1 1t ≤ 1 + log T.

F.7 Lipschitz Constants of Some Loss Functions
Let w∗ be a solution of the problem (9.17) with W = H. For certain loss functions, we may
obtain bounds of the form kw∗ k ≤ γ for some γ > 0, as the next proposition illustrates.
Therefore, we may redefine W = {w ∈ H | kwk ≤ γ} (a vacuous constraint) without
affecting the solution of (9.17).

M
2
Proposition F.1 Let Ω(w) = λ2 (∑m=1 k wm k) . Let LSSVM and LCRF be the structured hinge andlogistic losses (9.5). Assume that the average cost function (in the SVM case) or the average entropy(in the CRF case) are bounded by some Λ ≥ 0, i.e.,1

1 N
max ρ(y0 ; yi ) ≤ Λ
0 ∈Y( x i )
N i∑
y
=1

or

1 N
H (Y i ) ≤ Λ.
N i∑
=1

(F.14)

Then:
1. The solution of (9.17) with W = H satisfies kw∗ k ≤

√

2Λ/λ.

2. L is G-Lipschitz on H, with G = 2 maxu∈U k f (u)k.
3. Consider the following problem obtained from (9.17) by adding a quadratic term:σ
1 N
min kwk2 + Ω(w) +L(w; xi , yi ).
w 2
N i∑
=1
The solution of this problem satisfies kw∗ k ≤(F.15)p2Λ/(λ + σ).
nop
4. The modified loss L̃ = L + σ2 k.k2 is G̃-Lipschitz on w | kwk ≤ 2Λ/(λ + σ) , wherepG̃ = G + 2σ2 Λ/(λ + σ).
Proof. Let FSSVM (w) and FCRF (w) be the objectives of (9.17) for the SVM and CRF cases. We
have
FSVM (0) = Ω(0) +1 N
1 N
LSSVM (0; xi , yi ) =max ρ(y0 ; yi ) ≤ ΛSSVM∑
0i
N i =1N i∑=1 y ∈Y( x )(F.16)FCRF (0) = Ω(0) +

1 N
1 N
i i
L
(
0;
x
,
y
)
=
CRF
∑ log |Y(xi )| ≤ ΛCRF
N i∑
N
=1
i =1

(F.17)

Using the facts that F (w∗ ) ≤ F (0), that the losses are non-negative, and that (∑i | ai |)2 ≥∑i a2i , we obtain λ2 kw∗ k2 ≤ Ω(w∗ ) ≤ F (w∗ ) ≤ F (0), which proves the first statement.
To prove the second statement for the SVM case, note that a subgradient of LSSVM at w isg SSVM = f ( x, yb) − f ( x, y), where yb = arg maxy0 ∈Y(x) w> ( f ( x, y0 ) − f ( x, y)) + ρ(y0 ; y); and thatthe gradient of LCRF at w is g CRF = Ew f ( x, Y ) − f ( x, y). Applying Jensen’s inequality, we
have that k g CRF k ≤ Ew k f ( x, Y ) − f ( x, y)k. Therefore, both k g SSVM k and k g CRF k are upperbounded by maxx∈X,y,y0 ∈Y(x) k f ( x, y0 ) − f ( x, y)k ≤ 2 maxu∈U k f (u)k.
The same rationale can be used to prove the third and fourth statements.

F.8 Computing the proximity operator of the squared L1
We present an algorithm (Algorithm 19) that computes the Moreau projection of the squared,
weighted L1 -norm. Denote by the Hadamard product, [ a b]k = ak bk . Letting λ ≥ 0, d ≥ 0,
1 In sequence binary labeling, we have Λ = P̄ for the CRF case and for the SVM case with a Hamming costfunction, where P̄ is the average sequence length. Observe that the entropy of a distribution over labelings of asequence of length P is upper bounded by log 2P = P.

Algorithm 19 Moreau projection for the squared weighted L1 -norm
Input: A vector x0 ∈ R M , a weight vector d ≥ 0, and a parameter λ > 0
Set u0m = | x0m |/dm and am = d2m for each m = 1, . . . , M
Sort u0 : u0(1) ≥ . . . ≥ u0( M)jλFind ρ = max j ∈ {1, . . . , M} | u0( j) −∑ r =1 a (r ) u 0(r ) > 0j

1 + λ ∑ r =1 a (r )ρCompute u = soft(u0 , τ ), where τ = 1+λ λρ a ∑r=1 a(r) u0(r)
∑
(r )
r =1

Output: x s.t. xr = sign( x0r )dr ur .
and φd ( x) , 12 kd

xk21 , the underlying optimization problem is:1
λMλφd ( x0 ) , min k x − x0 k2 +M
2x ∈R 2M∑ dm | xm |!2.

(F.18)

m =1

This includes the squared L1 -norm as a particular case, when d = 1 (the case addressed in
Algorithm 15). The proof is somewhat technical and follows the same procedure employedby Duchi et al. (2008) to derive an algorithm for projecting onto the L1 -ball. The runtime isO( M log M) (the amount of time that is necessary to sort the vector), but a trick similar tothe one described by (Duchi et al., 2008) can be employed to yield O( M ) runtime.
Lemma F.2 Let x∗ = proxλφd ( x0 ) be the solution of (F.18). Then:
1. x∗ agrees in sign with x0 , i.e., each component satisfies x0i · xi∗ ≥ 0.
2. Let σ ∈ {−1, 1} M . Then proxλφd (σ x0 ) = σ
produces a x∗ with the same sign flipped.

proxλφd ( x0 ), i.e., flipping a sign in x0

Proof. Suppose that x0i · xi∗ < 0 for some i. Then, x defined by x j = x ∗j for j 6= i and xi = − xi∗achieves a lower objective value than x∗ , since φd (x) = φd ( x∗ ) and ( xi − x0i )2 < ( xi∗ − x0i )2 ;this contradicts the optimality of x∗ . The second statement is a simple consequence of thefirst one and that φd,λ (σ x) = φd,λ (σ x∗ ).
Lemma F.2 enables reducing the problem to the non-negative orthant, by writing x0 = σ ·x̃0 , with x̃0 ≥ 0, obtaining a solution x̃∗ and then recovering the true solution as x∗ = σ · x̃∗ . It
therefore suffices to solve (F.18) with the constraint x ≥ 0, which in turn can be transformed
into:
!2
M
1 M
λ
2
am um ,min F (u) , ∑ am (um − u0m ) +(F.19)u≥0
2 m =12 m∑=1where we made the change of variables am , d2m , u0m , x0m /dm and um , xm /dm .

2
M
M
2+ λ
The Lagrangian of (F.19) is L(u, ξ ) = 12 ∑m
a
(
u
−
u
)
a
u
− ξ > u,0m
=1 m m
2 ∑ m =1 m m
where ξ ≥ 0 are Lagrange multipliers. Equating the gradient (w.r.t. u) to zero givesMa(u − u0 ) + λ ∑ am um a − ξ = 0.
m =1(F.20)

From the complementary slackness condition, u j > 0 implies ξ j = 0, which in turn impliesM

a j (u j − u0j ) + λa j ∑ am um = 0.

(F.21)m =1

M
Thus, if u j > 0, the solution is of the form u j = u0j − τ, with τ = λ ∑m
=1 am um . The next
lemma shows the existence of a split point below which some coordinates vanish.

Lemma F.3 Let u∗ be the solution of (F.19). If u∗k = 0 and u0j < u0k , then we must have u∗j = 0.
Proof. Suppose that u∗j = e > 0. We will construct a ũ whose objective value is lower
than F (u∗ ), which contradicts the optimality of u∗ : set ũl = u∗l for l ∈/ { j, k }, ũk = ec,MM∗
and ũ j = e 1 − cak /a j , where c = min{ a j /ak , 1}. We have ∑m=1 am um = ∑m
=1 am ũm , andtherefore2( F (ũ) − F (u∗ )) =MM

∑ am (ũm − u0m )2 − ∑ am (u∗m − u0m )2

m =1= a j (ũ j − u0j )

2

m =1
∗
− a j (u j − u0j )2 + ak (ũk − u0k )2 − ak (u∗k − u0k )2 .

(F.22)
Consider the following two cases: (i) if a j ≤ ak , then ũk = ea j /ak and ũ j = 0. Substituting
in (F.22), we obtain 2( F (ũ) − F (u∗ )) = e2 a2j /ak − a j ≤ 0, which leads to the contradictionF (ũ) ≤ F (u∗ ). If (ii) a j > ak , then ũk = e and ũ j = e 1 − ak /a j . Substituting in (F.22),
2
we obtain 2( F (ũ) − F (u∗ )) = a j e2 1 − ak /a j + 2ak eu0j − 2ak eu0k + ak e2 − a j e2 < a2k /a j e2 −
2ak e2 + ak e2 = e2 a2k /a j − ak < 0, which also leads to a contradiction.
Let u0(1) ≥ . . . ≥ u0( M) be the entries of u0 sorted in decreasing order, and let u∗(1) , . . . , u∗( M)
be the entries of u∗ under the same permutation. Let ρ be the number of nonzero entries inu∗ , i.e., u∗(ρ) > 0, and, if ρ < M, u∗(ρ+1) = 0. Summing (F.21) for ( j) = 1, . . . , ρ, we getρ∑ρ∑ a (r ) u 0(r ) +a(r) u∗(r) −

r =1

r =1

ρ

∑ a (r )

r =1

!

ρ

λ ∑ a(r) u∗(r) = 0,

(F.23)

r =1

which implies
M

∑

ρ

u∗m =

m =1

and therefore τ = 1+λ λρ∑ r =1 a (r )∑u∗(r) =

r =1

1
ρ

ρ

∑ a (r ) u 0(r ) ,1 + λ ∑ r =1 a (r ) r =1

(F.24)

ρ∑r=1 a(r) u0(r) . The complementary slackness conditions forr = ρ and r = ρ + 1 implyρu∗(ρ) − u0(ρ) + λ ∑ a(r) u∗(r) = 0

ρ

and

r =1

− u0∗(ρ+1) + λ ∑ a(r) u∗(r) = ξ (ρ+1) ≥ 0;

(F.25)

r =1

therefore u0(ρ) > u0(ρ) − u∗(ρ) = τ ≥ u0(ρ+1) . This implies that ρ is such thatu 0( ρ ) >λρρ∑ a (r ) u 0(r ) ≥ u 0( ρ +1) .

1 + λ ∑ r =1 a (r ) r =1

(F.26)

The next proposition goes farther by exactly determining ρ.

Proposition F.4 The quantity ρ can be determined via:(
ρ = maxj ∈ {1, . . . , M}jλu 0( j ) −∑ a (r ) u 0(r ) > 0j)

1 + λ ∑ r =1 a (r ) r =1.

(F.27)

Let ρ∗ = max{ j|u∗( j) > 0}. We have that u∗(r) = u0(r) − τ ∗ for r ≤ ρ∗ , where

Proof.

ρ∗λ∑r=1 a(r) u0(r) , and therefore ρ ≥ ρ∗ . We need to prove that ρ ≤ ρ∗ , which weρ∗
1 + λ ∑ r =1 a (r )will do by contradiction. Assume that ρ > ρ∗ . Let u be the vector induced by the choice of

τ∗ =

ρ

ρ, i.e., u(r) = 0 for r > ρ and u(r) = u0(r) − τ for r ≤ ρ, where τ = 1+λ λρ∑ r =1 a (r )∑ r =1 a (r ) u 0(r ) .

From the definition of ρ, we have u(ρ) = u0(ρ) − τ > 0, which implies u(r) = u0(r) − τ > 0 foreach r ≤ ρ. In addition,Mρ∑ ar ur =ρρ∑ a (r ) u 0(r ) − ∑ a (r ) τ =

r =1

r =1

1

r =1
ρ

1−

λ ∑ r =1 a (r )ρ

1 + λ ∑ r =1 a (r )τ∑a (r ) u 0(r ) = ,ρ
λ
1 + λ ∑ r =1 a (r ) r =1∗ρρMa (r ) τ 2 +a (r ) τ 2 +a(r) u20(r)∗r =1
r = ρ +1
r = ρ +1=
M

∑ ar (ur − u0r )2 = ∑∑∑< sumr=1 a(r) τ 2 +∑∗ a(r) u20(r) .

r =1

!

ρ

∑ a (r ) u 0(r )r =1(F.28)

M

ρ∗(F.29)

r = ρ +1

We next consider two cases:ρ∗

1. τ ∗ ≥ τ. From (F.29), we have that ∑rM=1 ar (ur − u0r )2 < ∑r=1 a(r) τ 2 + ∑rM=ρ∗ +1 a(r) u20(r) ≤

2
ρ∗
M
M
∗ )2 + M2∗ − u )2 . From (F.28), we have that
a
(
τ
a
u
=
a
(
u
a
u
∑ r =1 (r )∑ r = ρ ∗ +1 (r ) 0(r ) ∑ r =1 r r∑ r =1 r r =0r
τ 2 /λ2 ≤ (τ ∗ )2 /λ2 . Summing the two inequalities, we get F (u) < F (u∗ ), which leads
to a contradiction.

2. τ ∗ < τ. We will construct a vector ũ from u∗ and show that F (ũ) < F (u∗ ). Defineũ(r) =

2a(ρ∗ +1)u∗ −e, if r = ρ∗ ( ρ ∗ ) a ( ρ ∗ ) + a ( ρ ∗ +1)
2a(ρ∗ )a ( ρ ∗ ) + a ( ρ ∗ +1) ∗u (r )e,

if r = ρ∗ + 1(F.30)

otherwise,

where e = (u0(ρ∗ +1) − τ ∗ )/2. Note that ∑rM=1 ar ũr = ∑rM=1 ar ur∗ . From the assumptions
that τ ∗ < τ and ρ∗ < ρ, we have that u∗(ρ∗ +1) = u0(ρ∗ +1) − τ > 0, which implies thatũ(ρ∗ +1) =a(ρ∗ ) u∗(ρ∗ +1)
a ( ρ ∗ ) ( u 0( ρ ∗ +1) − τ ∗ )a ( ρ ∗ ) ( u 0( ρ ∗ +1) − τ )>=a ( ρ ∗ ) + a ( ρ ∗ +1)
a ( ρ ∗ ) + a ( ρ ∗ +1)a(ρ∗ ) + a(ρ∗ +1) > 0, and thatu∗(ρ∗ )=

u 0( ρ ∗ ) − τ ∗ −=u 0( ρ ∗ ) −

>(i)

1−
1−

=

a ( ρ ∗ +1) ( u 0( ρ ∗ +1) − τ ∗ )a ( ρ ∗ ) + a ( ρ ∗ +1)

a ( ρ ∗ +1) u 0( ρ ∗ +1)
a ( ρ ∗ ) + a ( ρ ∗ +1)!
a ( ρ ∗ +1)

a ( ρ ∗ ) + a ( ρ ∗ +1)
a ( ρ ∗ +1)− 1−

a ( ρ ∗ +1)
a ( ρ ∗ ) + a ( ρ ∗ +1)!
τ∗( u 0( ρ ∗ +1) − τ )!

a ( ρ ∗ ) + a ( ρ ∗ +1)(u∗(ρ∗ +1) ) > 0,

(F.31)

where inequality (i) is justified by the facts that u0(ρ∗ ) ≥ u0(ρ∗ +1) and τ > τ ∗ . This
ensures that ũ is well defined. We have:2( F (u∗ ) − F (ũ))

=

M

M

r =1

r =1

∑ ar (ur∗ − u0r )2 − ∑ ar (ũr − u0r )2∗ 2= a (ρ∗ ) ( τ )− a ( ρ ∗ +1)+ a(ρ∗ +1) u20(ρ∗ +1) − a(ρ∗ )u 0( ρ ∗ +1) −2a(ρ∗ ) e∗

τ +2a(ρ∗ +1) e!2a ( ρ ∗ ) + a ( ρ ∗ +1)!2

a ( ρ ∗ ) + a ( ρ ∗ +1)

4a(ρ∗ ) a2(ρ∗ +1) e2
4a2(ρ∗ ) a(ρ∗ +1) e2( τ − u 0( ρ ∗ +1) ) − = −2 − 2a ( ρ ∗ ) + a ( ρ ∗ +1) |{z}a ( ρ ∗ ) + a ( ρ ∗ +1)
a ( ρ ∗ ) + a ( ρ ∗ +1)
4a(ρ∗ ) a(ρ∗ +1) e

∗

−2e

=

4a(ρ∗ ) a(ρ∗ +1) e2
a ( ρ ∗ ) + a ( ρ ∗ +1)≥ 0,

which leads to a contradiction and completes the proof.

(F.32)

Bibliography
Abney, S. (1996). Statistical methods and linguistics. In The balancing act: Combining symbolicand statistical approaches to language, pages 1–26. MIT Press, Cambridge, MA. 1
Ackley, D., Hinton, G., and Sejnowski, T. (1985). A learning algorithm for Boltzmann machines. Cognitive science, 9(1):147–169. 51
Afonso, M., Bioucas-Dias, J., and Figueiredo, M. (2010). Fast image recovery using variable
splitting and constrained optimization. IEEE Transactions on Image Processing, 19. 105, 107,
219, 227
Aji, S. and McEliece, R. (2000). The generalized distributive law. IEEE Transactions on Information Theory, 46(2):325–343. 48
Alshawi, H. (1996). Head automata and bilingual tiling: Translation with minimal representations. In Proc. of Annual Meeting of the Association for Computational Linguistics, pages
167–176. 21, 147
Altun, Y., Tsochantaridis, I., and Hofmann, T. (2003). Hidden Markov support vector machines. In Proc. of International Conference of Machine Learning. 1, 4, 27, 37, 174, 203, 206
Amari, S. and Nagaoka, H. (2001). Methods of Information Geometry. Oxford University Press.
51
Andrew, G. and Gao, J. (2007). Scalable training of L1-regularized log-linear models. In Proc.
of International Conference of Machine Learning. 211
Anthony, M. and Bartlett, P. (2009). Neural network learning: Theoretical foundations. Cambridge
University Press. 29
Auli, M. and Lopez, A. (2011). A Comparison of Loopy Belief Propagation and Dual Decomposition for Integrated CCG Supertagging and Parsing. In Proc. of Annual Meeting of the
Association for Computational Linguistics. 61, 130, 159
Bach, F. (2008a). Consistency of the group Lasso and multiple kernel learning. Journal of
Machine Learning Research, 9:1179–1225. 189, 194
Bach, F. (2008b). Exploring large feature spaces with hierarchical multiple kernel learning.
Neural Information Processing Systems, 21. 186, 193, 195, 203, 218
Bach, F., Jenatton, R., Mairal, J., and Obozinski., G. (2011). Convex optimization withsparsity-inducing norms. In Optimization for Machine Learning. MIT Press. 4, 206, 207,
211
273

Bach, F., Lanckriet, G., and Jordan, M. (2004). Multiple kernel learning, conic duality, andthe SMO algorithm. In International Conference of Machine Learning. 186, 189, 190
Baker, J. (1979). Trainable grammars for speech recognition. The Journal of the AcousticalSociety of America, 65:S132. 1, 16
Bakin, S. (1999). Adaptive regression and model selection in data mining problems. PhD thesis,
Australian National University. 188, 203, 207, 218
Bakır, G., Hofmann, T., Schölkopf, B., Smola, A., Taskar, B., and Vishwanathan, S. (2007).
Predicting Structured Data. The MIT Press. 1, 27
Bar-Hillel, Y., Perles, M., and Shamir, E. (1964). On formal properties of simple phrase structure grammars. Language and Information: Selected Essays on their Theory and Application,
pages 116–150. 159
Barman, S., Liu, X., Draper, S., and Recht, B. (2011). Decomposition methods for large scale
LP decoding. In 49th Annual Allerton Conference on Communication, Control, and Computing,pages 253–260. IEEE. 132, 133
Baum, L. and Petrie, T. (1966). Statistical inference for probabilistic functions of finite statemarkov chains. The Annals of Mathematical Statistics, 37(6):1554–1563. 13, 48
Baum, L., Petrie, T., Soules, G., and Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains. The Annals of
Mathematical Statistics, 41(1):164–171. 13
Bauschke, H. and Combettes, P. (2011). Convex Analysis and Monotone Operator Theory in
Hilbert Spaces. Springer Verlag. 185, 235
Bayati, M., Shah, D., and Sharma, M. (2005). Maximum weight matching via max-productbelief propagation. In Proceedings of International Symposium on Information Theory, pages
1763–1767. IEEE. 50, 103
Beck, A. and Teboulle, M. (2009). A fast iterative shrinkage-thresholding algorithm for linear
inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202. 197, 219, 226
Berg-Kirkpatrick, T., Gillick, D., and Klein, D. (2011). Jointly learning to extract and compress. In Proc. of Annual Meeting of the Association for Computational Linguistics. 168, 225
Berrou, C., Glavieux, A., and Thitimajshima, P. (1993). Near Shannon limit error-correcting
coding and decoding. In Proc. of International Conference on Communications, volume 93,
pages 1064–1070. 135
Bertsekas, D., Hager, W., and Mangasarian, O. (1999). Nonlinear programming. Athena Scientific. 61, 63, 106, 109, 235, 237
Bertsekas, D., Nedic, A., and Ozdaglar, A. (2003). Convex analysis and optimization. Athena
Scientific. 260
Besag, J. (1974). Spatial interaction and the statistical analysis of lattice systems. Journal of
the Royal Statistical Society. Series B (Methodological), pages 192–236. 46

Bethe, H. (1935). Statistical theory of superlattices. Proceedings of the Royal Society of London.
Series A, Mathematical and Physical Sciences, pages 552–575. 57
Bicego, M., Martins, A., Murino, V., Aguiar, P., and Figueiredo, M. (2010a). 2d shape recognition using information theoretic kernels. In Proc. of International Conference on Pattern
Recognition, pages 25–28. IEEE. 5
Bicego, M., Perina, A., Murino, V., Martins, A., Aguiar, P., and Figueiredo, M. (2010b).
Combining free energy score spaces with information theoretic kernels: Application toscene classification. In Proc. of IEEE International Conference on Image Processing, pages
2661–2664. 5
Bioucas-Dias, J. and Figueiredo, M. (2007). A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Transactions on Image Processing,
16(12):2992–3004. 219
Bishop, C. (2006). Pattern recognition and machine learning. Springer New York. 1, 28
Black, E., Jelinek, F., Lafferty, J., Magerman, D., Mercer, R., and Roukos, S. (1993). Towards
history-based grammars: Using richer models for probabilistic parsing. In Proc. of theAnnual Meeting on Association for Computational Linguistics, pages 31–37. 19, 34
Booth, T. and Thompson, R. (1973). Applying probability measures to abstract languages.
IEEE Transactions on Computers, 100(5):442–450. 16
Borchardt, C. (1860). Über eine interpolationsformel für eine art symmetrischer functionen
und über deren anwendung. Abhandlungen der Königlichen Akademie der Wissenschaften zu
Berlin, pages 1–20. 33
Boros, E. and Hammer, P. (2002). Pseudo-Boolean optimization. Discrete Applied Mathematics,
123(1-3):155–225. 81, 145
Bottou, L. (1991a). Stochastic gradient learning in neural networks. In Proc. of Neuro-Nı̂mes.
4, 39
Bottou, L. (1991b). Une Approche Theorique de l’Apprentissage Connexionniste: Applications a la
Reconnaissance de la Parole. PhD thesis, Université de Paris XI. 186, 192
Bottou, L. (2004). Stochastic learning. In Bousquet, O. and von Luxburg, U., editors, Advanced
Lectures on Machine Learning, Lecture Notes in Artificial Intelligence, LNAI 3176, pages
146–168. Springer Verlag. 39
Bottou, L. and Bousquet, O. (2007). The tradeoffs of large scale learning. Neural Information
Processing Systems, 20. 38
Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. (2011). Distributed Optimization andStatistical Learning via the Alternating Direction Method of Multipliers. Now Publishers. 105,
107, 109, 111, 114, 129, 130, 255
Boyd, S. P. and Vandenberghe, L. (2004). Convex optimization. Cambridge University Press.
31, 235

Boyle, J. and Dykstra, R. (1986). A method for finding projections onto the intersections ofconvex sets in Hilbert spaces. In Advances in order restricted statistical inference, pages 28–47.
Springer Verlag. 261
Bradley, P. and Mangasarian, O. (1998). Feature selection via concave minimization andsupport vector machines. In Proc. of International Conference of Machine Learning, pages
82–90. 33
Bresnan, J. (2001). Lexical-functional syntax. Wiley-Blackwell. 22
Buchholz, S. and Marsi, E. (2006). CoNLL-X shared task on multilingual dependency parsing. In International Conference on Natural Language Learning. 151, 162, 182, 216
Carpenter, B. (1997). Type-logical semantics. The MIT Press. 225
Carpenter, B. (2008). Lazy sparse stochastic gradient descent for regularized multinomial
logistic regression. Technical report, Alias-i. 211, 213
Carreras, X. (2007). Experiments with a higher-order projective dependency parser. In International Conference on Natural Language Learning. 24, 137, 145, 162
Carreras, X., Collins, M., and Koo, T. (2008). TAG, Dynamic Programming, and the Perceptron for Efficient, Feature-rich Parsing. In International Conference on Natural Language
Learning. 162
Caruana, R. (1997). Multitask learning. Machine Learning, 28(1):41–75. 208, 218
Casella, G. and Robert, C. (1999). Monte Carlo statistical methods. Springer-Verlag, New York.
50
Cayley, A. (1889). A theorem on trees. Quarterly Journal of Mathematics, 23(376-378):69. 33
Censor, Y. and Zenios, S. A. (1997). Parallel optimization: Theory, algorithms, and applications.
Oxford University Press. 181
Cesa-Bianchi, N., Conconi, A., and Gentile, C. (2004). On the generalization ability of on-line
learning algorithms. IEEE Transactions on Information Theory, 50(9):2050–2057. 39, 197
Chang, M., Ratinov, L., and Roth, D. (2007). Guiding semi-supervision with constraint-driven
learning. In Proc. of Annual Meeting of the Association for Computational Linguistics. 69
Chang, M., Ratinov, L., and Roth, D. (2008). Constraints as prior knowledge. In International
Conference of Machine Learning: Workshop on Prior Knowledge for Text and Language Processing.
68, 73, 137
Chang, Y.-W. and Collins, M. (2011). Exact decoding of phrase-based translation models
through lagrangian relaxation. In Proc. of Empirical Methods for Natural Language Processing.
61, 130, 159, 225
Chapelle, O. and Rakotomamonjy, A. (2008). Second order optimization of kernel parameters. In Neural Information Processing Systems: Workshop on Kernel Learning and Automatic
Selection of Optimal Kernels. 186, 190, 203

Charniak, E. (1996a). Statistical language learning. The MIT Press. 11, 19
Charniak, E. (1996b). Tree-bank grammars. In Proc. of the National Conference on Artificial
Intelligence, pages 1031–1036. 1, 19
Charniak, E. (1997). Statistical parsing with a context-free grammar and word statistics. In
Proc. of the National Conference on Artificial Intelligence, pages 598–603. 19, 35, 136
Charniak, E. and Johnson, M. (2005). Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proc. of Annual Meeting of the Association for Computational Linguistics,
pages 173–180. 162
Charniak, E., Johnson, M., Elsner, M., Austerweil, J., Ellis, D., Haxton, I., Hill, C., Shrivaths,
R., Moore, J., Pozar, M., et al. (2006). Multilevel coarse-to-fine pcfg parsing. In Proc. of
Human Language Technology Conference of the North American Chapter of the Association for
Computational Linguistics, pages 168–175. 20, 225
Chen, S. and Rosenfeld, R. (1999). A Gaussian prior for smoothing maximum entropy models. Technical report, CMU-CS-99-108. 31
Chen, S. F. and Rosenfeld, R. (2000). A survey of smoothing techniques for maximum entropy
models. IEEE Transactions on Speech and Audio Processing, 8(1):37–50. 31
Chomsky, N. (1956). Three models for the description of language. IRE Transactions on
Information Theory, 2(3):113–124. 15
Chomsky, N. (1965). Aspects of the Theory of Syntax, volume 119. The MIT press. 5, 15, 16, 21
Chomsky, N. (1993). Lectures on government and binding: The Pisa lectures, volume 9. Walterde Gruyter. 22
Chomsky, N. (1995). The minimalist program, volume 28. Cambridge Univ Press. 22
Chu, Y. J. and Liu, T. H. (1965). On the shortest arborescence of a directed graph. Science
Sinica, 14:1396–1400. 25, 144
Clarke, J. and Lapata, M. (2008). Global Inference for Sentence Compression An Integer
Linear Programming Approach. Journal of Artificial Intelligence Research, 31:399–429. 69,
137, 168, 225
Cocke, J. and Schwartz, J. (1970). Programming languages and their compilers; preliminary
notes. Technical report, Courant Institute of Mathematical Sciences, NYU. 19
Collins, M. (1999). Head-driven statistical models for natural language parsing. PhD thesis,
University of Pennsylvania. 19, 35
Collins, M. (2002a). Discriminative training methods for hidden Markov models: theory andexperiments with perceptron algorithms. In Proc. of Empirical Methods for Natural Language
Processing. 1, 4, 27, 41, 42, 174, 186, 203, 206
Collins, M. (2002b). Ranking algorithms for named-entity extraction: Boosting and the voted
perceptron. In Proc. of Annual Meeting of the Association for Computational Linguistics. 181,215

Collins, M., Globerson, A., Koo, T., Carreras, X., and Bartlett, P. (2008). Exponentiated gradient algorithms for conditional random fields and max-margin Markov networks. Journalof Machine Learning Research, 9:1775–1822. 38, 181, 186
Collobert, R., Sinz, F., Weston, J., and Bottou, L. (2006). Trading convexity for scalability. In
International Conference of Machine Learning. 175
Combettes, P. and Wajs, V. (2006). Signal recovery by proximal forward-backward splitting.
Multiscale Modeling and Simulation, 4(4):1168–1200. 191, 192, 203
Coppersmith, D. and Winograd, S. (1990). Matrix multiplication via arithmetic progressions.
Journal of symbolic computation, 9(3):251–280. 25
Covington, M. (1990). Parsing discontinuous constituents in dependency grammar. Computational Linguistics, 16(4):234–236. 21
Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., and Singer, Y. (2006). Online PassiveAggressive Algorithms. Journal of Machine Learning Research, 7:551–585. 4, 42, 43, 162, 173,
174, 177, 180, 181, 186, 215, 224
Crammer, K. and Singer, Y. (2002). On the algorithmic implementation of multiclass kernelbased vector machines. Journal of Machine Learning Research, 2:265–292. 32
Crammer, K. and Singer, Y. (2003). Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951–991. 42, 173, 174, 224
Culotta, A. and Sorensen, J. (2004). Dependency tree kernels for relation extraction. In
Annual Meeting of the Association for Computational Linguistics. 5, 21
Culotta, A., Wick, M., Hall, R., and McCallum, A. (2007). First-order probabilistic models
for coreference resolution. In Human Language Technology Conference of the North American
Chapter of the Association for Computational Linguistics (HLT/NAACL), pages 81–88. 69
Danskin, J. (1966). The theory of max-min, with applications. SIAM Journal on Applied
Mathematics, pages 641–664. 237
Dantzig, G. and Wolfe, P. (1960). Decomposition principle for linear programs. Operations
Research, 8(1):101–111. 61
Darmois, G. (1935). Sur les lois de probabilitéa estimation exhaustive. CR Académie desSciences de Paris, 260:1265–1266. 51
Das, D. (2012). Semi-Supervised and Latent-Variable Models of Natural Language Semantics. PhD
thesis, Carnegie Mellon University. 129, 130, 225
Daumé, H., Langford, J., and Marcu, D. (2009). Search-based structured prediction. Machinelearning, 75(3):297–325. 34, 50
Dawid, A. (1992). Applications of a general propagation algorithm for probabilistic expert
systems. Statistics and Computing, 2(1):25–36. 48

Dechter, R. (1999). Bucket elimination: A unifying framework for reasoning. Artificial Intelligence, 113(1):41–85. 48
Dechter, R. and Mateescu, R. (2007). And/or search spaces for graphical models. Artificial
intelligence, 171(2):73–106. 101, 224
Della Pietra, S., Della Pietra, V., and Lafferty, J. (1997). Inducing features of random fields.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 19:380–393. 206
Dempster, A., Laird, N., and Rubin, D. (1977). Maximum likelihood from incomplete data
via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological), pages
1–38. 13
Denis, P. and Baldridge, J. (2007). Joint Determination of Anaphoricity and Coreference
Resolution using Integer Programming. In Proc. of Human Language Technology Conferenceof the North American Chapter of the Association for Computational Linguistics. 69, 137, 225
Ding, Y. and Palmer, M. (2005). Machine translation using probabilistic synchronous dependency insertion grammar. In Proc. of Annual Meeting of the Association for Computational
Linguistics. 5, 21
Donoho, D. and Johnstone, J. (1994).
Biometrika, 81(3):425. 237

Ideal spatial adaptation by wavelet shrinkage.

Dreyer, M. and Eisner, J. (2006). Better Informed Training of Latent Syntactic Features. In
Proc. of Empirical Methods in Natural Language Processing, pages 317–326. 20
Duchi, J., Shalev-Shwartz, S., Singer, Y., and Chandra, T. (2008). Efficient projections onto
the L1-ball for learning in high dimensions. In Proc. of International Conference of Machine
Learning. 118, 268
Duchi, J. and Singer, Y. (2009). Efficient online and batch learning using forward backward
splitting. Journal of Machine Learning Research, 10:2873–2908. 186, 193, 196, 211
Duchi, J., Tarlow, D., Elidan, G., and Koller, D. (2007). Using combinatorial optimization
within max-product belief propagation. Advances in Neural Information Processing Systems,
19. 3, 68, 69, 82
Duda, R., Hart, P., and Stork, D. (2001). Pattern classification, volume 2. Wiley New York. 28
Eckstein, J. and Bertsekas, D. (1992). On the Douglas-Rachford splitting method and theproximal point algorithm for maximal monotone operators. Mathematical Programming,
55(1):293–318. 105, 109, 115, 132
Edmonds, J. (1967). Optimum branchings. Journal of Research of the National Bureau of Standards, 71B:233–240. 25, 144
Eisenstein, J., Smith, N. A., and Xing, E. P. (2011). Discovering sociolinguistic associations
with structured sparsity. In Proc. of Annual Meeting of the Association for Computational
Linguistics. 206, 208, 218

Eisner, J. (1996). Three new probabilistic models for dependency parsing: An exploration.
In Proc. of International Conference on Computational Linguistics, pages 340–345. 5, 19, 24, 35,
136, 146, 147
Eisner, J. (2000). Bilexical grammars and their cubic-time parsing algorithms. Advances in
Probabilistic and Other Parsing Technologies, pages 29–62. 24
Eisner, J. and Satta, G. (1999). Efficient parsing for bilexical context-free grammars andhead automaton grammars. In Proc. of Annual Meeting of the Association for Computational
Linguistics. 24
Everett III, H. (1963). Generalized Lagrange multiplier method for solving problems of
optimum allocation of resources. Operations Research, 11(3):399–417. 61
Facchinei, F. and Pang, J. (2003). Finite-dimensional variational inequalities and complementarity
problems, volume 1. Springer Verlag. 252
Feldman, J., Wainwright, M., and Karger, D. (2005). Using linear programming to decodebinary linear codes. IEEE Transactions on Information Theory, 51(3):954–972. 77
Figueiredo, M. (2002). Adaptive sparseness using Jeffreys’ prior. Advances in Neural Information Processing Systems. 210
Fillmore, C. (1976). Frame Semantics and the Nature of Language. Annals of the New York
Academy of Sciences, 280(1):20–32. 128
Finkel, J., Grenager, T., and Manning, C. (2005). Incorporating non-local information into
information extraction systems by Gibbs sampling. In Proc. of Annual Meeting on Association
for Computational Linguistics, pages 363–370. 50
Finkel, J., Kleeman, A., and Manning, C. (2008). Efficient, feature-based, conditional random
field parsing. Proc. of Annual Meeting on Association for Computational Linguistics, pages
959–967. 19, 20, 36, 136
Finley, T. and Joachims, T. (2008). Training structural SVMs when exact inference is intractable. In Proc. of International Conference of Machine Learning. 184, 226
Forney, G. (1973). The Viterbi algorithm. Proceedings of the IEEE, 61(3):268–278. 13, 48
Fortin, M. and Glowinski, R. (1983). Augmented Lagrangian Methods, volume 15. Elsevier. 109
Frank, M. and Wolfe, P. (1956). An algorithm for quadratic programming. Naval researchlogistics quarterly, 3(1-2):95–110. 121
Friedman, J., Hastie, T., and Tibshirani, R. (2010). A note on the group lasso and a sparsegroup lasso. Technical report, Stanford University. 193, 195, 203, 209, 218
Friedman, N., Getoor, L., Koller, D., and Pfeffer, A. (1999). Learning probabilistic relationalmodels. In International Joint Conference on Artificial Intelligence, volume 16, pages 1300–
1309. 3, 68, 71

Gabay, D. and Mercier, B. (1976). A dual algorithm for the solution of nonlinear variational
problems via finite element approximation. Computers and Mathematics with Applications,
2(1):17–40. 105, 107, 109, 132
Gabow, H., Galil, Z., Spencer, T., and Tarjan, R. (1986). Efficient algorithms for finding
minimum spanning trees in undirected and directed graphs. Combinatorica, 6(2):109–122.
25, 144
Gaifman, H. (1964). Concerning measures in first order calculi. Israel journal of mathematics,
2(1):1–18. 68
Gaifman, H. (1965). Dependency systems and phrase-structure systems. Information andcontrol, 8(3):304–337. 22
Gallager, R. (1962). Low-density parity-check codes. IRE Transactions on Information Theory,8(1):21–28. 77
Galley, M. (2006). A skip-chain conditional random field for ranking meeting utterances byimportance. In Proc. of Empirical Methods in Natural Language Processing, pages 364–372. 14
Ganchev, K., Graca, J., Gillenwater, J., and Taskar, B. (2010). Posterior regularization forstructured latent variable models. Journal of Machine Learning Research, 11:2001–2049. 101,
225
Gao, J., Andrew, G., Johnson, M., and Toutanova, K. (2007). A comparative study of parameter estimation methods for statistical natural language processing. In Proc. of Annual
Meeting of the Association for Computational Linguistics. 31, 207
Gelfand, A. and Smith, A. (1990). Sampling-based approaches to calculating marginal densities. Journal of the American statistical association, pages 398–409. 50
Geman, S. and Geman, D. (1984). Stochastic relaxation, gibbs distributions, and the bayesian
restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:721–
741. 50
Germann, U., Jahr, M., Knight, K., Marcu, D., and Yamada, K. (2001). Fast decoding andoptimal decoding for machine translation. In Proc. of Annual Meeting on Association for
Computational Linguistics, pages 228–235. 137
Giménez, J. and Marquez, L. (2004). SVMTool: A general POS tagger generator based on
Support Vector Machines. In Proc. of International Conference on Language Resources and
Evaluation. 34, 162
Gimpel, K. and Smith, N. A. (2010). Softmax-Margin CRFs: Training Log-Linear Models
with Loss Functions. In NAACL. 175
Globerson, A. and Jaakkola, T. (2008). Fixing max-product: Convergent message passing
algorithms for MAP LP-relaxations. Neural Information Processing Systems, 20. 3, 59, 60, 61,
93, 106, 126, 127, 233

Glowinski, R. and Le Tallec, P. (1989). Augmented Lagrangian and operator-splitting methods innonlinear mechanics. Society for Industrial Mathematics. 105, 113, 132
Glowinski, R. and Marroco, A. (1975). Sur l’approximation, par éléments finis d’ordre
un, et la résolution, par penalisation-dualité, d’une classe de problèmes de Dirichlet nonlinéaires. Rev. Franc. Automat. Inform. Rech. Operat., 9:41–76. 105, 107, 109, 132
Goldberg, Y. and Elhadad, M. (2010). An efficient algorithm for easy-first non-directional
dependency parsing. In Proc. of Annual Conference of the North American Chapter of the
Association for Computational Linguistics, pages 742–750. 34
Goldfarb, D., Ma, S., and Scheinberg, K. (2010). Fast alternating linearization methods forminimizing the sum of two convex functions. Technical report, UCLA CAM 10-02. 105,
107
Goodman, J. (1999). Semiring parsing. Computational Linguistics, 25(4):573–605. 48
Goodman, J. (2004). Exponential priors for maximum entropy models. In Proc. of Annual
Meeting of the North American Chapter of the Association for Computational Linguistics. 31
Graça, J., Ganchev, K., Taskar, B., and Pereira, F. (2009). Posterior vs. parameter sparsity inlatent variable models. Advances in Neural Information Processing Systems. 218
Grave, E., Obozinski, G., and Bach, F. (2011). Trace lasso: a trace norm regularization forcorrelated designs. Arxiv preprint arXiv:1109.1990. 219
Greig, D., Porteous, B., and Seheult, A. (1989). Exact maximum a posteriori estimation forbinary images. Journal of the Royal Statistical Society, Series B (Methodological), pages 271–279.
52
Grünbaum, B. (2003). Convex polytopes, volume 221. Springer Verlag. 226
Guyon, I. and Elisseeff, A. (2003). An introduction to variable and feature selection. Journalof Machine Learning Research, 3:1157–1182. 206
Halevy, A., Norvig, P., and Pereira, F. (2009). The unreasonable effectiveness of data. Intelligent Systems, IEEE, 24(2):8–12. 1
Halpern, J. (1990). An analysis of first-order logics of probability. Artificial Intelligence,46(3):311–350. 68
Hammersley, J. and Clifford, P. (1971). Markov field on finite graphs and lattices. Unpublished manuscript. 46
Hastie, T., Tibshirani, R., and Friedman, J. (2001). The Elements of Statistical Learning. New
York: Springer-Verlag. 29
Hazan, E., Agarwal, A., and Kale, S. (2007). Logarithmic regret algorithms for online convex
optimization. Machine Learning, 69(2):169–192. 39, 195
Hazan, T. and Shashua, A. (2010). Norm-product belief propagation: Primal-dual messagepassing for approximate inference. IEEE Transactions on Information Theory, 56(12):6294–
6316. 59, 93, 101, 225

He, B. and Yuan, X. (2011). On the O(1/t) convergence rate of alternating direction method.
SIAM Journal of Numerical Analysis (to appear). 105, 115, 132, 251, 252
Henderson, J. (2003). Inducing history representations for broad coverage statistical parsing.
In Proc. of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 24–31. 20
Hestenes, M. (1969). Multiplier and gradient methods. Journal of Optimization Theory andApplications, 4:302–320. 107, 109
Hofmann, T., Scholkopf, B., and Smola, A. J. (2008). Kernel methods in machine learning.
Annals of Statistics, 36(3):1171. 188
Huang, B. and Jebara, T. (2007). Loopy belief propagation for bipartite maximum weightb-matching. International Conference of Artificial Intelligence and Statistics. 50, 103
Huang, J., Huang, X., and Metaxas, D. (2009). Learning with dynamic group sparsity. In
Proc. International Conference of Computer Vision, pages 64–71. 219
Huang, L. (2008). Forest reranking: Discriminative parsing with non-local features. In ACL.
136
Huang, L. and Sagae, K. (2010). Dynamic programming for linear-time incremental parsing.
In Proc. of Annual Meeting of the Association for Computational Linguistics, pages 1077–1086.
5, 24, 34, 163, 166
Hudson, R. (1984). Word grammar. Blackwell Oxford. 5, 21
Ising, E. (1925). Beitrag zur theorie des ferromagnetismus. Zeitschrift für Physik A Hadrons
and Nuclei, 31(1):253–258. 51
Jaakkola, T. and Haussler, D. (1998). Exploiting generative models in discriminative classifiers. Technical report, Dept. of Computer Science, Univ. of California. 32
Jebara, T., Kondor, R., and Howard, A. (2004). Probability product kernels. Journal of Machine
Learning Research, 5:819–844. 32
Jelinek, F. (1997). Statistical methods for speech recognition. the MIT Press. 1, 12
Jenatton, R., Audibert, J.-Y., and Bach, F. (2009). Structured variable selection with sparsityinducing norms. Technical report, arXiv:0904.3523. 4, 186, 195, 203, 206, 207, 209, 218
Jenatton, R., Mairal, J., Obozinski, G., and Bach, F. (2010). Proximal methods for sparse
hierarchical dictionary learning. In Proc. of International Conference of Machine Learning.
206, 211, 218
Jie, L., Orabona, F., Fornoni, M., Caputo, B., and Cesa-Bianchi, N. (2010). OM-2: An online
multi-class Multi-Kernel Learning algorithm. In Proc. of the 4th IEEE Online Learning for
Computer Vision Workshop, pages 43–50. IEEE. 203
Johnson, J. (2008). Equivalence of Entropy Regularization and Relative-Entropy Proximal
Method. Unpublished manuscript. 107

Johnson, J., Malioutov, D., and Willsky, A. (2007). Lagrangian relaxation for MAP estimationin graphical models. In 45th Annual Allerton Conference on Communication, Control andComputing. 61, 106, 107
Johnson, M. (1998). PCFG models of linguistic tree representations. Computational Linguistics,
24(4):613–632. 19, 35, 136
Jojic, V., Gould, S., and Koller, D. (2010). Accelerated dual decomposition for MAP inference.
In International Conference of Machine Learning. 107, 115, 126, 127, 130, 132, 256
Jurafsky, D. and Martin, J. (2000). Speech & Language Processing. Prentice Hall. 11
Kahane, S., Nasr, A., and Rambow, O. (1998). Pseudo-projectivity: a polynomially parsable
non-projective dependency grammar. In International Conference on Computational Linguistics. 22
Kakade, S. and Shalev-Shwartz, S. (2008). Mind the Duality Gap: Logarithmic regret algorithms for online optimization. In Neural Information Processing Systems. 44, 174, 177,
178
Kasami, T. (1966). An efficient recognition and syntax-analysis algorithm for context-freelanguages. Technical report, Air Force Cambridge Research Lab (Scientific report AFCRL65-758). 19
Kazama, J. and Torisawa, K. (2007). A new perceptron algorithm for sequence labeling with
non-local features. In Proc. of Empirical Methods for Natural Language Processing. 181
Kazama, J. and Tsujii, J. (2003). Evaluation and extension of maximum entropy models withinequality constraints. In Proc. of Empirical Methods for Natural Language Processing. 31
Kim, S. and Xing, E. (2010). Tree-guided group lasso for multi-task regression with structured
sparsity. In Proc. of International Conference of Machine Learning. 206, 218
Kimeldorf, G. and Wahba, G. (1971). Some results on Tchebycheffian spline functions. Journalof Mathematical Analysis and Applications, 33(1):82–95. 33
Kirchhoff, G. (1847). Ueber die auflösung der gleichungen, auf welche man bei der untersuchung der linearen vertheilung galvanischer ströme geführt wird. Annalen der Physik,148(12):497–508. 25
Klein, D. and Manning, C. (2003). Accurate unlexicalized parsing. In Proc. of Annual Meetingon Association for Computational Linguistics, pages 423–430. 19
Kloft, M., Brefeld, U., Sonnenburg, S., and Zien, A. (2010). Non-Sparse Regularization andEfficient Training with Multiple Kernels. Arxiv preprint arXiv:1003.0079. 186, 189, 190, 191,
194, 200, 203
Koehn, P. (2010). Statistical Machine Translation, volume 9. Cambridge University Press. 11
Koller, D. and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques.
The MIT Press. 3, 45, 46, 50

Kolmogorov, V. (2006). Convergent tree-reweighted message passing for energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28:1568–1583. 3, 59, 93,
106
Komodakis, N. and Paragios, N. (2009). Beyond pairwise energies: Efficient optimization forhigher-order MRFs. In Proc. of Conference on Computer Vision and Pattern Recognition. 106
Komodakis, N., Paragios, N., and Tziritas, G. (2007). MRF optimization via dual decomposition: Message-passing revisited. In Proc. of International Conference on Computer Vision. 3,
61, 63, 106, 126, 223, 225, 256
Koo, T. and Collins, M. (2010). Efficient third-order dependency parsers. In Proc. of Annual
Meeting of the Association for Computational Linguistics, pages 1–11. 5, 24, 130, 163, 164
Koo, T., Globerson, A., Carreras, X., and Collins, M. (2007). Structured prediction models
via the matrix-tree theorem. In Empirical Methods for Natural Language Processing. 25, 177
Koo, T., Rush, A. M., Collins, M., Jaakkola, T., and Sontag, D. (2010). Dual decomposition
for parsing with non-projective head automata. In Proc. of Empirical Methods for Natural
Language Processing. 5, 26, 61, 114, 135, 147, 149, 151, 152, 153, 159, 160, 163, 164, 165, 166,
167, 224, 226
Koopman, B. (1936). On distributions admitting a sufficient statistic. Transactions of theAmerican Mathematical Society, 39(399):10. 51
Kovalevsky, V. and Koval, V. (1975). A diffusion algorithm for decreasing energy of max-sum
labeling problem. Technical report, Glushkov Institute of Cybernetics, Kiev, USSR. 60, 126
Kowalski, M. and Torrésani, B. (2009). Structured sparsity: From mixed norms to structured
shrinkage. In Workshop on Signal Processing with Adaptive Sparse Structured Representations.
194
Kschischang, F. R., Frey, B. J., and Loeliger, H. A. (2001). Factor graphs and the sum-productalgorithm. IEEE Transactions on Information Theory, 47. 45, 47
Kulesza, A. and Pereira, F. (2007). Structured Learning with Approximate Inference. Neural
Information Processing Systems. 114, 184, 226
Lacoste-Julien, S., Taskar, B., Klein, D., and Jordan, M. I. (2006). Word alignment via
quadratic assignment. In Proc. of Annual Meeting of the North American Chapter of the Association for Computational Linguistics. 137
Lafferty, J., McCallum, A., and Pereira, F. (2001). Conditional random fields: Probabilistic
models for segmenting and labeling sequence data. In Proc. of International Conference ofMachine Learning. 1, 4, 13, 27, 35, 36, 174, 186, 203, 206
Lafferty, J., Sleator, D., and Temperley, D. (1992). Grammatical trigrams: A probabilistic
model of link grammar. In Proceedings of the 1992 AAAI Fall Symposium on Probabilistic
Approaches to Natural Language, pages 89–97. 21

Lanckriet, G. R. G., Cristianini, N., Bartlett, P., Ghaoui, L. E., and Jordan, M. I. (2004). Learning the kernel matrix with semidefinite programming. Journal of Machine Learning Research,
5:27–72. 185, 186, 188, 190, 203, 218
Langford, J., Li, L., and Zhang, T. (2009). Sparse online learning via truncated gradient.
Journal of Machine Learning Research, 10:777–801. 193, 196, 211, 212, 213
Lauritzen, S. (1996). Graphical Models. Clarendon Press, Oxford. 45, 50
Lavergne, T., Cappé, O., and Yvon, F. (2010). Practical very large scale CRFs. In Proc. ofAnnual Meeting of the Association for Computational Linguistics. 31, 207, 211
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to
document recognition. Proceedings of the IEEE, 86(11):2278–2324. 174, 181
Liang, P. and Jordan, M. (2008). An asymptotic analysis of generative, discriminative, andpseudolikelihood estimators. In Proc. of International Conference on Machine learning, pages
584–591. ACM. 32
Liang, P., Jordan, M., and Klein, D. (2011). Learning dependency-based compositional semantics. In Proc. fo Annual Meeting of the Association for Computational Linguistics (ACL).
168, 225
Liao, L., Fox, D., and Kautz, H. (2005). Location-based activity recognition. In Advances in
Neural Information Processing Systems. 102
Lions, P. and Mercier, B. (1979). Splitting algorithms for the sum of two nonlinear operators.
SIAM Journal on Numerical Analysis, 16(6):964–979. 203
Liu, D. C. and Nocedal, J. (1989). On the limited memory BFGS method for large scale
optimization. Mathematical Programming, 45:503–528. 182
Liu, J. and Ye, J. (2010a). Fast Overlapping Group Lasso. Technical report, arXiv:1009.0306.
187, 195
Liu, J. and Ye, J. (2010b). Moreau-Yosida regularization for grouped tree structure learning.
In Advances in Neural Information Processing Systems. 187, 195, 207, 211, 218
Lorbert, A., Eis, D., Kostina, V., Blei, D., and Ramadge, P. (2010). Exploiting covariate similarity in sparse regression via the pairwise elastic net. In Proc. of International Conference on
Artificial Intelligence and Statistics, volume 9, pages 477–484. 219
Low, Y., Gonzalez, J., Kyrola, A., Bickson, D., Guestrin, C., and Hellerstein, J. (2010).
Graphlab: A new parallel framework for machine learning. In International Conference
on Uncertainty in Artificial Intelligence. 131
Magerman, D. (1995). Statistical decision-tree models for parsing. In Proc. of Annual Meetingon Association for Computational Linguistics, pages 276–283. 19, 34, 35
Magnanti, T. and Wolsey, L. (1994). Optimal Trees. Technical Report 290-94, Massachusetts
Institute of Technology, Operations Research Center. 140, 141, 142, 143

Mairal, J., Jenatton, R., Obozinski, G., and Bach, F. (2010). Network flow algorithms forstructured sparsity. In Advances in Neural Information Processing Systems. 187, 195, 218
Manning, C. and Schütze, H. (1999). Foundations of Statistical Natural Language Processing.
MIT Press, Cambridge, MA. 1, 11, 19
Marcus, M., Marcinkiewicz, M., and Santorini, B. (1993). Building a large annotated corpus
of English: The Penn Treebank. Computational linguistics, 19(2):313–330. 1, 4, 18
Marroquin, J., Mitter, S., and Poggio, T. (1987). Probabilistic solution of ill-posed problems
in computational vision. Journal of the American Statistical Association, pages 76–89. 50
Martin, R., Rardin, R., and Campbell, B. (1990). Polyhedral characterization of discrete
dynamic programming. Operations research, pages 127–138. 102
Martins, A., Bicego, M., Murino, V., Aguiar, P., and Figueiredo, M. (2010a). Information
theoretical kernels for generative embeddings based on hidden markov models. Structural,
Syntactic, and Statistical Pattern Recognition, pages 463–472. 5
Martins, A. F. T., Das, D., Smith, N. A., and Xing, E. P. (2008a). Stacking Dependency Parsers.
In Proc. of Empirical Methods for Natural Language Processing. 5, 163, 164
Martins, A. F. T., Figueiredo, M. A. T., and Aguiar, P. M. Q. (2008b). Tsallis Kernels on
Measures. In IEEE Information Theory Workshop. 5
Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., and Xing, E. P. (2008c).
Nonextensive Entropic Kernels. In Proc. of International Conference of Machine Learning. 5
Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., and Xing, E. P. (2009a).
Nonextensive Information Theoretic Kernels on Measures. Journal of Machine Learning
Research, 10:935–975. 5, 32
Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., and Xing, E. P. (2010b).
Online MKL for Structured Prediction. Arxiv preprint arXiv:1010.2770. 185
Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., and Xing, E. P.
(2011a). An Augmented Lagrangian Approach to Constrained MAP Inference. In Proc. ofInternational Conference on Machine Learning. 105, 168, 225
Martins, A. F. T., Figueiredo, M. A. T., Aguiar, P. M. Q., Smith, N. A., and Xing, E. P. (2011b).
Online learning of structured predictors with multiple kernels. In Gordon, G., Dunson,
D., and Dudı́k, M., editors, International Conference on Artificial Intelligence and Statistics,
volume 15. JMLR W&CP. 185, 205, 218
Martins, A. F. T., Gimpel, K., Smith, N. A., Xing, E. P., Aguiar, P. M. Q., and Figueiredo,
M. A. T. (2010c). Learning Structured Classifiers with Dual Coordinate Ascent. Technicalreport, CMU-ML-10-109. 173, 181
Martins, A. F. T. and Smith, N. A. (2009). Summarization with a Joint Model for Sentence
Extraction and Compression. In North American Chapter of the Association for Computational
Linguistics: Workshop on Integer Linear Programming for NLP. 5, 168, 225

Martins, A. F. T., Smith, N. A., Aguiar, P. M. Q., and Figueiredo, M. A. T. (2011c). Dual
Decomposition with Many Overlapping Components. In Proc. of Empirical Methods for
Natural Language Processing. 135, 136
Martins, A. F. T., Smith, N. A., Aguiar, P. M. Q., and Figueiredo, M. A. T. (2011d). Structured Sparsity in Structured Prediction. In Proc. of Empirical Methods for Natural Language
Processing. 205
Martins, A. F. T., Smith, N. A., and Xing, E. P. (2009b). Concise Integer Linear Programming
Formulations for Dependency Parsing. In Proc. of Annual Meeting of the Association for
Computational Linguistics. 135, 136, 145, 147, 150, 151, 152, 155, 158, 162, 163, 177, 182
Martins, A. F. T., Smith, N. A., and Xing, E. P. (2009c). Polyhedral Outer Approximations with
Application to Natural Language Parsing. In Proc. of International Conference of Machine
Learning. 5, 114, 162, 173, 184, 226
Martins, A. F. T., Smith, N. A., Xing, E. P., Aguiar, P. M. Q., and Figueiredo, M. A. T. (2010d).
Augmented Dual Decomposition for MAP Inference. In Neural Information Processing Systems: Workshop in Optimization for Machine Learning. 105
Martins, A. F. T., Smith, N. A., Xing, E. P., Aguiar, P. M. Q., and Figueiredo, M. A. T. (2010e).
Online MKL for Structured Prediction. In Neural Information Processing Systems: Workshop
in New Directions in Multiple Kernel Learning. 185, 189
Martins, A. F. T., Smith, N. A., Xing, E. P., Figueiredo, M. A. T., and Aguiar, P. M. Q. (2010f).
Turbo Parsers: Dependency Parsing by Approximate Variational Inference. In Proc. of
Empirical Methods for Natural Language Processing. 67, 100, 135, 136, 154, 158, 163, 164, 173
Matsuzaki, T., Miyao, Y., and Tsujii, J. (2005). Probabilistic CFG with latent annotations. In
Proc. of Annual Meeting on Association for Computational Linguistics, pages 75–82. 20
McAllester, D., Collins, M., and Pereira, F. (2008). Case-factor diagrams for structured probabilistic modeling. Journal of Computer and System Sciences, 74(1):84–96. 98, 101, 224
McCallum, A. (2003). Efficiently inducing features of conditional random fields. In Proc. ofUncertainty in Artificial Intelligence. 206
McDonald, R., Crammer, K., and Pereira, F. (2005a). Online large-margin training of dependency parsers. In Proc. of Annual Meeting of the Association for Computational Linguistics. 42,
162, 173
McDonald, R., Lerman, K., and Pereira, F. (2006). Multilingual dependency analysis witha two-stage discriminative parser. In Proc. of International Conference on Natural Language
Learning. 23, 25, 136, 137, 162, 163
McDonald, R. and Satta, G. (2007). On the complexity of non-projective data-driven dependency parsing. In Proc. of International Conference on Parsing Technologies. 25, 145, 146,
177

McDonald, R. T., Pereira, F., Ribarov, K., and Hajic, J. (2005b). Non-projective dependency
parsing using spanning tree algorithms. In Proc. of Empirical Methods for Natural Language
Processing. 5, 25, 35, 136, 216
McDonald, R. T. and Pereira, F. C. N. (2006). Online learning of approximate dependency
parsing algorithms. In Proc. of Annual Meeting of the European Chapter of the Association for
Computational Linguistics. 5, 23, 25, 145, 146
McEliece, R. J., MacKay, D. J. C., and Cheng, J. F. (1998). Turbo decoding as an instance of
Pearl’s “belief propagation” algorithm. IEEE Journal on Selected Areas in Communications,16(2). 135
Mel’čuk, I. (1988). Dependency syntax: theory and practice. State University of New York Press.
5, 21
Meshi, O. and Globerson, A. (2011). An Alternating Direction Method for Dual MAP LP Relaxation. In European Conference on Machine Learning and Principles and Practice of Knowledge
Discovery in Databases. 132
Meza-Ruiz, I. and Riedel, S. (2009). Jointly Identifying Predicates, Arguments and Senses
using Markov Logic. In Proc. of North American Chapter of the Association for Computational
Linguistics. 68
Michelot, C. (1986). A finite algorithm for finding the projection of a point onto the canonicalsimplex of Rn . Journal of Optimization Theory and Applications, 50(1):195–200. 118
Milch, B., Marthi, B., Russell, S., Sontag, D., Ong, D., and Kolobov, A. (2007). BLOG: Probabilistic Models with Unknown Objects. Statistical relational learning, page 373. 68
Mitchell, T. (1997). Machine learning. McGraw Hill. 1, 28
Mohri, M. (2002). Semiring frameworks and algorithms for shortest-distance problems. Journal of Automata, Languages and Combinatorics, 7(3):321–350. 48
Moreau, J. (1962). Fonctions convexes duales et points proximaux dans un espace hilbertien.
CR de l’Académie des Sciences de Paris Série A Mathematics, 255:2897–2899. 191, 192, 195, 203
Mota, J., Xavier, J., Aguiar, P., and Puschel, M. (2010). Distributed basis pursuit. IEEE
Transactions on Signal Processing, 99. 105
Murray, M. and Rice, J. (1993). Differential Geometry and Statistics. Number 48 in Monographson Statistics and Applied Probability. Chapman and Hall, 1st edition. 51
Nesterov, Y. (1983). A method of solving a convex programming problem with convergence
rate O(1/k2 ). Soviet Math. Doklady, 27:372–376. 39, 107
Nesterov, Y. (2005). Smooth minimization of non-smooth functions. Mathematical Programming, 103(1):127–152. 107, 130
Ng, A. and Jordan, M. (2002). On discriminative vs. generative classifiers: A comparison oflogistic regression and naive bayes. Advances in neural information processing systems, 14:841.
32

Nivre, J. (2009). Non-projective dependency parsing in expected linear time. In Annual
Meeting of the Association for Computational Linguistics, pages 351–359. 26
Nivre, J., Hall, J., Nilsson, J., Eryiǧit, G., and Marinov, S. (2006). Labeled pseudo-projective
dependency parsing with support vector machines. In Procs. of International Conference on
Natural Language Learning. 24, 34, 163
Nivre, J. and McDonald, R. (2008). Integrating graph-based and transition-based dependency
parsers. In Proc. of Annual Meeting of the Association for Computational Linguistics. 163
Nocedal, J. and Wright, S. (1999). Numerical optimization. Springer verlag. 121, 123, 124, 125
Nowozin, S. and Lampert, C. (2009). Global connectivity potentials for random field models.
In IEEE Conference on Computer Vision and Pattern Recognition, pages 818–825. IEEE. 68
Obozinski, G., Taskar, B., and Jordan, M. (2010). Joint covariate selection and joint subspace
selection for multiple classification problems. Statistics and Computing, 20(2):231–252. 206,
218
Pal, C., Sutton, C., and McCallum, A. (2006). Sparse forward-backward using minimum
divergence beams for fast training of conditional random fields. In IEEE International
Conference on Acoustics, Speech and Signal Processing, volume 5. IEEE. 34
Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference.
Morgan Kaufmann. 3, 45, 48
Pereira, F. (2000). Formal grammar and information theory: together again? PhilosophicalTransactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering
Sciences, 358(1769):1239–1253. 1
Pereira, F. and Schabes, Y. (1992). Inside-outside reestimation from partially bracketed corpora. In Proc. of Annual Meeting on Association for Computational Linguistics, pages 128–135.
19
Petrov, S. (2009). Coarse-to-Fine Natural Language Processing. PhD thesis, University of California at Bekeley, Berkeley, CA, USA. 20
Petrov, S. and Klein, D. (2007). Improved inference for unlexicalized parsing. In Proc. ofAnnual Meeting of the North American Chapter of the Association for Computational Linguistics,
pages 404–411. 20
Petrov, S. and Klein, D. (2008a). Discriminative log-linear grammars with latent variables.
Advances in Neural Information Processing Systems, 20:1153–1160. 20, 207
Petrov, S. and Klein, D. (2008b). Sparse multi-scale grammars for discriminative latent variable parsing. In Proc. of Empirical Methods for Natural Language Processing. 20, 207
Pitman, E. (1936). Sufficient statistics and intrinsic accuracy. In Mathematical Proceedings of
the Cambridge Philosophical Society, volume 32, pages 567–579. Cambridge Univ Press. 51
Poole, D. (1993). Probabilistic horn abduction and bayesian networks. Artificial Intelligence,64(1):81–129. 68

Poon, H. and Domingos, P. (2009). Unsupervised semantic parsing. In Proc. of Empirical
Methods in Natural Language Processing. 68
Poon, H. and Domingos, P. (2011). Sum-product networks: A new deep architecture. In IEEE
International Conference on Computer Vision Workshops, pages 689–690. IEEE. 101, 224
Powell, M. (1969). A method for nonlinear constraints in minimization problems. In Fletcher,R., editor, Optimization, pages 283–298. Academic Press. 107, 109
Punyakanok, V., Roth, D., Yih, W., and Zimak, D. (2004). Semantic role labeling via integer
linear programming inference. In Internacional Conference on Computational Linguistics. 137
Punyakanok, V., Roth, D., Yih, W., and Zimak, D. (2005). Learning and Inference overConstrained Output. In Proc. of International Joint Conference on Artificial Intelligence. 68, 69,
73
Quattoni, A., Carreras, X., Collins, M., and Darrell, T. (2009). An efficient projection for l1,∞
regularization. In Proc. of International Conference of Machine Learning. 208
Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications inspeech recognition. Proceedings of the IEEE, 77(2):257–286. 1, 12, 13, 35
Rakotomamonjy, A., Bach, F., Canu, S., and Grandvalet, Y. (2008). SimpleMKL. Journal of
Machine Learning Research, 9:2491–2521. 186, 189, 190, 200, 203
Ratliff, N., Bagnell, J., and Zinkevich, M. (2006). Subgradient methods for maximum margin
structured learning. In International Conference of Machine Learning: Workshop on Learning in
Structured Outputs Spaces. 40, 186
Ratnaparkhi, A. (1999). Learning to parse natural language with maximum entropy models.
Machine Learning, 34(1):151–175. 20, 34
Ravikumar, P., Agarwal, A., and Wainwright, M. (2010). Message-passing for graphstructured linear programs: Proximal methods and rounding schemes. Journal of Machine
Learning Research, 11:1043–1080. 59, 106
Richardson, M. and Domingos, P. (2006).
62(1):107–136. 2, 3, 68, 71, 137, 160, 223

Markov logic networks.

Machine Learning,

Richardson, T. and Urbanke, R. (2008). Modern coding theory. Cambridge University Press.
68, 77
Riedel, S. and Clarke, J. (2006). Incremental integer linear programming for non-projective
dependency parsing. In Proc. of Empirical Methods for Natural Language Processing. 68, 69,
135, 138, 140, 141, 167, 224
Riezler, S. (1998).
Tübingen. 68

Probabilistic constraint logic programming.

PhD thesis, University ofRobbins, H. and Monro, S. (1951). A stochastic approximation model. Annals of Mathematical
Statistics, 22:400–407. 39

Rockafellar, R. (1970). Convex Analysis. Princeton University Press. 54, 139, 235, 236
Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage andorganization in the brain. Psychological review, 65(6):386. 41, 42
Roth, D. and Yih, W. (2004). A linear programming formulation for global inference innatural language tasks. In International Conference on Natural Language Learning. 2, 3, 68,
73, 223
Roth, D. and Yih, W. T. (2005). Integer linear programming inference for conditional random
fields. In International Conference of Machine Learning. 69, 137
Rush, A., Sontag, D., Collins, M., and Jaakkola, T. (2010). On dual decomposition and linear
programming relaxations for natural language processing. In Proc. of Empirical Methods for
Natural Language Processing. 3, 61, 114, 159, 223, 225
Rush, A. M. and Collins, M. (2011). Exact decoding of syntactic translation models throughlagrangian relaxation. In Proc. of Annual Meeting on Association for Computational Linguistics.
61, 114, 130, 159, 225
Sag, I., Wasow, T., Bender, E., and Sag, I. (1999). Syntactic theory: A formal introduction,
volume 92. Center for the Study of Language and Information. 22
Sang, E. (2002). Introduction to the CoNLL-2002 shared task: Language-independent namedentity recognition. In Proc. of International Conference on Natural Language Learning. 215
Sang, E. and Buchholz, S. (2000). Introduction to the CoNLL-2000 shared task: Chunking.
In Proc. of International Conference on Natural Language Learning. 214
Sang, E. and De Meulder, F. (2003). Introduction to the CoNLL-2003 shared task: Languageindependent named entity recognition. In Proc. of International Conference on Natural Language Learning. 215
Schlesinger, M. (1976). Syntactic analysis of two-dimensional visual signals in noisy conditions. Kibernetika, 4:113–130. 3, 59
Schmidt, M. and Murphy, K. (2010). Convex structure learning in log-linear models: Beyond
pairwise potentials. In Proc. of International Conference on Artificial Intelligence and Statistics.
218
Schölkopf, B. and Smola, A. J. (2002). Learning with Kernels. The MIT Press, Cambridge, MA.
1, 28, 29, 33, 187
Schrijver, A. (2003). Combinatorial Optimization: Polyhedra and Efficiency, volume 24 of Algorithms and Combinatorics. Springer. 102, 140, 143, 226
Sha, F. and Pereira, F. (2003). Shallow parsing with conditional random fields. In Proc. ofAnnual Meeting of the North American Chapter of the Association for Computational Linguistics.
39
Shafer, G. and Shenoy, P. (1990). Probability propagation. Annals of Mathematics and Artificial
Intelligence, 2(1):327–351. 48

Shalev-Shwartz, S. (2011). Online learning and online convex optimization. Foundations andTrends in Machine Learning, 4(2):107–194. 185
Shalev-Shwartz, S. and Singer, Y. (2006). Online learning meets optimization in the dual. In
Proc. of Conference on Learning Theory. 44, 173, 174, 177
Shalev-Shwartz, S., Singer, Y., and Srebro, N. (2007). Pegasos: Primal estimated sub-gradientsolver for svm. In Proc. of International Conference of Machine Learning. 39, 40, 41, 193, 197,
199, 213
Shalev-Shwartz, S., Singer, Y., Srebro, N., and Cotter, A. (2010). Pegasos: primal estimatedsub-gradient solver for SVM. Mathematical Programming, 125. 39, 40, 41, 193
Shor, N. (1985). Minimization methods for non-differentiable functions. Springer. 61
Smith, D. and Eisner, J. (2006). Minimum risk annealing for training log-linear models. In
Proc. of Annual Meeting of the Association for Computation Linguistics. 175
Smith, D. and Eisner, J. (2008). Dependency parsing by belief propagation. In Proc. of Empirical Methods for Natural Language Processing. 3, 5, 25, 68, 69, 77, 78, 79, 82, 135, 146, 151, 152,
153, 158, 160, 162, 167, 174, 177, 182, 224
Smith, D. and Smith, N. (2007). Probabilistic models of nonprojective dependency trees. In
Proc. of Empirical Methods for Natural Language Processing. 25, 177
Smith, N. A. (2011). Linguistic Structure Prediction, volume 13 of Synthesis Lectures on Human
Language Technologies. Morgan and Claypool. 1, 11
Sokolovska, N., Lavergne, T., Cappé, O., and Yvon, F. (2010). Efficient learning of sparseconditional random fields for supervised sequence labelling. IEEE Journal of Selected Topics
in Signal Processing, 4(6):953–964. 211
Sonnenburg, S., Rätsch, G., Schäfer, C., and Schölkopf, B. (2006). Large scale MKL. Journalof Machine Learning Research, 7:1565. 186, 190, 201, 203
Sontag, D. (2010). Approximate Inference in Graphical Models using LP Relaxations. PhD thesis,
MIT. 50, 59, 226
Sontag, D., Globerson, A., and Jaakkola, T. (2011). Introduction to dual decomposition for
inference. In Optimization for Machine Learning. MIT Press. 60, 61, 62, 126
Sontag, D. and Jaakkola, T. (2009). Tree block coordinate descent for MAP in graphicalmodels. In Proc. of International Conference on Artificial Intelligence and Statistics, volume 8,
pages 544–551. Journal of Machine Learning Research: W&CP. 60
Sontag, D., Meltzer, T., Globerson, A., Weiss, Y., and Jaakkola, T. (2008). Tightening LP
relaxations for MAP using message-passing. In Proc. of Uncertainty in Artificial Intelligence.
3, 61, 128, 131, 225
Surdeanu, M., Johansson, R., Meyers, A., Màrquez, L., and Nivre, J. (2008). The CoNLL-2008
Shared Task on Joint Parsing of Syntactic and Semantic Dependencies. Proc. of International
Conference on Natural Language Learning. 162, 182

Sutton, C. (2004). Collective segmentation and labeling of distant entities in information
extraction. Technical report, DTIC Document. 68
Sutton, C. and McCallum, A. (2006). An introduction to conditional random fields for relational learning. In Introduction to statistical relational learning, pages 95–130. MIT press.
14
Suzuki, T. and Tomioka, R. (2009). SpicyMKL. Arxiv preprint arXiv:0909.5026. 194, 203
Tanner, R. (1981). A recursive approach to low complexity codes. IEEE Transactions on
Information Theory, 27(5):533–547. 45
Tarjan, R. (1977). Finding optimum branchings. Networks, 7(1):25–36. 25, 144
Taskar, B., Abbeel, P., and Koller, D. (2002). Discriminative probabilistic models for relational
data. In Proc. of Uncertainty in Artificial Intelligence, pages 895–902. 68
Taskar, B., Chatalbashev, V., and Koller, D. (2004a). Learning associative Markov networks.
In Proc. of International Conference of Machine Learning. 52
Taskar, B., Guestrin, C., and Koller, D. (2003). Max-margin Markov networks. In Proc. of
Neural Information Processing Systems. 1, 4, 15, 27, 37, 174, 186, 189, 199, 200, 203, 206
Taskar, B., Klein, D., Collins, M., Koller, D., and Manning, C. (2004b). Max-margin parsing.
In Proc. of Empirical Methods for Natural Language Processing, pages 1–8. 19, 20, 98
Taskar, B., Lacoste-Julien, S., and Jordan, M. (2006a). Structured Prediction via the Extragradient Method. In Neural Information Processing Systems. 38
Taskar, B., Lacoste-Julien, S., and Jordan, M. I. (2006b). Structured prediction, dual extragradient and Bregman projections. Journal of Machine Learning Research, 7:1627–1653. 38,
176
Tesnière, L. (1959). Eléments de syntaxe structurale. Libraire C. Klincksieck. 5, 21
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal
Statistical Society B., pages 267–288. 31, 207
Tikhonov, A. (1943). On the stability of inverse problems. In Doklady Akademii Nauk SSSR,
volume 39, pages 195–198. 31
Tjong Kim Sang, E. F. and De Meulder, F. (2003). Introduction to the CoNLL-2003 sharedtask: Language-independent named entity recognition. In International Conference on Natural Language Learning. 181
Tomioka, R. and Suzuki, T. (2010). Sparsity-accuracy trade-off in MKL. Technical report,
arXiv:1001.2615. 193, 194
Tromble, R. and Eisner, J. (2006). A fast finite-state relaxation method for enforcing global
constraints on sequence decoding. In Proc. of Annual Meeting of the North American Chapterof the Association for Computational Linguistics, pages 423–430. 159

Tsochantaridis, I., Hofmann, T., Joachims, T., and Altun, Y. (2004). Support vector machinelearning for interdependent and structured output spaces. In Proc. of International Conference of Machine Learning. 1, 4, 27, 37, 174, 186, 203, 206
Tsuruoka, Y., Tsujii, J., and Ananiadou, S. (2009). Stochastic gradient descent training forl1-regularized log-linear models with cumulative penalty. In Proc. of Annual Meeting of the
Association for Computational Linguistics. 211
Tutte, W. (1984). Graph Theory. Addison-Wesley, Reading, MA. 25
Valiant, L. (1984). A theory of the learnable. Communications of the ACM, 27(11):1134–1142.
39
Vapnik, N. V. (2000). The Nature of Statistical Learning Theory. Springer-Verlag, New York. 32
Vapnik, V. (1998). Statistical Learning Theory. Wiley. 29
Vazirani, V. (2001). Approximation Algorithms. Springer. 184, 226
Vishwanathan, S. V. N., Schraudolph, N. N., Schmidt, M. W., and Murphy, K. P. (2006).
Accelerated training of conditional random fields with stochastic gradient methods. In
International Conference of Machine Learning. 4, 39, 186
Viterbi, A. (1967). Error bounds for convolutional codes and an asymptotically optimum
decoding algorithm. IEEE Transactions on Information Theory, 13(2):260–269. 13, 48
Wainwright, M., Jaakkola, T., and Willsky, A. (2005a). MAP estimation via agreement on
trees: message-passing and linear programming. IEEE Transactions on Information Theory,51(11):3697–3717. 3, 59, 93, 106
Wainwright, M. and Jordan, M. (2008). Graphical Models, Exponential Families, and Variational
Inference. Now Publishers. 3, 46, 50, 51, 53, 54, 55, 57, 59, 60, 75, 182
Wainwright, M. J. (2006). Estimating the” Wrong” Graphical Model: Benefits in theComputation-Limited Setting. Journal of Machine Learning Research, 7:1829–1859. 184, 226
Wainwright, M. J., Jaakkola, T., and Willsky, A. (2005b). A new class of upper bounds on thelog partition function. IEEE Transactions on Information Theory, 51(7):2313–2335. 59, 93, 101,
225
Wang, H. and Banerjee, A. (2012). Online Alternating Direction Method. In Proc. of International Conference on Machine Learning. 105, 115, 132, 251, 252
Wang, M., Smith, N. A., and Mitamura, T. (2007). What is the Jeopardy model? A quasisynchronous grammar for QA. In Proceedings of Empirical Methods for Natural Language
Processing-CoNLL. 5, 21
Weiss, D. and Taskar, B. (2010). Structured prediction cascades. In Proc. of International
Conference on Artificial Intelligence and Statistics, volume 1284. 225
Weiss, Y. and Freeman, W. (2001a). Correctness of belief propagation in gaussian graphicalmodels of arbitrary topology. Neural computation, 13(10):2173–2200. 50, 103

Weiss, Y. and Freeman, W. (2001b). On the optimality of solutions of the max-productbelief-propagation algorithm in arbitrary graphs. IEEE Transactions on Information Theory,47(2):736–744. 50, 103
Weiss, Y., Yanover, C., and Meltzer, T. (2007). Map estimation, linear programming and belief
propagation with convex free energies. In Proc. of Uncertainty in Artificial Intelligence. 59,
93, 101, 225
Werner, T. (2007). A linear programming approach to max-sum problem: A review. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 29:1165–1179. 59, 60, 93, 106, 126
Wiegerinck, W. and Heskes, T. (2003). Fractional belief propagation. In Advances in Neural
Information Processing Systems, volume 15, page 455. 59, 93, 101, 225
Wong, R. (1984). A dual ascent approach for steiner tree problems on a directed graph.
Mathematical programming, 28(3):271–287. 140
Wright, S., Nowak, R., and Figueiredo, M. A. T. (2009). Sparse reconstruction by separable
approximation. IEEE Transactions on Signal Processing, 57(7):2479–2493. 191, 203, 208, 214,
219, 226
Xiao, L. (2009). Dual averaging methods for regularized stochastic learning and online optimization. In Advances in Neural Information Processing Systems. 211
Xiao, L. (2010). Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 11:2543–2596. 219, 227
Xu, Z., Jin, R., King, I., and Lyu, M. (2009). An extended level method for efficient multiple
kernel learning. Neural Information Processing Systems, 21:1825–1832. 186, 191, 197, 203
Yamada, H. and Matsumoto, Y. (2003). Statistical dependency analysis with support vector
machines. In Proc. of International Conference on Parsing Technologies. 34, 162
Yanover, C., Meltzer, T., and Weiss, Y. (2006). Linear programming relaxations and belief
propagation–an empirical study. Journal of Machine Learning Research, 7:1887–1907. 59, 128,
129
Yedidia, J., Freeman, W., and Weiss, Y. (2004). Constructing free-energy approximations andgeneralized belief propagation algorithms. Technical Report TR2004-040, MERL. 74
Yedidia, J., Freeman, W., and Weiss, Y. (2005). Constructing free-energy approximations
and generalized belief propagation algorithms. IEEE Transactions on Information Theory,
51(7):2282–2312. 59
Yedidia, J., Wang, Y., and Draper, S. (2011). Divide and concur and difference-map BP
decoders for LDPC codes. IEEE Transactions on Information Theory, 57(2):786–802. 132
Yedidia, J. S., Freeman, W. T., and Weiss, Y. (2001). Generalized belief propagation. In Neural
Information Processing Systems. 58, 59
Younger, D. (1967). Recognition and parsing of context-free languages in time n3 . Informationand control, 10(2):189–208. 19

Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with groupedvariables. Journal of the Royal Statistical Society Series B (Statistical Methodology), 68(1):49.
188, 203, 207, 218
Zhang, N. and Poole, D. (1994). A simple approach to Bayesian network computations. In
Tenth Canadian Conference on Artificial Intelligence, pages 171–178. 48
Zhang, Y. and Clark, S. (2008). A tale of two parsers: investigating and combining graphbased and transition-based dependency parsing using beam-search. In Empirical Methods
for Natural Language Processing. 50
Zhao, P., Rocha, G., and Yu, B. (2009). The composite absolute penalties family for groupedand hierarchical variable selection. Annals of Statistics, 37(6A):3468–3497. 4, 186, 193, 195,
203, 206, 208, 218
Zien, A. and Ong, C. (2007). Multiclass multiple kernel learning. In Proc. of International
Conference of Machine Learning. 186, 203
Zinkevich, M. (2003). Online Convex Programming and Generalized Infinitesimal Gradient
Ascent. In Proc. of International Conference of Machine Learning. 39, 193, 195
Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journalof the Royal Statistical Society Series B(Statistical Methodology), 67(2):301–320. 31, 207