Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

1 INTRODUCTION . Deep structured networks such as deep convolutional ( CNN ) and recurrent ( RNN ) neural networks have enjoyed great success in many real-world problems , including scene classification ( 8 ) , semantic segmentation ( 16 ) , speech recognition ( 6 ) , gaming ( 12 ; 13 ) , and image captioning ( 7 ) . However , the current configuration of these networks is restricted to accept and predict structured inputs and outputs such as vectors , matrices , and tensors1 . If the problem ’ s inputs or/and outputs can not be modelled in this way , these learning approaches all fail to learn a proper model ( 26 ) . However , many real-world problems are naturally described as unstructured data such as sets ( 21 ; 26 ) . A set is a collection of elements which is invariant under permutation and the size of a set is not fixed in advance . Set learning using deep networks is an emerging field of study that has generated substantial interest very recently ( 20 ; 21 ; 23 ; 26 ) . Consider the task of object detection as an example . Given a structured input , e.g . an image as a tensor , the goal is to predict a set of orderless locations , e.g . bounding boxes , from an unknown and varying number of objects . Therefore , the output of this problem can be properly modelled as a set of entities . However , a deep learning network can not be simply trained to learn a proper model in order to directly predict unfixed number of orderless locations . Existing approaches formulate this problem using as a pre-defined and fixed-sized grid ( 18 ; 17 ) or anchor boxes ( 19 ) representing all possible locations and scales of the objects . Then , each location and scale is scored independently to contain an object or not . The final output is generated heuristically by a discretization process such as non-maximum suppression ( NMS ) , which is not part of the learning process . Therefore , their performance is hindered by this heuristic procedure . To this end , the current solutions can only deal with moderate object occlusion . We argue that object detection problem can be properly formulated as a set prediction problem , where a deep learning network is trained to output a set of locations without any heuristic . This shortcoming concerns not only object detection but also all problems where a set of instances ( as input or/and output ) is involved , e.g . a set of topics or concepts in documents ( 3 ) , segmentation of object instances ( 10 ) and a set of trajectories in multiple object tracking ( 2 ) . In contrast to problems such as classification , where the order of categories or the labels can be fixed during the training 1Throughout the paper , we use the term structured data for vectors , matrices and generally tensors which are fixed in size and ordered , i.e . any permutation of its values changes its meaning . This is not to be confused with structured learning which is commonly used for graphs . process and the output can be well represented by a fixed-sized vector , the instances are often unfixed in number and orderless . More precisely , it does not matter in which order the instances are labelled and these labels do not relate the instances to a specific visual category . To this end , training a deep network to predict instances seems to be non-trivial . We believe that set learning is a principled solution to all of these problems . In this paper , we present a novel approach for learning to deal with sets using deep learning . More clearly , in the presented model , we assume that the input ( the observation ) is still structured , e.g . an image , but the annotated output is available as a set . Our approach is inspired by a recent work on set learning using deep neural networks ( 21 ) . Although the work in ( 21 ) handles orderless outputs in the testing/inference step , it requires ordered outputs in the learning step . This is a significant limitation , because in many applications , such as object detection , the outputs are naturally not ordered . When that happens , the approach in ( 21 ) can not learn a sensible model ( see Appendix for the experiment ) . In this paper , we propose a complete set prediction formulation to address this limitation . This provides a potential to tackle many more problems compared to ( 21 ) . The main contribution of the paper is summarised as follows : 1 . We propose a principled formulation for neural networks to deal with arbitrary sets with unknown permutation and cardinality as available annotated outputs . This makes a neural network , for the first time , able to truly handle orderless outputs at both training and test time . 2 . Additionally , our formulation allows us to learn the distribution over the unobservable permutation variables , which can be used to identify the most likely orders of the set . In some applications , there may exist one or several dominant orders , which are unknown from the annotations . 3 . For the first time , we reformulate object detection as a set prediction problem , where a deep network is learned end-to-end to generate the detection outputs with no heuristic involved . We outperform the state-of-the art object detectors , such as Faster R-CNN and YOLO v2 , on both simulated and real data with high level of occlusions . 4 . We also demonstrate the applicability of our framework algorithm for a complex CAPTCHA test which can be formulated as a set prediction problem . 2 RELATED WORK . Handling unstructured input and output data , such as sets or point patterns , for both learning and inference is an emerging field of study that has generated substantial interest in recent years . Approaches such as mixture models ( 3 ; 5 ; 22 ) , learning distributions from a set of samples ( 14 ; 15 ) , model-based multiple instance learning ( 25 ) and novelty detection from point pattern data ( 24 ) , can be counted as few out many examples that use point patterns or sets as input or output and directly or indirectly model the set distributions . However , existing approaches often rely on parametric models , e.g . i.i.d . Poisson point or Gaussian Process assumptions ( 1 ; 24 ) . Recently , deep learning has enabled us to use less parametric models to capture highly complex mapping distributions between structured inputs and outputs . Somewhat surprisingly , there are only few works on learning sets using deep neural networks . One interesting exception in this direction is the recent work of ( 23 ) , which uses an RNN to read and predict sets . However , the output is still assumed to have a single order , which contradicts the orderless property of sets . Moreover , the framework can be used in combination with RNNs only and can not be trivially extended to any arbitrary learning framework such as feed-forward architectures . Another recent work proposed by ( 26 ) is a deep learning framework which can deal with sets as input with different sizes and permutations . However , the outputs are either assumed to be structured , e.g . a scalar as a score , or a set with the same entities of the input set , which prevents this approach to be used for the problems that require output sets with arbitrary entities . Perhaps the most related work to our problem is a deep set network recently proposed by ( 21 ) which seamlessly integrates a deep learning framework into set learning in order to learn to output sets . However , the approach only formulates the outputs with unknown cardinality and does not consider the permutation variables of sets in the learning step . Therefore , its application is limited to the problems with a fixed order output such as image tagging and diverges when trying to learn unordered output sets as for the object detection problem . In this paper , we incorporate these permutations as unobservable variables in our formulation , and estimate their distribution during the learning process . This technical extension makes our proposed framework the only existing approach in literature which can truly learn to predict sets with arbitrary entities and permutations . It has the potential to reformulate some of the existing problems , such as object detection , and to tackle a set of new applications , such as a logical CAPTCHA test which can not be trivially solved by the existing architectures . 3 DEEP PERM-SET NETWORK . A set is a collection of elements which is invariant under permutation and the size of a set is not fixed in advance , i.e . Y = { y1 , · · · , ym } , m ∈ N∗ . A statistical function describing a finite-set variable Y is a combinatorial probability density function p ( Y ) defined by p ( Y ) = p ( m ) Umpm ( { y1 , y2 , · · · , ym } ) , where p ( m ) is the cardinality distribution of the set Y and pm ( { y1 , y2 , · · · , ym } ) is a symmetric joint probability density distribution of the set given known cardinality m. U is the unit of hyper-volume in the feature space , which cancels out the unit of the probability density pm ( · ) making it unit-less , and thereby avoids the unit mismatch across the different dimensions ( cardinalities ) ( 25 ) . Throughout the paper , we use Y = { y1 , · · · , ym } for a set with unknown cardinality and permutation , Ym = { y1 , · · · , ym } m for a set with known cardinality m but unknown permutation and Ymπ = ( yπ1 , · · · , yπm ) for an ordered set with known cardinality ( or dimension ) m and permutation π , which means that the m set elements are ordered under the permutation vector π = ( π1 , π2 , . . . , πm ) . Note that an ordered set with known dimension and permutation is exactly a structured data such as vector , matrix and tensor . According to the permutation invariant property of the sets , the set Ym with known cardinality m can be expressed by an ordered set with any arbitrary permutation , i.e . Ym : = { Ymπ |∀π ∈ Π } , where , Π is the space of all feasible permutation Π = { π1 , π2 , · · · , πm ! } and |Π| : = m ! . Therefore , the probability density of a set Y with unknown permutation and cardinality conditioned on the input x and the model parameters w is defined as p ( Y|x , w ) = p ( m|x , w ) × Um × pm ( Ym|x , w ) , = p ( m|x , w ) × Um × ∑ ∀π∈Π pm ( Y m π , π|x , w ) . ( 1 ) The parameters w models both the cardinality distribution of the set elements p ( m|· ) as well as the joint state distribution of set elements and their permutation for a fixed cardinality pm ( Ymπ , π|· ) . The above formulation represents the probability density of a set which is very general and completely independent of the choices of cardinality , state and permutation distributions . It is thus straightforward to transfer it to many applications that require the output to be a set . Definition of these distributions for the applications in this paper will be elaborated later .

Summary: The method extends [21], which proposes an unordered set prediction model for multi-class classification. For that problem, [21] can assume logistic outputs for all distinct classes. This work extends set prediction to the object detection task, where box identity is not distinct — this is handled by an additional model output that reasons about the most likely object permutations. The permutation predictions are used during training, but are not needed at inference time — as shown in Fig1 and Eq 7. Results are on detection of overlapping objects and a CAPTCHA toy summation example.