Title: UNIFYING PART DETECTION ASSOCIATION FOR MULTI-PERSON POSE ESTIMATION Anonymous

Abstract: Current bottom-up approaches for 2D multi-person pose estimation (MPPE) detect joints collectively without distinguishing between individuals. Associating the joints into individual poses is done independently of the learning algorithm, therefore requires formulating a separate problem in a post-processing step, which relies on relaxations or sophisticated heuristics. We propose a differentiable learning-based model that performs part detection and association jointly, thereby eliminating the need for further post-processing. The approach introduces a recurrent neural network (RNN), which takes dense low-level features as input and predicts the heatmaps of a single person's joints in each iteration, then refines them in a feedback loop. In addition, the network learns a stopping criterion in order to halt once it has identified all individuals in an image, allowing it to output any number of poses. Furthermore, we introduce an efficient implementation that allows training on memory-constrained machines. The approach is evaluated on the challenging COCO and OCHuman datasets and substantially outperforms the baseline. On OCHuman, which contains severe occlusions, we achieve state-ofthe-art results even compared to top-down approaches. Our results demonstrate the advantage of a learning-based detection and association framework, and the advantage of bottom-up approaches over top-down approaches in complex scenarios.

1. INTRODUCTION:

The task of multiperson human pose estimation is defined as the localization of a predefined set of anatomical joints and their grouping distinctly into individuals. It is an integral component for many applications in computer vision in which humans are active participants such as assisted driving Fridman et al. (2017) , assisted living Planinc et al. (2016) and sports analysis Shih (2017) . Challenges associated with the tasks include variation in joint articulation, occlusions, overlaps or close proximity of joints. Current prevalent approaches are largely based on two paradigms, the first follows a top-down design Iqbal & Gall (2016) , in which first a single person is detected and then his/her joints are localized. Such approaches rely on bounding box proposals and are therefore not robust to occlusions or partial visibility of people. Incorrect detections in this stage carry over to the joint localization stage, which is not aware of association. The second paradigm follows a bottom-up approach design in which first a set of joint candidates of all people are collectively identified without distinction, and then grouped into poses for each person individually Pishchulin et al. (2016) ; Cao et al. (2017) ; Newell et al. (2017) . In current state-of-the-art bottom-up approaches, grouping is performed independently of the training, and entails formulating a separate problem that usually relies on greedy heuristics and relaxations. In these approaches, the ultimate objective remains unknown to the training algorithm as it optimizes over an intermediate objective, thereby limiting the network in using its full potential. Furthermore, when such heuristics are employed, it remains unclear how the association can be further refined or generalized. Cao et al. (2017) who follow a bottom-up approach, train two different branches for the part confidence map and for the association, referred to as Part Affinity Fields, which are used to capture the relationships between the joints and are used as edge weights for optimization in a graph matching problem only during inference. Since the formulation results in a k-dimensional matching problem, an NP-hard problem, the problem was solved using greedy relaxation thereby yielding a suboptimal solution. Newell et al. (2017) have devised a loss that tries to guide the network in learning distinct embeddings for each individual.

However, these embeddings are not employed for association during training and are only used in a greedy post-processing step. In this work, we propose a end-to-end differentiable approach which conflates part detection and association into a single model. We aim to show the benefit of such a learning-based approach for detection and association, as well as the advantage of bottom-up approaches over top-down. The approach is evaluated on the challenging COCO keypoint dataset Lin et al. (2014) and extremely challenging OCHuman dataset Zhang et al. (2019a) .

1.1. RELATED WORK:

In multi-person pose estimation, methods that follow a top-down design such as Carreira et al. (2016) ; Iqbal & Gall (2016) ; Chen et al.; Xiao et al. (2018) ; Huang et al. (2017) ; He et al. (2017) rely on person detectors and estimate the pose for each detected bounding box individually. The performance of these approaches therefore is tightly coupled with the performance of the underlying person detector. Furthermore, if people overlap heavily, non-maximum-suppression (NMS) results in eliminating some of them. Conversely, bottom up approaches such as Pishchulin et al. ( 2016 2018) predict features such as vector fields Cao et al. (2017) ; Raaj et al. (2018) which indicate the correspondence between detected joints. This allows reducing the pose assembly into a greedy bipartite graph matching problem as in Cao et al. (2017) . These approaches formulate association as a separate optimization problem that does not participate in the training. Kocabas et al. (2018) propose a method that combines person and part detections, such that keypoints and person bounding boxes are simultaneously detected, followed by assigning the keypoints to person boxes using a learned function. Carreira et al. (2016) introduce a corrective iterative feedback method, in which the CNN predicts an additive correction to the current joint estimate in the Cartesian representation. However, part detection based on confidence maps has been shown as more powerful at capturing context and relationships between the joints Tompson et al. (2014) ; Newell et al. (2017; 2016) ; Cao et al. (2017) ; Pishchulin et al. (2016) ; Wang et al. (2018) than regression based bottom-up methods. Another problem that is related to multi-person pose estimation is the task of instance segmentation. Romera-Paredes & Torr (2016) propose using an RNN for instance segmentation, but without inferring the label of an instance. Salvador et al. (2017) eliminate this shortcoming by introducing an additional branch that predicts the class of each instance. However, they do not localize the exact joint locations nor address the distinct case of people instances which have unique articulation features. Zhang et al. (2019a) extract human pose features in order to perform person instance segmentation instead of relying on bounding-box proposals. To the best of our knowledge, there is no bottom-up method that estimates the heatmaps of each person individually or performs learning-based association of detected parts.

2. BACKGROUND:



2.1. ASSOCIATIVE EMBEDDING (AE):

In implementing our approach, we make use of the extracted features of AE's Stacked Hourglass network Newell et al. (2017) . It is noteworthy, however, that the proposed approach is independent of the underlying network and can be combined with any bottom-down approach. In their work, the authors have devised a network for dense predictions that can be used for solving multi-person pose estimation. Since we use the extracted features of Newell et al. (2017) and compare our approach with theirs, we first give a brief description of their work. In the Stacked Hourglass architecture, several CNNs that have a symmetric structure are stacked together. The output of the stacks are similar, and each stack output comprises maps of part detections, associative embeddings, and representa-tion features extracted at the intermediate layers. A part detection heatmap I j detects all joints of type j without distinguishing which person they belong to. The embedding maps aim at producing distinct values for each person instance, with similar values for the same instance.

2.2. HEURISTIC-BASED ASSOCIATION:

The embeddings of the AE network are utilized in a post-processing step in order to group the joints into individual poses. Since the heatmaps are shared for all people, they perform several additional steps that are based on hand-crafted heuristics in order to find the grouping. In what follows we describe these operations in order to show their potential disadvantages: before taking candidate detections of every joint confidence map I j , NMS is applied on each of the confidence maps. This operation requires specifying a neighborhood size from which the maximum detection is taken, therefore may erroneously suppress nearby joint detections. The candidate detections are thresholded such that only detections that are above a predefined threshold can be associated with individuals. While iterating over the joint types j ∈ J in order to decide which person every detection of this type is assigned to, a greedy approach is employed, in which an assignment problem is solved with a cost matrix consisting of the distance between the embeddings corresponding to unassigned detections and the average embedding values of the joints that have been aggregated up until this point. The assignment problem finds a locally optimal choice since it uses only the knowledge about the joints that have been aggregated up until joint j, indicating that it is not necessarily the optimal solution. For associating these joint detections with a person or deciding that a new person is discovered, an embedding threshold value needs to be specified such that only joints within this distance may be grouped together. The authors also favor matching an unassigned joint to a higher confidence detection, that is, if an unassigned joint is close to more than one person by the embedding distance, then it is assigned to the person with the highest score. Such a heuristic assumption is hard, and does not necessarily hold true. A non-learned heuristic approach can work well in scenarios when people do not overlap or occlude each other, but will struggle in more challenging scenarios. For instance, in AE, when deciding if a new person is discovered, the method relies solely on a threshold value. Unlike a learning algorithm, it has no notion of what a person is and it tends to overestimate the number of person poses, many of which are invalid.

3. APPROACH:



3.1. UNIFIED MODEL OF DETECTION AND ASSOCIATION:

Having motivated a learning-based model, we now introduce our approach. We use a recurrent neural network (RNN) as a decoder network intended to decode the extracted features of the Stacked Hourglass network into the final distinct human poses. In general, RNNs have a remarkable expressiveness power whose mechanism design allows for weight sharing across time steps and for predicting a variable-length output. In addition, an RNN is an effective model for predicting variable-length sequences that contain dependency along the domain relevant to the prediction. RNNs have been successfully applied for modeling sequential outputs with dependency along the temporal domain in various tasks such as machine language translation Sutskever et al. ( 2014 2016). The motivation behind using them in these tasks is their ability in learning long range dependencies and propagating global contextual information across the time steps. In the case of multi-person pose estimation, the sequence consists of the individual poses appearing in the image and the RNN decides where in the image to estimate a new person's pose using its state, such that it should not estimate the same person's pose repetitively. Such behavior is prevented by the memory maintained in the RNN hidden state, which accumulates information in its internal representation as it processes the sequence. Every iteration in the RNN is responsible for estimating a single person's part heatmaps. Since any permutation of the output sequence is valid, we allow the network to decide its ordering as well. This freedom allows the network to learn an ordering that bests fits its objective, based on its current hidden state and input features. The RNN variant used in this task is based on Convolutional Short-Term Memory network (ConvLSTM) SHI et al. (2015) . A ConvLSTM replaces the fully connected layers of LSTM network Hochreiter & Schmidhuber (1997) with convolutional ones and attains the same advantages that convolutional layers do over fully connected layers for image recognition, such as more efficient training due to a reduced number of parameters, better encoding of spatial information and less proneness to overfitting.

3.1.1. TRAINING PROCEDURE:

As input to the LSTM, we use intermediate features of AE which have been extracted from the Stacked Hourglass module in Newell et al. (2017) ] and are attuned towards part detection. The input is fed-forward through several ConvLSTM layers which decode the extracted features into the final part heatmaps. Additionally, in order to further refine the heatmap predictions, we introduce a feedback loop which takes the distinct confidence maps for each person and concatenates them along with the rest of the input features to the LSTM in two consecutive iterations. The motivation behind introducing this loop is to encourage the network to produce a single peak in each of its estimates of the disjoint confidence maps. It is noteworthy that since the input is fed again through a feed-back loop to the same network, there is no overhead of additional parameters. The hidden state of the last LSTM unit is then used as input features to a convolution layer at the output, which yields J heatmaps belonging to a single person. In every iteration t of the LSTM, J confidence maps for a single person are inferred directly. If ẑ poses are predicted, then the LSTM is unrolled 2 × (ẑ + 1) times before it finds all the poses, where the additional ẑ iterations are for the refinement feedback loop. Each value in the heatmap Ĥt,j represents the network's confidence about the corresponding pixel being the location of joint j. The ground-truth (GT) confidence maps H t,j ∈ R m×n for person t and joint j are created with a 2D Gaussian distribution centered at the GT location of the joint with a small variance. Unlike in Newell et al. (2017) , the GT heatmaps are created disjointly for each person and are not aggregated into a single heatmap. The number of GT heatmaps is therefore equal to z × J instead of J, where z is the true number of people in the image. Accordingly, a GT heatmap H t,j contains a single peak corresponding to the joint's location. This implies that a single prominent peak should be predicted in a single output heatmap Ĥt,j as opposed to k peaks in shared heatmaps. In order to halt the LSTM, we need an additional prediction value that indicates when all poses present in the input image have been estimated. To achieve that, we predict an additional value pt ∈ [0, 1] associated with Ĥt , which signifies the network's confidence in its current prediction being a new valid pose and halt when p < ω for a predefined threshold ω ∈ [0, 1]. This value is calculated using an additional fully-connected layer with the hidden state matrices from the LSTM layers as input, followed by a sigmoid function that converts the output to a valid probability value in the range [0, 1]. The GT for these values is a vector p created as follows:

p =        [1...1 z 0] ẑ ≤ z [1...1 z 0...0 (ẑ-z)+1

] otherwise (1) the length of the vector |p| = max{ẑ, z} + 1, where the purpose of the additional iteration is providing knowledge to the network about the stopping criterion. During training, we have set the stop probability threshold to ω = sigmoid(-1) ≈ 0.26. For a threshold value close to 0.5, false negatives such as blurred people can be missed, and for a threshold value close to 0.1 more false positives are predicted.

To learn the part heatmaps and the stop probability, we add two loss terms on top of the LSTM, the first is the squared l 2 loss between the distinct confidence maps per person and corresponding GT heatmaps. And the second term is calculated using the cross entropy loss between the prediction p and vector p defined in eq. 1. During training, the network may fail to identify all individuals, i.e. ẑ < z, in this case we let the RNN iterate until (z + 1) to obtain all values of p, but penalize only heatmaps with pt ≥ ω. Conversely, if the number of people is overestimated, i.e. ẑ > z, we take into account only heatmaps until z. The motivation is to encourage the network to learn to predict the correct number of people and to make the correct prediction of the heatmaps from the first attempt, i.e. before overestimating z. This reasoning is supported by additional experiments in table 4.

3.1.2. APPENDING A SECOND LSTM:

The order in which the poses are estimated by the LSTM is determined by its weights, while the network will implicitly try to learn an ordering that best benefits its objective. Since ordering is irrelevant in pose estimation, it is not clear how to define a GT permutation for a given image. As such, if an image contains k persons, explicitly optimizing over ordering entails searching in a solution space of k! permutations, which is intractable for a large k. However, since in LSTMs the weights are shared across all iterations, as opposed to feed-forward models, explicitly optimizing over ordering as done in feed-forward models Rezatofighi et al. ( 2018) is not required. Predictably, when we implemented our approach using a feed-forward CNN from samples with a fixed number of people, the approach broke down. This is caused by discontinuity in assigning a certain ordering to the network's outputs, where a certain weight is attuned to a specific output, as shown in Zhang et al. (2019b) . Several works have tackled the ordering problem in deep learning. Vinyals et al. (2015) have shown that in Seq2Seq the ordering in either the input or output affects performance. Rezatofighi et al. ( 2018) estimate the posterior distribution of all permutations using alternating optimization during learning, which requires searching in a solution space ok k! for a permutation size of size k. In Murphy et al. (2019) , the authors propose permutation sampling to overcome the complexity of searching in the entire permutation space and show that it has similar convergence properties to an optimal solution with SGD. In order to examine the optimality of the prediction of the first LSTM (LSTM1) and compare it with random sampling, we introduce a second LSTM (LSTM2) whose input at every iteration t is a concatenation of the intermediate features and the individual HM Ĥt . The advantage of the second LSTM is twofold. Firstly, it enables further refinement of the predicted poses. Secondly, having the poses in a sequence provides a simple way to enforce a certain ordering at LSTM2's output. For instance, we can feed the poses in a random, inverse or canonical ordering, and then force LSTM2 to output a refined heatmap of its input heatmap in the same ordering as the input.

3.2. LOSS TERMS:

Since LSTM1 is not constrained to output the poses in any particular ordering, in order to compute the heatmap loss, the pose instances need to be associated with the correct individual GT heatmap. This can be done by solving an assignment problem using the Hungarian algorithm, in which given the cost matrix C ∈ R z×ẑ , the problem requires finding an assignment S ∈ N r such that r t=1 J j=1 ||H st,j -Ĥt,j || is minimized, r = min{z, ẑ}. Each element in the cost matrix C is given by C kk = J j=1 ||H k,j -Ĥk ,j ||, which is the sum of distances between each pair of a person's heatmaps in the prediction and GT. Taking the sum of the pixel-wise distance between each pair of heatmaps in the GT and output is less sensitive to inaccurate predictions than taking the distance between the maximum activations.The loss term of LSTM2 consists of the heatmap loss between the GT heatmaps reordered in the same input permutation u. Ĥ1 , Ĥ2 denote the HM output of LSTM1 and LSTM2 respectively. In summary, the total loss is a sum of all loss terms:

EQUATION

EQUATION

and all loss terms are weighted equally. Rearranging the GT heatmaps based on the input permutation enforces the input permutation at the output. Figure 3 illustrates the approach.

To obtain the final joint location x j , an arg max operation is applied on each joint's heatmap, i.e.

x j = arg max ( Ĥt,j ). We specify a threshold value τ = 0.015, such that if Ĥt,j (x) < τ , the joint is discarded. For further refinement and in order to provide insight to the optimality on the ordering decided by the LSTM, we add an additional LSTM whose input is a concatenation of the AE features is the assignment of the GT heatmaps, which appear in some arbitrary order. In this example, z = 8 and ẑ = 6, hence s 7,8 = -1. and an individuals heatmap Ĥt in each iteration. The loss on the second LSTM is the square loss between the output Ĥ2 and the GT heatmaps H.

3.2.1. WITHIN-IMAGE BATCHING USING APPROXIMATE SGD:

For an image with a large number of people, a GPU will run out of memory. Our 1080 Ti machine for instance supports a maximum of 8 iterations. Simply cutting and stopping the LSTM after T = 8 iterations despite pt ≥ ω means losing within-image person samples. Accordingly, in order not to overlook these remaining samples, we apply within-image batching, wherein we allow the number of iterations to grow until p < ω or until a predefined number of iterations is reached (suitably the maximum number of people in a dataset), but we backpropagate the accumulated gradient after every T = 8 iterations, where T denotes the mini-batch size. The LSTM's hidden state is not cleared after every T , since it will be needed to keep track of poses that have already been estimated, enabling the network to pick up estimating the remaining poses from where it last backpropagated. In order not to assign the same GT pose to a predicted pose again, we remove the assigned GT heatmaps from the vector of GT heatmaps. The assignment in this case is not necessarily optimal, as it may happen that an assigned GT heatmap in the first mini-batch should rather be assigned in the second mini-batch, in practice, however, we observed that this is not an issue. The subsequent gradient in this case no longer depends on the earlier hidden states in the chain rule. The algorithm is described in alg. 1. Note that if we ignore the remaining within-image samples by simply stopping after only 8 iterations, the performance drops by only 0.7 in the average precision (AP).

3.3. IMPLEMENTATION:

We implement our approach in Pytorch on top of the publicly available library1 of Newell et al. (2016) . The AE model had been trained on COCO dataset, and we use the same dataset for training our model. The input to the AE model is an image resized to a 512 × 512 resolution, and its output is a set of maps M ∈ R 4×68×m×n , m = n = 128, where 4 corresponds to the number of 

4. EXPERIMENTS:

We evaluate our model model on two datasets, COCO validation dev and OCHuman validation and test sets. We summarize the results on COCO in table 1. On COCO, the gap in the single scale performance is larger than in multi-scale (MS) inference and the heuristics of AE seem to benefit more with from MS. It can also be seen that current top-down (TD) approaches outperform bottom-up (BU) approaches on COCO. Coco, however, contains too many easy examples where the persons do not occlude or overlap with each other heavily, therefore does not reflect the performance of TD approaches in more challenging real-life scenarios. In contrast, OCHuman dataset contains

Input: extracted features F, H GT , P GT Initialize Ĥ ← [], P ← [], t ← 1, S ← (0)

1 while true and t < MAX ITERATIONS :

2 Ĥt ← unif orm random(0, 0.1) AE Newell et al. (2017) , the gain in accuracy is larger than on COCO since the separation of highly overlapping people is more difficult. In Zhang et al. (2019a) , through evaluation of person detection on OCHuman, the authors show that for human detection in complex scenarios, thanks to the distinctiveness of the human pose, pose skeleton features obtained by a BU approach should be used, instead of the reverse direction employed in TD approaches, where bounding-box proposals of two heavily overlapped people will likely result in eliminating one of them when applying NMS.

3 i ← 0 4 repeat : 5 B ← concat(F, Ĥt ) 6 Ĥt , pt , S ← LST M (B, S) 7 i ← i + 1 8 until i = 2 9 Ĥ.

4.1. ORDERING:

By adding a second LSTM, examining the ordering effect on performance has become straightforward. The optimal results are obtained when we use the same ordering determined by the first LSTM. This indicates that a data-driven learned ordering is nearly the best we can do at this complexity. Feeding the heatmaps in an inverse ordering degrades the performance substantially. Canonical ordering obtained by sorting based on the maximum activation of the first LSTM yields a low precision too. In table 3, the experiment "learned" is the permutation decided by LSTM1. Experiment "random" is a random permutation of LSTM1's output. "Confidence" is the confidence value of LSTM1 sorted in descending order and canonical is sorting of the permutation based on ascending order of the joint location (based on the maximum activation of every joint's heatmap).

It can be seen that learned performs best followed closely by "random". Sorting by the confidence value is not optimal since the network does not always output poses in descending order of confidence, indicating that the ordering does not necessarily always correlate with the confidence value. "Inverse" shows the importance of ordering and that while the ordering of LSTM1 might not be optimal, the inverse solution is very detrimental. "Canonical" shows that enforcing a certain ordering is also detrimental, since the ordering that most benefits the network does not depend on the location for instance. We presume that a favorable ordering is one in which easy poses are estimated first, followed by hard ones. Hardness of a pose can be assessed by conditions such as resolution, scale, blurriness and illumination. In such an ordering, the network can use the accumulated contextual information in its hidden state in order to estimate an occluded person following an occluding person for instance. If LSTM2 consists of a larger number of hidden units, there's sometimes an advantage in a random permutation over a learned permutation (see 7 in the appendix). With two LSTMs, if the first LSTM does not have a feedback loop and the second does, the performance drops substantially, indicating that it is more critical to refine the heatmaps at an earlier stage. We designed an additional experiment in which we have a single loss on only LSTM2 only. In this case the performance drops from 60.3 to 53.2 indicating that simply increasing the number of layers is not advantageous and that having an initial prediction is very helpful for the additional layers.

To get a fair comparison between an exact SGD (E-SGD) and approximate SGD (A-SGD), we train our network with images that have a maximum 8 people so that approximate SGD, in which we backpropagate after estimating 4 persons and solve the assignment with respect to the first subset of 4 poses, and then the second subset, A-SGD achieves 56.8 while E-SGD achieves 56.2, which is an interesting result showing that the long term dependencies propagated by the gradient are not always beneficial. 

4.2. ABLATION STUDIES:

There are two cases in mispredicting ẑ: (1) ẑ < z, and then we either take into account heatmaps (a) only with pt ≥ ω, or (b) all heatmaps including pt < ω.

(2) ẑ > z and then we solve the assignment (a ) for only the first z of ẑ heatmaps, or (b ) for all ẑ heatmaps. We train using all four configurations and summarize the results in 4. Configuration (aa ) performs best, confirming our intuition that through (a) it is better to disregard predictions in the heatmap loss that the network is not confident about, and let the network improve its prediction by backpropagating the stop loss where max{|ẑ|, |z|} is taken (eq. 1). Through a , it is better to penalize only heatmaps associated with a correct prediction of p. Note that our ablative studies are conducted in single-scale with a single LSTM but generalize to two LSTMs and multi-scale inference too. For examining the effect of the feedback loop, we draw a comparison with a model that has been trained without the loop and observe that the network suppresses redundant or wrong peaks in the output. The difference is illustrated visually using output heatmaps in figure 4 .

5. CONCLUSION:

We have introduced a unified differentiable model for part detection and association. Our approached surpassed the performance of the baseline on COCO and achieved state-of-the-art results on OCHuman. We have provided an efficient implementation that allows controlling the withinimage batch size thereby enabling training on memory-constrained GPUs. We have introduced a method that allows enforcing ordering in the input and given insight to their performance in different settings. Through our experiments, we were able to emphasize the importance of a unified learning-based framework for detection and association, in particular for difficult cases of human poses such as occlusions. Additionally, we have shown the limitations of top-down approaches in such cases and reinstated the importance of bottom-up approaches.

A APPENDIX:

A.1 ADDITIONAL ABLATIVE STUDIES Table 5 shows the advantage of having an feedback loop, where the performance with 3 loops slightly drops due to a larger cumulative gradient.

Table 6 shows the effect the in-image batch sizt T has on performance. T = 1 indicates cutting the dependency between each predicted pose in the image affects the performance very negatively.

In fig. 6 , we plot the precision versus the stop threshold ω. During training, for too low a threshold, outliers may be penalized, whereas with too high a threshold, difficult cases like a blurred or lowresolution person can be missed. Too low a value, we would be penalizing outlier predictions, too high we would be missing potentially difficult cases like a low-resolution person. The results of a random permutation versus a learned permutation do not appear to be conclusive, as evident in table 7 . For example, when increasing the number of hidden units of LSTM2, we still get comparable results to the LSTM with two hidden units, but in this setting, sometimes a random permutation performs better like in the experiment with 3 hidden units or with 4 hidden units on COCO. This indicates that there is no clear winner, unlike for instance an inverse permutation that would performs very bad in all settings. 

:

https://github.com/princeton-vl/pose-ae-train

Figures and Tables:

); Cao et al. (2017); Newell et al. (2017); Iqbal et al. (2017); Doering et al. (2018); Raaj et al. (2018); Papandreou et al. (2018); Haoshu Fang & Lu (2017); Papandreou et al. (2017); Huang et al. (2017) directly estimate joint locations for individuals all at once without distinction, and require assembling the joints into individual poses subsequently. Several works have proposed to merge the joint estimates into poses by generating a fully-connected graph Pishchulin et al. (2016); dee (2016); Iqbal et al. (2017); Insafutdinov et al. (2017) based on the joint estimates, and solving a matching problem by utilizing ILP (Integer Linear Programming). A closely related work is Wang et al. (2018) who introduce a minimum weight set packing formulation to solve the association problem of the part detections while applying Nested Benders Decomposition to achieve a more efficient inference time. Other works Cao et al. (2017); Newell et al. (2017); Raaj et al. (

), image captioning Donahue et al. (2015), action recognition Richard et al. (2017), and motion prediction Martinez et al. (2017). They have been additionally employed for various visual tasks where the sequential dependency lies in the spatial domain, such tasks include object recognition Visin et al. (2015); Mnih et al. (2014), semantic segmentationVisin et al. (2016), scene labeling Byeon et al. (2015); Pinheiro & Collobert (2014), instance segmentation Romera-Paredes & Torr (

Figure 1: The first raw are the confidence map predictions of the RNN, where each person i is a sum of the J confidence maps predicted at iteration i. s k = i indicates H k is matched with Ĥi . S = {s} r 1

Figure 2: An example output of a single LSTM iteration, in which J heatmaps are predicted for a single person. We overlay them as is on the image, and sum them up for the last image to show correspondence for all the joints of the person.

Figure3: Overview of the approach. AE network predicts joint heatmaps collectively. It follows this stage by a heuristic-based association using the embeddings. Instead, our approach estimates an individual human pose directly in every iteration t. The input to the first ConvLSTM is a concatenation of the current hidden state from the previous iteration, the intermediate features extracted from the AE network, and the LSTM prediction of the individual heatmaps using the feedback loop. The input to the subsequent layers is a concatenation of the output of the previous layer and previous hidden state. The features used to calculate the confidence value p are the concatenation of the maximum activations of the hidden state from each layer (Due to space, LSTM2 is not shown).

Figure 5: Qualitative results from COCO in the first row, and OCHuman in the second row[add examples from OCHuman. maybe also show results of the other approaches

Figure 7: Qualitative comparison between Associative Embedding and our learning-based approach.

Figure 8: Qualitative results on OCHuman including some failure cases such as in the last row where the poses are mixed or slightly overestimated such as in the third row.

append( Ĥt ); P .append( pt ) The training algorithm. Line 17 removes GT heatmaps that were last assigned. Results on COCO validation set. only difficult cases of occluded and interwined persons and the average IoU of the bounding boxes is 67%. It consists of 4731 images that comprise a validation and test set with a total of 8110 annotated humans. The purpose of this dataset is to examine the limitations of human detection in highly challenging scenarios. Accordingly, it does not contain training samples and is intended to be used for evaluating existing models. The results are summarized in table 2. The results on this dataset show the limitations of approaches in which association is not part of the learning algorithm, and of TD approaches. Our approach achieves state-of-the-art results on OCHuman and outperforms both TD approaches on both the validation and test subsets although both Haoshu Fang & Lu (2017);Xiao et al. (2018) use a much stronger backbone architecture. Compared to

Table content: <table><tr><td>10</td><td>if pt &lt; ω or t mod T = 0:</td><td/><td/><td/><td/></tr><tr><td>11</td><td>assignment ← match( Ĥ, H)</td><td/><td/><td/><td/></tr><tr><td>12</td><td>H ← H[assignment]</td><td/><td/><td/><td/></tr><tr><td>19</td><td>end while</td><td/><td/><td/><td/></tr><tr><td/><td>Algorithm 1: Method</td><td>AP</td><td>AP 50</td><td colspan="3">AP 75 AP M AP L</td></tr><tr><td/><td/><td/><td>Bottom-Up</td><td/><td/></tr><tr><td/><td>OpenPoseCao et al. (2017)</td><td>61.8</td><td>84.9</td><td>67.5</td><td>57.1</td><td>68.2</td></tr><tr><td/><td colspan="2">PersonLabPapandreou et al. (2018) 68.7</td><td>89.0</td><td>75.4</td><td>64.1</td><td>75.5</td></tr><tr><td/><td colspan="2">AE singlescaleNewell et al. (2017) 56.6</td><td>81.8</td><td>61.8</td><td>49.8</td><td>67.0</td></tr><tr><td/><td>Ours singlescale</td><td>60.1</td><td>82.2</td><td>65.4</td><td>51.9</td><td>72.5</td></tr><tr><td/><td colspan="2">AE multiscale Newell et al. (2017) 65.5</td><td>86.8</td><td>72.3</td><td>60.6</td><td>72.6</td></tr><tr><td/><td>Ours multiscale</td><td>66.8</td><td>85.2</td><td>73.0</td><td>59.7</td><td>77.3</td></tr><tr><td/><td/><td/><td>Top-Down</td><td/><td/></tr><tr><td/><td>Mask-RCNN He et al. (2017)</td><td>63.1</td><td>87.3</td><td>68.7</td><td>57.8</td><td>71.4</td></tr><tr><td/><td>G-RMI Papandreou et al. (2017)</td><td>64.9</td><td>85.5</td><td>71.3</td><td>62.3</td><td>70.0</td></tr><tr><td/><td>CPN Chen et al.</td><td>72.1</td><td>91.4</td><td>78.9</td><td>68.7</td><td>77.2</td></tr><tr><td/><td>RMPE Haoshu Fang &amp; Lu (2017)</td><td>72.3</td><td>89.2</td><td>79.1</td><td>68.0</td><td>78.6</td></tr><tr><td/><td>CFN Huang et al. (2017)</td><td>72.6</td><td>86.1</td><td>69.7</td><td>78.3</td><td>64.1</td></tr><tr><td/><td>MSRA Xiao et al. (2018)</td><td>73.7</td><td>91.9</td><td>81.1</td><td>70.3</td><td>80.0</td></tr></table>

Evaluation on OCHuman Zhang et al. (2019a) validation and test sets. SS/MS indicates single/multi scale, respectively, and * indicates a top-down approach.

Table content: <table><tr><td/><td colspan="5">learned random confidence inverse canonical</td></tr><tr><td>COCO</td><td>60.1</td><td>59.3</td><td>58.2</td><td>42.1</td><td>38.8</td></tr><tr><td>OC-val</td><td>39.4</td><td>39.1</td><td>37.7</td><td>25.4</td><td>23.4</td></tr><tr><td>OC-test</td><td>32.5</td><td>32.2</td><td>30.4</td><td>30.2</td><td>20.6</td></tr></table>

AP with several ordering variants on the input to LSTM2.

Table content: <table><tr><td/><td colspan="4">(a, a ) (a, b ) (b, a ) (b, b )</td></tr><tr><td>COCO</td><td>58.7</td><td>57.1</td><td>58.0</td><td>54.7</td></tr><tr><td>OC-val</td><td>37.8</td><td>35.7</td><td>36.9</td><td>35.8</td></tr><tr><td>OC-test</td><td>31.7</td><td>30.5</td><td>31.0</td><td>30.1</td></tr></table>

Results with different configurations in calculating the heatmap loss.

Table content: <table/>

Examining the influence of the number of hidden units on the performance with a learned and random permutation at LSTM2.

Table content: <table><tr><td>AE</td><td>Ours</td><td>AE</td><td>Ours</td></tr></table>

Peer review (iclr2020):
The paper proposes an end-to-end solution to accurately perform multi-person pose estimation. Existing approaches can be characterised as being either *bottom-up* (parts localisation then person localisation) or *top-down* (the opposite direction). In particular, existing techniques in the bottom-up regime use greedy heuristics to refine the predictions, which makes the overall optimisation process unclear. The proposed technique performs both parts jointly through the proposal of an end-to-end differentiable model (a CNN-LSTM on top of an image*s extracted features) to perform multi-person pose estimation. - I found the paper quite difficult to follow in some parts. For instance, it disrupts the flow of the paper by having related works / lit review while describing the model (you should put this in a separate section). It would have also been nice to have more equations complementing the text. - Many of your citations need to be in parentheses (i.e. use citep). - Figure 3 should be moved up to be closer to the text which explains the training procedure, right now it*s too far down. - You should also show LSTM2 in the figure, or have it in its own figure - If the final output of the LSTM are the confidence maps for each person, you should make this clear in the diagram, e.g. drawing arrows from each of the z time steps (since each of the z time steps corresponds to heat map predictions for one person). - Refer to Algorithm 1 in the text. What is this algorithm referring to? LSTM2? - Center table captions - I am confused about how LSTM1 and LSTM2 interact. When I first read the paper I thought that the LSTM in Figure 3 was *LSTM1* and that you were replacing this by an *LSTM2*. - It says in the training procedure that 	ilde{z} poses are predicted (or more technically, 2x that amount +1 because of the refinement loop). This to me implies that 	ilde{z} can change for each input over the course of training. You also have a fully-connected network which predicts the ground truth stopping vector p (whose length is determined by max(z,	ilde{z}+1) and therefore appears to be of variable length), but because this is a fully-connected network how can p be variable length? It doesn*t seem clear from the writing. Perhaps I can suggest how I would have implemented this: - Run the LSTM for T=z+1 iterations, where z is the ground truth number of people in the image. The input to the LSTM at each iteration is simply the input x from the AE concatenated with the output heat map from previous timestep (like LSTM2). The output at each iteration t will be a tuple (	ilde{H}_t, 	ilde{p}_t}), where 	ilde{p}_t in [0,1] is the probability that we halt the LSTM. For iterations 1..z, the GT value for 	ilde{p}_t will be 0 and for iteration z+1 it will be 1. This would not require a refinement network nor a separate network to estimate stopping probabilities. - I find the ordering issue to be an interesting problem. Unfortunately, as you have mentioned in Section 3.1.2, it still requires one to define permutations of the poses during training. Perhaps future work could somehow utilize permutation-invariant modules (e.g. DeepSets).