Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 3: Borderline
review_content: summary: The paper introduces anatomically constrained losses in 3D human pose estimation; and joint training with two groups of datasets where one of the groups has annotated labels of the joints providing supervisory signal (for “task loss”) and the other one uses the anatomical constrained loss penalizing asymmetric limb lengths, implausible bone lengths, and implausible joint angles. The authors pose this as a domain adaptation where the source domain is the set of point clouds obtained from the first group of images of humans lying in the bed, and the target domain point clouds obtained from the second group of images of humans lying in the bed but occluded by the blanket cover of varying thickness. The comparison of the proposed method against various other methods show that the proposed method can provide better results.
strengths: Introducing anatomical constraints in losses and using joint training to learn a common feature extractor is a simpler approach but looks quite effective compared to adversarial regularizer. 

The method is compared against several other state-of-the-art methods and the results of show that the proposed approach is quite effective for this particular dataset and setup.
weaknesses: The title and the positioning of the paper as domain adaptation seem to be a bit too broad than the setup where the paper focuses on images of a person covered and not covered in blankets. While this is technically “domain adaptation” for a very specific source domain and target domain type, it is not clear how much of the benefit this method brings to other kinds of domain shifts, especially population shift (age, pathology etc.) where the anatomical prior obtained from the source domain may not work well in the target domain. 

If other forms of domain shift where there is no occlusion are considered, it is not clear how much benefit the proposed method brings compared to other domain adaptation methods because the input data would have more information about the joints than the occluded scenario. The paper does not explore this in the experiments or at least discuss it in the discussion. 
detailed_comments: One of the motivations for using point clouds is mentioned to be privacy reasons. However, it is not clear how this translates in the practical use case where estimating point cloud step needs to use the original images. 

The paper mentions that the prior knowledge about human anatomy in the unlabeled scenario has not been fully exploited. Perhaps, the positioning of the work may be improved by adding literature on anatomical constraints with template matching that is common in medical imaging. It has been explored for human pose estimation as well [A] 

Sec 4.1 train-test split has ensured no leak w.r.t patients but it seems like leak w.r.t to the pose is not considered. It would be interesting to see how well the method performs when test images contain poses that are not seen at all in training. 

Sec 3.2 explanation about splitting the neural network into feature extractor g and network heads h is a bit confusing. Are there two heads or only one head for the source domain while using g for the target domain? I think f(h(g)) is used for both but h is not updated with anatomical loss. A figure showing g and h would be helpful.  

Minor typos: 

Sec 3.1 “Let further denote …" Grammar 

Just before Eq (6), fix -> fixed 

Schmidtke, Luca, et al. "Unsupervised Human Pose Estimation through Transforming Shape Templates." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. 
recommendation: Poster
review_title: null

final_score: rating: 5: Strong accept
review_content: summary: This paper tackles the problem of 3D pose estimation in bed-ridden patients, with a 3D point cloud as an input. One of the main difficulty of the task is the impossibility to directly annotate data, due to bed-covers. 
What the authors do is to resort to domain adaptation, using as a source a specially crafted dataset with and without cover (so that the pose is known and available), and then targetting (fine-tuning) a set with a cover (and without the pose available). As such, we go from the "no cover" domain, to the "cover" domain.

The novelty here is to regularize the fine-tuning using anatomical priors (symmetry, limb size and joints angles), which are encoded as series of constraints into the combined loss function. This improves the results by penalizing unrealistic predictions. Such constraints are not needed as inference time, this is only done during the fine-tuning stage.

The framework presented is very simple and elegant, and can be easily extended to encode other priors, and/or corner cases. 


---
I do not see a field for that this year, so I'll state it here: I am not familiar with the pose estimation litterature, and if the authors had decided to ignore a whole class of papers I would not be aware of it. Hence the '4' on my own confidence score.
strengths: - The paper is very well written and well motivated.
- Simple and elegant framework
- can be extended with other priors, and/or plugged into existing methods as additional regularization
- compare to numerous other domain adaptation methods (this should not be down-played, as it is often the most time-consuming task)
- The experimental protocol is sound
weaknesses: - The training regiment could be described in more details. At the moment, it is not clear how exactly the two sets (target and source) are used. 


No more weaknesses, but I have to fill the 200 chars count.
detailed_comments: While the paper is very well explained and motivated overall, I find that Section 3 falls into the (somewhat common) caveat of presenting the method, without necessarily motivating the details. My personal take would be to first present it as a constrained optimization problem, and _then_ mention how they are embedding at training (slightly tuned alphabet):

$\\begin{align}
\\min_{\\theta} \\sum_{s \\in \\mathcal S} & \\sum_{k \\in \\mathcal K} ||\\hat y_s^{(k)} - y_s^{(k)}||_1 \\\\
\\text{s.t. } & ||b^{(r)}||_2 = ||b^{(l)}||_2 & \\forall (b^{(r)}, b^{(l)}) \\in \\mathcal B_l^t, \\forall t \\in \\mathcal T \\\\
        & l_i \\leq ||b\_{t, i}|| \\leq {u_i} & \\forall i \\in \\{1, ..., N_b\\}, \\forall t \\in \\mathcal T \\\\
        & l^{(\\alpha)}\_{ij} \\leq \\frac{b_i}{||b_i||_2} . \\frac{b_j}{||b_j||_2} \\leq u^{(\\alpha)}\_{ij} & \\forall ij \\in \\mathcal B^t\_{\\text{conn.}}, \\forall t \\in \\mathcal T
\\end{align}$

(With $\mathcal S$ the source set, $\mathcal T$ the target set, and the rest should be sufficiently close to the initial notation to be understandable.)

It would then be easier to discuss (and motivate):
- why a quadratic penalty is used for the equality constraint (Eq (2)), while a linear penalty is used for the inequality constraints (Eq (3) and (4));
- would make it easier to relax the symmetry equality for patients with non-symmetric limbs;
- as you discuss very interesting other priors in the conclusion (the overall consistency across all joints), I suspect that optimizing that function might require more powerful optimization methods, so decoupling the model from the optimization might help;
- how the regularization could easily be applied to the source set as well;
- highlights more how the bounds for Eq. (3) are patient-independent, but could easily embed other information (as you mention in the conclusion).

---
Minor:
- few notation clashes, for instance the $l$ of $N_l$ is re-used for $l_i^b$; $b$ in $N_b$ and $u_i^b$ reads a bit like an index. It slightly hamper the reading, and could easily be avoided by using other alphabets (e.g. greek) or fancy formatting (mathcal, mathbb, mathfrak)
- A Table 2 containing the results from the last paragraph of 4.4 is needed, I almost missed those (really interesting) results
- Reminding in 4.2 that your code is available online can be useful, for readers that haven't checked the abstract yet or simply forgot about it
- page 4: should either be "or -- in other words -- what" or "or---in other words---what" (en or em-dash, not a single dash)

recommendation: Best Paper Award
review_title: null

final_score: rating: 5: Strong accept
review_content: summary: The paper addresses domain transfer for human pose estimation. The scenario is simple: there are only pose datasets with humans without a cover, we want to generalize to subjects that have a blanket covering them. We do not have ground truth poses for this second scenario. In this context, the authors introduce additional anatomically motivated losses to regularize the network outputs with "plausible" poses, thus introducing domain (anatomic) knowledge (even if explicit values are again derived from the dataset). The paper has a strong baseline set of 8 methods and achieves superior performance overall and across 6 of 8 individual "end points", i.e., the method outperforms the state-of-the-art. The errors significantly increase with distance from the core and "size" of the target point, which is plausible, as this information will be hard to uncover from the data.
strengths: 1. The paper follows best-practice for dataset hygiene and splits. Dedicated and plausible decisions are made for training/test/validation and even domain splits. I wish every paper was following this basic best-practice. You can even add that dividing the training data into split for the different domains is necessary to reduce confounders (otherwise could cause overfitting to unrelated features).
2. In total 8 different baselines allow for a thourough evaluation and comparison of the method. Very much appreciated!
3. The paper outperforms the state-of-the-art clearly.
4. Even Results of combinations of the proposed loss and baseline method are reported, however not included in the Table 1.
5. Very well written and structured, easy to read.
weaknesses: Major
1. The impact of the work is probably somewhat limited. At least for me it is unclear, how this would translate to other applications. The authors suggest landmark detection as a more general scenario for its application, but I am not aware of any landmark detection secario that uses point clouds. So I believe the solution is mostly limited to this specific problem.
2. The problem seems plaugued by limited information. The authors even experienced difficulty to estimate deviations from the mean pose in the initial training. I would like to see a new baseline added to understand the "quality". The trivial baseline: the mean pose. In my mind, this is equivalent to a coin-flip in classification and helps understand whether this approach is actually going anywhere. I would anchor the mean pose on head and core. If the error of the source training (worst-case) is less than this mean pose approach, a simple comment suffices.

Minor
1. I don't like the naming 
$\cal{L}_{bone}$,
I'd prefer this to be something like 
$\cal{L}_{length}$
 this causes a bunch of little changes throughout the paper. [sorry openreview is bad at multiple equations per line :( ]
2. Results of combinations of the losses with the baselines are not reported in Table 1. It is unclear to me, why results are reported in the text, but not in the Table -- or even an additional Table in the appendix if space-constraints were the issue.
3. Discussion of Limitations:
-  What happens, when the subject violates the anatomical constraints? Do you expect any performance difference in methods then? Even is there a performance difference within the dataset? Examples: amputated/deformed limbs, children/very small humans
- Does the symmetric limb loss introduce a bias towards people, that actually do have asymmetric limbs (as part of a pathelogical condition)? Likely, this will not be covered in the dataset, but from a clinical standpoint, this is relevant.
- What are the limitations of the approach?
- I would expect different priors for patients covered with a blanket and without a blanket, at least this should be disscussed.
4. page 6 para 2: The statement of "challenging setting ... for human observers" requires qualifiers: 1. based on depth images? or RGB? 2. evidence (appendix) or add "in your experience" or citation.
5. page 6, para 1 last sentence: I find this a bit unclear. What happens here? how does this processing work?
detailed_comments: 1. Why are those hard limits (bone and angle loss) good? Would a statistical distribution loss (e.g. KL-divergence on the observed distribution) not be a better idea?
2. Loss Normalization: in your formulation, there is a fixed factor for the loss, but the magnitude of the loss might be very different, specifically, for angles vs. lengths. Maybe losses should be normalized to account for this phenomenon of different magnitudes.
3. Your architecture already is an Encoder-Decoder design. If you wanted to ensure the network is more constrained to "plausible" poses, you could randomly sample the latent space and apply your plausability constraints on the "random poses". I have actually implemented such designs for encoder-decoder and auto-encoder designs and achieved good results with that.
recommendation: Oral
review_title: null

Meta-review: metareview: This paper introduces anatomically constrained losses for 3D human pose estimation, in a domain-adaptation setting. The three reviewers agree that this is an excellent and well-executed work, and I concur with this. The method is simple/elegant/well-motivated, and the comprehensive experiments/comparisons show clearly the usefulness of embedding this type of prior-knowledge constraints as loss functions. The paper is well-written, and the authors provided a detailed answer to the reviews, including additional ablation studies.

confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)