Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an early attempt at learning geometric lengths from trails and explores how to transfer the learned geometric knowledge to solve a different task. Besides offering a problem formulation, this paper presents an evaluation framework based on finding evaluation correlations against ground truth lengths. 
strength_and_weaknesses: Strengths
----------

Ideas are clearly elaborated through the paper with adequate support of intuition in explanations.

Elaboration of experimentations is adequate for an early exploration of learning object lengths. 

Initial results are promising on the capability of learning lengths from trials.


Weaknesses
-------------
This paper is an early exploration of the topic. Consequently, the experimentation is limited to a single dataset and simplistic tasks. Can this methodology be used to improve challenging tasks such as relative pose estimation and active visual navigation?

clarity,_quality,_novelty_and_reproducibility: There are no major issues related to the clarity of this paper as the idea is simple and intuitive.
summary_of_the_review: Overall this is initial research. My main criticism is the lack of exploration or at least discussion on how to use this idea in more complex problems. I would like to see at least some discussion in this regard.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper explores an under-investigated problem, that of automatically discovering geometric quantities that indicate the potential success or failure of fitting tasks. These include passing an object through a tube-like container, putting an object in a drawer or a sphere, etc. The authors use the term eigen-lengths for these measurements and explore three approaches for learning them from data, with the help of some manual annotation in the form of specifying the number of relevant eigen-lengths. Several tasks accompanied with relevant data are defined and quantitative results are presented. (Presumably, the data and test suite will be released if the paper is accepted.)
strength_and_weaknesses: Assessing the strengths and weaknesses of this paper is difficult because the problem addressed is under-explored in the literature and the solutions presented here are not mature yet. This seems to be an interesting direction for research, but there are several shortcuts and limitations in the formulation. While this is partially understandable, I have doubts whether this is the right direction and whether extensions of this formulation are likely to be adopted by other researchers.

Strengths

This is an interesting problem and a thought-provoking paper presenting a novel take on it.

The claim that the eigen-lengths are interpretable and universal is true, but also self-evident. Determining if an object fits in different objects depends on it dimensions. I do not consider this a major strength.

The concept of grounding, as defined in the paper, leads to improved performance. 

The comparison of the direct, implicit and grounded methods is informative, but see below about my concerns on the representation.

Weaknesses

The restriction of object motion to translation only appears reasonable for a first attempt at the proposed formulation, but reduces the search for “eigen” lengths to essentially the selection of relevant dimensions of the bounding box. Since both objects and containers/tubes are axis-aligned, the problem is greatly simplified and does not require the learning of affordances other than the concept of support surfaces in some cases. (It is not clear to me if points of contact are detected in the sphere fitting task, for example.)

Success or failure of a given task is determined based on the existence of suitable final placement of the object in the environment. In some cases of interest, the existence of a feasible trajectory is also needed. An object may fit in a cabinet or drawer but there may be no way to put it in there.

The fact that the number of relevant eigen-lengths, S, has to be set manually is a limitation. 

The chosen representation of the objects and environments requires further justification. PointNet descriptors are not well suited for capturing object properties such as dimensions. It seems that some other representation, possibly in the form of random projections, would be more effective.

Section 6.1 is hard to connect to a practical scenario. (It is possible that I am missing something.)

The test set in Table 4 is dominated by jars. I do not see any reason to construct an imbalanced dataset for an unexplored problem.

clarity,_quality,_novelty_and_reproducibility: The concepts are presented clearly, but sometimes at a level that is too abstract.

Failure cases and limitations are discussed in the supplement.

The work is original, to the best of my knowledge.
Reproducing the methods based on the descriptions in the main text and supplement is not an easy task. The use of PointNets for classification and segmentation is briefly mentioned without many other details.

Minor Comment: No publication venues are provided for both papers by Higgins et al. 

summary_of_the_review: On one hand, accepting the paper may inspire other researchers to make progress on an interesting problem, surpassing the baseline algorithms presented here. On the other hand, some of the limitations listed above are serious. I recommend borderline accept for now to give the benefit of the doubt to the authors, but justification for the chosen representations, the use of PointNet, and the restriction to translation-only motion should be provided in the rebuttal. Section 6 should be explained more clearly.

++++++++++++ Dec. 11 update ++++++++++++++++++

After the discussion among the reviewers and area chair and the latest author responses, I am leaning towards recommending rejection. Even though the idea is interesting, I had undervalued the limitations, which outweigh the contributions. The handling of rotation in particular is unsatisfactory. According to the authors’ latest response, rotations in rectangular environments cannot be handled. This is an important limitation. I also do not understand the value of rotating spheres or sphere-shaped environments when the representation is purely geometric. Moreover, the response on the ordering of eigen-lengths so that they correspond across the object and environment is unsatisfactory. I do not see how the network can learn that based on the loss function.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This article is concerned with the notion of eigen-lengths of objects, canonically width, height and depth. The authors argue that those eigen-lengths are key to common-sense reasoning (eg, can object A fit in container B). The authors propose that eigen-lengths should be grounded in the scene geometry. 
The paper attempts to ground the eigen-lenths using a numbe of manipulation tasks. 
Experiments demonstrate that the approach allowed for accurate estimation of the eigen-lengths. 
strength_and_weaknesses: ### Strengths
- The paper is well written and pleasant to read. 
- This is an interesting problem. The extraction of intrinsic properties of objects such as eigen-dimensions is an interesting and important avenue of research. 
- The use of the multi-task learning and task adaptation is well adapted to the problem.
### Weakness
- Although the general concept of eigen-dimensions is intuitive and several examples are discussed, the article lacks a formal definition of eigen-dimensions and their meaning for a range of objects would be useful to clarify the applicability of the approach.
- The broader implication of the results for AI and robotics should be made explicit. What is the possible usage outside of the narrow tasks described in this paper?

clarity,_quality,_novelty_and_reproducibility: This is a very interesting problem, tackling the interesting question of how to what mode of description of 3D objects mediates common-sense reasoning. I found the paper was well written, clear and interesting to read. The proposed approach is, to my knowledge, novel. 

The description of the experiments is clear and well detailed, but it would be important for reproducibillity that the code and data be made available. 

summary_of_the_review: In summary, I found this paper  interesting and pleasant to read. The approach is novel and tackling an important problem for the community. I did not find any major concerns on this paper, although some of the discussion could be improved to clarify the contribution (see weaknesses above)
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new problem setting that learns the geometric eigen-lengths for robot fitting tasks. The authors build a benchmark for this task, propose two learning methods (learning directly from binary supervision and learning with geometric primitive), and test the methods with single-task and multi-task(transfer learning) settings.
strength_and_weaknesses: ###### Pros
- The idea of learning task-specific geometry lengths of the object and the environment is interesting.
- This paper is written clearly, with the motivation and methodology easy to follow.
- The authors build a comprehensive benchmark suite including 6 kinds of tasks, ~1200 object models from ShapeNet, the corresponding datasets, and evaluation metrics.
- The findings of being able to learn length directly from binary supervision are interesting and impressive.

###### Cons
- To apply the eigen-length learning step to robot fitting or other robot manipulation problems, it seems that how to choose and define the geometric primitive and the creteria is essential to this problem. After that step, parameter learning is relatively easy. Therefore, whether the proposed problem setting and method are practical remains unclear. This method design is contrary to the goal of the paper, which aims to rely on the least human prior. 
- To predict geometric information, there are tons of ways, including traditional image processing methods like edge detection, deep learning methods over computer vision like 3D object detection[1], segmentation[2], 6 DoF pose estimation[3] and computer stereo vision system[4] with necessary pre-process and post-process operations. The proposed method for the robot fitting task is one of the extended applications of the geometric information prediction problem, and thus the novelty is limited.
- There are no experimental comparisons to baselines. For example, other robot fitting methods (or simpler, binary-classification methods) that explicitly estimate or implicitly learn the geometric information should be compared; Comparisons of the low-dimensional eigen-length to the high-dimensional latent codes and structured representations discussed in the Introduction section should also be empirically verified.
- The constraints of the problem setting, i.e., only translation (without rotation) of pre-aligned objects and environments is too strict to be feasible in practice.
- For Table 2, should the proposed method be compared with other transfer learning methods instead of direct training without knowing the training tasks?


###### Refs
- [1] Zhou, Yin, and Oncel Tuzel. "Voxelnet: End-to-end learning for point cloud based 3d object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
- [2] Wu, Bichen, et al. "Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.
- [3] Xiang, Yu, et al. "Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes." arXiv preprint arXiv:1711.00199 (2017).
- [4] Wikipedia: Computer stereo vision. https://en.wikipedia.org/wiki/Computer_stereo_vision
clarity,_quality,_novelty_and_reproducibility: ###### Clarity
- The article is well-written, and the authors express their thoughts very clearly.
- However, for some statements, the paper does not provide enough results to verify them, such as lacking the comparison between different kinds of representations.
- The texts in figures could be larger for better readability.

###### Quality
- The quality is good. I believe the authors spent a lot of time and effort to build this benchmark.

###### Novelty
- The novelty is limited.

###### Reproducibility
- If the authors release the benchmark, the results will be easy to reproduce.

summary_of_the_review: This paper proposes an interesting topic, and it is clear to follow.
However, the validity of the problem setting and the comparisons to relevant methods should be addressed more carefully.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of learning low-dimensional geometric embeddings for robot fitting tasks. The paper proposes a new benchmark for robot fitting and investigates different strategies for learning these features, which the paper refer to as "eigen-lengths". Using the proposed benchmark, these strategies are evaluated for different single- and multi-task learning settings.

The paper received four reviews that are largely in agreement that the problem of learning task-relevant representations for robot fitting tasks is interesting. However there was notable disagreement among the reviewers regarding the significance of the paper's contributions with regards to the benchmark and proposed solutions. In particular, Reviewer VXuJ finds the approach to be novel, while two of the other reviewers (rqrT and rLUp) emphasize the limitations of the formulation in light of what the reviewers find to be a overly constrained problem setting and strong assumptions, notably regarding the (in)ability to handle rotations and the differences between the learned representation and traditional geometric features (e.g., bounding boxes). Because of this discrepancy in the initial overall recommendations, Reviewers VXuJ, rqrT, and rLUp met (virtually) with the AC to discuss the merits of the paper. As a result of the discussion, questions were posed to the authors in an attempt to get clarification about the method's generalizability to rotations, as well as the reproducibility of the work. The authors' response suggests that the practicality of the solutions are quite limited due to the inability to handle rotations when the shapes are rectangular. It was then questioned why rotations are important in the context of spherical objects. Future versions of the paper would benefit from a clear discussion of the differences between the learned representations and vanilla geometric features as well as the extent to which the methods generalize to different object shapes.

Note: The review provided by 4TGL is short on details. The reviewer did not reply to or acknowledge the authors' response, nor did they participate in the virtual reviewer-AC discussion. As a result, less weight was placed on their review.
justification_for_why_not_higher_score: The problem setting itself is of little value and the proposed solutions are quite limited. In the AC's opinion, the limitations outweigh the contributions, and the AC believes that most of the other reviewers would agree.
Decision: Reject