Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper presents a framework to synthesize images with hand-object interactions using the 3D model of the object and the hand pose (also in 3D). The paper also claims to introduce a new task named hand-object interaction image generation. A set of experiments in two different datasets showed that the proposed framework outperformed constructed baselines in several metrics, including LPIPS, FID, AUC, PA-MPJPE, and ADD-01.D.
strengths_and_weaknesses: - Strengths: 
    - The paper is well-presented and easy to understand. No obvious typos.
    - The technical novelty looks incremental, but the experimental results and conclusions may be interesting for the community.
    - The proposed framework is well-validated, and the results showed clearly that it is superior to the baselines.

- Weakness:
    - There are two major weaknesses. First, it seems that the proposed framework is too complex to solve the problem. Since it is provided a 3D model for the object and the hand indicating the pose of both, why not apply a photorealistic render algorithm and compose the image using the background of the input image?
    - The second weakness is the novelty of the technical solution feels incremental. Most of the building blocks of the proposed frameworks come from related work (e.g., UNets and SPADE). For instance, after computing a projection of the hand with the missing texture (occlusion-aware topology modeling component), the hand-object generator is basically applying a UNet to inpaint the hands and object texture and using SPADE to take into account the three-dimensional shape.
    - Another concern refers to the baselines and the evaluation metrics, which seem they are not challenging the proposed framework. For example, why not compare against a photorealist render algorithm and not use a pixel-wise error (e.g., MSE) using the ground-truth images to evaluate the quality of image generation?

questions: 1. Were all UNet trained from scratch or pre-trained using some dataset in an inpaint task?
2. It is not clear the fusion step. Is it a learning layer? If so, is it an MLP?
3. Is the hand-object generator training end-to-end? Are all the networks being training simultaneously, or was used a training-wise strategy?
4. There are three independent branches in the network; therefore, it seems that each branch is unaware of the other, and the interaction between the hand and the object is not considered when training the model. How does the framework manage to explore the hand-object interaction to generate a more realistic image?
5. At last, following my concern about the complexity and necessity of using a learning approach to solve the problem, why would not a good solution apply a rendering algorithm and after improving the photorealism of the synthetic image using a style transfer method? This kind of solution feels more practical and less complex than the proposed framework.

limitations: Yes. The limitations were adequately addressed in the Limitations and Future Work section.
review_title: null

final_score: rating: 10: Award quality: Technically flawless paper with groundbreaking impact, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
review_content: summary: The authors have investigated the task of hand-object interaction generation (HOIG) which is inverse problem of hand-object pose estimation (HOPE) problem. 

Target image for hand, object, and the interaction between the two is created using a split-and-combine manner. 

HOGAN architecture based on conditional GANs is proposed for creating a unified space for hand and object. In their task, they have synthesized hand and object interaction conditioned on a target pose while preserving the appearance of the source image. This framework consists of background, object, and hand streams. Eventually, using a fusion model, all these streams are merged. 

The hand-object interaction generation framework is finally tested on two large-scale datasets, HO3Dv3 and DexYCB that have annotated hand-object mesh representations. 

The generated interaction hands and objects are finally evaluated both quantitatively and qualitatively under various methods for fidelity (e.g. FID), structure preservation (e.g. PA-MPJPE), as well as human subject evaluations.

strengths_and_weaknesses: This research work is of great importance because its direct use in AR/VR applications such as augmented shopping or mixed-reality gaming experiences. 

Their framework processes complex self- and mutual occlusion for both hand and object. For example, because hand is articulated, there is self-occlusion between the hand joints. 
Based on their claims, these authors are the first to study the HOIG task which counts as novelty. Their other aspect of novelty is propose of HOGAN architecture for conditional generation of interacting hands and objects.

I think it’s very impressive that the correct texture is rendered for the object in different pose despite the inherent occlusion. 

The comprehensive benchmarking and evaluation methodology followed by ablation studies by author is a very strong point in this paper. Further, given that the authors are the first to define this new task and there’s a lack of baselines, they did a great job of modifying the closest baselines to their needs and use them for comparison. 

Not really weakness, but adding a paragraph or two about importance of synthetic dataset (not just HOI dataset) creation and its application in deep learning would be to the benefit of this paper. 

Weakness: Provide mechanism for hyperparamater tuning of your framework hyperparamaters. Question 3.b answer states the author have told how they have chosen the hyperparameters but it is not provided.

I think there might have been a need for IRB due to use of human subjects for qualitative analysis of produced HOI. However, in Question 5.b, the answer is N/A. I would say the qualitative study should be outsourced to others not the authors because if it is done by the authors, there is a lot of bias involved. Please provide some clearance on this. 

It is not clear to me how the participants for the qualitative analysis are hired. Are they hired in-person or through an online crowdsourcing platform such as Amazon Turk? In either case, please provide details of subject recruiting. 


questions: 1- How can your model apply to non-rigid object interaction?

2- How would you expand your model to hand-object interaction without an initial source?

3- How does your model perform in case of non-symmetric objects vs symmetric objects?

4- Does your model perform greatly independently of the object size? For example, does it do a great job when interacting with a soccer ball rather than holding a toothbrush?

5- In figure 2, Inpainted Background, why didn’t you remove the entire hand? Wouldn’t the remaining of hand cause you problem further down the line?

6- Line 180: please state how you choose your hyperpatameters such as LR, number of epochs, etc. Did you perform any hyperparameter tuning (e.g. Grid Search or Random Search)? Please provide the mechanism for so as well as potential results of your hyperparameter tuning. 

7- Have you tested your model with darker color hands? How does your model perform in such cases? Is your framework agnostic to hand texture, size, and color? For example, if someone is wearing a ring on her finger, how does your model handle the reconstruction of the ring in the target pose?


Suggestions:

1- VR / AR → AR/VR

Line 68: face reenactment and sign language production → face reenactment, and sign language production

Line 171: please state which VGG, e.g. VGG-16 or VGG-19 in the writing

Line 175: implementation details and evaluation metrics → implementation details, and evaluation metrics


limitations: It is stated that the link code is provided but I cannot find it in the paper. Please provide the link to the source-code. It is also not clear to me if the code to modified baselines as well as HOGAN framework would be open-sourced in case the paper gets published. 

I do see the code in the supplementary materials however it is just not clear to me if the code will be released in an open-source fashion. Also, please let us if you would share the trained model checkpoints for other researchers publicly.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a new task of synthesizing photorealistic images of hand-object interaction. In addition, this work introduces a strong baseline method by conditioning an image-to-image translation network with hand/object topology as well as warped texture from the source image. The experiments show that the proposed approach outperforms the naive extention of existing approaches. Also the paper provides several applications such as texture editing and real hand texture transfer. 
strengths_and_weaknesses: The paper has the following strengths:
- The paper introduces a novel task called hand-object image synthesis. As discussed in the paper, the existing literature mostly focus on scene analysis, not synthesis. 
- The proposed HOGAN presents a strong baseline for this newly introduced task. The experimental results also show that existing approaches are not suitable for this novel task. 
- The paper is well written and easy to follow.

On the other hand, this work has the following weaknesses:
- While the paper constantly argues that HOIG is of broad interest to the community, the listed applications and demonstration in the paper are not convincing enough. In my view, there is no immediate applications in AR/VR or online shopping with texture editing or real-hand texture transfer only inside the small cropping of hand and object images. More immediate applications of this problem would be to use synthesized data to bootstrap the training of hand-object interaction analysis (e.g., pose estimation, object localization). I would highly recommend adding experiments to show that the presented task is useful for downstream analysis tasks similarly to [Shrivastava et al. 2017]. 

Learning from Simulated and Unsupervised Images through Adversarial Training
AuthorsAshish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb
CVPR 2017

- Another issue of this work is the expected input seems unrealistic. More specifically, it remains challenging to obtain accurate hand and object poses in a single image. Noise perturbation in input poses would be great to have to assess the robustness of each approach in the presence of pose misalignment.

While the paper presents a novel task and its strong baseline method, its significant remains unclear with the current state. As I mentioned earlier, providing more convincing use cases would make the paper stronger. Thus, I would recommend resubmitting.
questions: Related to the weaknesses above, if AR/VR, and online shopping are the primary applications of HOIG, why not proposing the task of “human”-object interaction image synthesis instead of hand-object? I can see the immediate applications in those domains if it were full-body, but not clear with only hands unless it’s used for bootstrapping pose estimation tasks.
limitations: The paper discusses its limitations and societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a new task, i.e., hand-object interaction image generation. The proposed method uses occlusion-aware topology modeling, which captures the occlusion relationship between objects and hands, and warps the unoccluded region. The warped hand input and object input, together with their topologies, are fed into the hand-object generator. The hand-object generator deals with the hand and object generation separately.

The contributions lie in that 1) the paper proposes a new task; 2) the proposed method generates plausible results for this new task.
strengths_and_weaknesses: Strengths:
1) The proposed method is intuitive and achieves plausible results on hand-object interaction generation.
2) The paper is well-written and easy to follow.


Weaknesses:
1) The proposed method takes too much information containing rich priors as inputs, which greatly eases the tasks. Specifically, with the source posture, it is easy to model the occlusion between hands and objects. With the source posture and target posture, the only thing the network needs to learn is to inpaint the warped images. As for the object generation, with the object model, the network only needs to adjust the lighting of the object. Admittedly, the task itself is challenging as suggested by the author. However, with these complicated inputs, the task is easy to solve. In my understanding, the authors tackle the problem by feeding more inputs rather than solving the problem in the technical aspect.
2) The comparison with GestureGAN+OBJ is not fair. The input to the GestureGAN is only the source image and target pose represented by sparse keypoints. The input contains much less prior information compared to the proposed method.
questions: 1) Apart from proposing a new task, the author should further clarify their technical contribution.
2) With the inputs containing rich prior information, the author should address the key technical challenges. In my understanding, with these inputs, the challenges mentioned in the paper are addressed. I wonder if there are any other technical challenges after using these complicated inputs. Please refer to my concerns about this issue in weakness (1).
limitations: The authors have addressed the limitations. For example, the posture representation is the dense mesh representation, which is not easy to get in real-world applications.
review_title: null

Meta-review: metareview: On the surface, this paper seems to be split between three borderline rejects (4) and one strong champion of the paper (10). However, this is not the full story, since two of the reject-inclined reviewers, Bc9p and zW9y did not participate post-rebuttal, despite multiple prods from the AC. The AC examined the stated weaknesses from Bc9p and zW9y, the authors' response to them, as well as the paper. The AC does not think that the concerns are paper stopping if they were left unaddressed (e.g., a glaring experimental weakness, an incorrect statement) and moreover finds the authors' response to these concerns satisfactory. The AC is then left with the reviews from p4XG (4) and aYmL (10). The remaining primary concern comes from p4XG, who points out that the method has a lot of inputs, which makes the problem considerably easier. The AC understands this concern (and thinks lowering the input requirements would be a great next step), but on balance is inclined to accept the paper. This is motivated by the quality of the results as well as aYmL's enthusiasm for the work. The AC would encourage the authors to use the extra page to give clear responses to the reviewers' questions  (e.g., what are the applications, why not just do a direct render) in the final version of the paper. Others will have similar questions. 

confidence: Certain
Decision: Accept