Hand-Object Interaction Image Generation
Authors: Hezhen Hu
Weilun Wang
Wengang Zhou
Houqiang Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: hand-object interaction
complex occulsion
image generation

Abstract: In this work, we are dedicated to a new task, i.e., hand-object interaction image generation, which aims to conditionally generate the hand-object image under the given hand, object and their interaction status. This task is challenging and research-worthy in many potential application scenarios, such as AR/VR games and online shopping, etc. To address this problem, we propose a novel HOGAN framework, which utilizes the expressive model-aware hand-object representation and leverages its inherent topology to build the unified surface space. In this space, we explicitly consider the complex self- and mutual occlusion during interaction. During final image synthesis, we consider different characteristics of hand and object and generate the target image in a split-and-combine manner. For evaluation, we build a comprehensive protocol to access both the fidelity and structure preservation of the generated image. Extensive experiments on two large-scale datasets, i.e., HO3Dv3 and DexYCB, demonstrate the effectiveness and superiority of our framework both quantitatively and qualitatively. The code will be available at https://github.com/play-with-HOI-generation/HOIG.

Hand-Object Interaction Image Generation
Hezhen Hu'
Weilun Wang !+ Wengang Zhou!.21 1.2
Houqiang Li
CAS Key Laboratory of GIPAS, EEIS Department University of Science and Technology of China (USTC)
2 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center falexhu, wwlustc)@mail ustc. edu. {zhug, lihq] edu..edducco cn
Abstract
In this work, we are dedicated to a new task, i.e., hand-object interaction image generation, which aims to conditionally generate the hand-object image under the given hand, object and their interaction status. This task challenging and research- worthy in many potential application scenarios, such AR/VR games and online shopping, etc. To address this problem, we propose a novel HOGAN framework, which utilizes the expressive model-aware hand-object representation and leverages inherent topology to build the unified surface space. In this space, we explicitly consider the complex self- and mutual occlusion during interaction. During final image synthesis, we consider different characteristics of hand and object and generate the target image in split-and sitt-and-combine manner. For evaluation, build comprehensive protocol to access both the fidelity and structure preservation of the generated image. Extensive experiments and Dex YCB, demonstrate the effectiveness and superiority of our framework both quantitatively and qualitatively. The code will be available at https:/ thub. com/ com/play-aae Sateyy-wtt--tttt wwwch-HH--ggneeatiiin wwwr--gnneattoo/HOOO0 HHOI.
large-scale datasets, i.e., HO3Dy3
Introduction
As crucial step for analyzing human actions, hand-object interaction understanding is research- worthy in broad range of applications related virtual or augmented reality. Current works largely focus on hand-object pose estimation (HOPF) [16, 19, 21]. which aims to capture the pose configuration of the given hand-object image. In contrast, its inverse counterpart is seldom considered. In this work, we aim to explore this novel task and term it Hand-Object Interaction image Generation (HOIG). Its objective is to generate the interacting hand-object image under the guidance of the target posture, while preserving the appearance of the source image, illustrated in Figure 1.
This HOIG task is of both application and research value the community. On one hand, HOIG can be potentially applied to many scenarios, such as AR/VR games, online shopping, and data augmentation, etc. For online shopping, HOIG can give the consumer an immersive experience and flexibility on object customization. On the other hand, HOIG is of board research interest, Since explicitly involves simultaneous generation of two instances (hand and object) with high interaction relationship. HOIG brings many new challenges to resolve. These characteristies thus bring new challenges. 1) It requires process the complex self- and mutual occlusion between the interacting hand and object. 2) involves image translation of co-occurring instances, where different appearance characteristics of the hand and object need be considered.
"Contribute equally with the first author. 'Coresponding authors: Wengang Zhou and Houqiang Li.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Source Image
Object Info.
Source Posture
Target Posture
Generated Image
Forela
Figure 1: Definition of our proposed task. It aims to generate the target interacting hand-object image under the target pose, while persevering the source image appearance.
The effectiveness of Generative Adversarial Networks (GANs) has been validated in realistic image synthesis, such human, face and hand [8, 27, 29, 39]. These GAN-based synthesis methods can be conditioned on different input information, such as simple-to-draw sketches, 2D sparse keypoints and dense semantic masks, etc. Among them, GestureGAN [39] resolves the isolated hand generation, which is mostly related to our task. It utilizes 2D sparse hand keypoints as the condition and attempts to generate the target hand image according to the optical flow learned from the source and target conditions. However, these methods do not consider the generation of two interacting instances, which cannot meet the requirement of the HOIG task.
In this work, we propose the HOGAN framework to deal with the challenges this novel task. HOGAN utilizes the expressive model-aware representation the condition and leverages its inherent structure topology to build the unified hand-object surface space. In this space, complex self- and mutual occlusion among hand and object explicitly modeled. Specifically, the visible parts of hand and object are mapped to the target image plane, along with their corresponding fine ame-ggained topology map. Meanwhile, the transformation flow is computed between the source and target. These middle results provide abundant information for final image synthesis. During synthesis, our framework considers appearance difference between hand and object and generates the target hand-object image in split-and-combine manner.
To systematically explore this task, build the comparison baselines via adopting representative methods from the most related single hand generation task with few modifications. The image generation quality is carefully evaluated from multiple perspectives, including the fidelity (FID and LPIPS), structure preservation (AUC, PA-MPJPE and ADD-0. ID) and subjective user study.
Our contributions are summarized follows,
To our best knowledge, we are the first explore the novel task named the hand-object interaction image generation, i.e., conditionally generating the hand-object image under the given hand, object and their interaction status. This task of board research and application value to the community.
To deal with the challenges of this task, we propose the HOGAN framework, which considers the hand-obe hand-object occlusion and generates the target image in aplit-and-ommiing manner.
To systematically explore this task, we present comparison baselines from related single-hand generation. Besides, the comprehensive metrics are chosen to evaluate the both fidelity and structure preservation of the generated image. Extensive experiments on two datasets demonstrate the effectiveness and superiority of our method over baselines.
2
Related Work
In this section, we will briefly review the related topics, including pose-guise euse-guided image synthesis and hand-object interaction.
Pose-guided image synthesis. Pose-guided image synthesis is conditional generation task, which aims to generate image under the condition of the target pose while preserving the identity of

---

the source image. This problem involves instances with rigid parts such as bodies [1. 5, 11, 12, 13, 20, 29, 36, 34, 38, 45], faces [8, 18, 24, 28, 35]. and hands [17. 22, 39, 40], and can be utilized in various scenarios, such as image animation, face reenactment, and sign language production, etc. In recent work, Ren el al. [29] generate person images under target posture with a differential global- flow local-attention framework in multi-scale manner. Deng et al. [8] utilize 3DMM [2] which parameterizes pose and shape to disentangle posture representation for face generation. Hu et al. [17] incorporate hand prior for pose-guided hand image synthesis instead of 2D joint representation. However, previous methods mainly focus on single-object pose translation problems. Different from them, the novel problem formulate, i.e., HOIG, involves generation of the co-occurring subjects, which brings new characteristics worth exploring.
Hand-objes interaction. Most current works on hand-object interaction focus on simultaneously estimating hand-object pose aligning the given image [4, 7, 9, 15, 16, 19, 21, 23, 25, 32, 42]. To better depict hand-object interaction, they resort to dense triangle meshes with the pre-defined topology as the representations, which are produced by the MANO hand model [30] and the known object model [3, 41J. Hasson et al. [16] leverage physical constraints to better estimate hand and object meshes. Cao et al. [4] propose a optimization-base based method, which leverages 2D image cues and 3D contact priors for hand-object interaction estimation. Liu et al. [21] further boost the estimation performance by supersupe learning with the assistance of in-the-wild hand-object videos. To our best knowledge, there exists work the inverse task of hand-object interaction estimation. In this work, we aim to explore this novel task and term hand-object interaction image generation.
3 Methodology
In this section, we first discuss our problem formulation. Then we elaborate the architecture of our proposed HOGAN framework. Finally, we introduce the loss function during optimization.
3.1 Problem Formulation
Hand-object interaction image generation is conditional generation task. Its objective to generate the interacting hand-object image under the target pose condition, while preserving the source appearance. Specifically, the object is well-modeled with texture known. Resolving this task is challenging, since it is non-trivial to understand the complex interacting relationship between hand and object during image generation.
We summarize the main challenges as follows. Firstly, it requires modeling the occlusion between co-occurring instances. In the interacting hand-object scenario, complex self- and mutual occlusion usually occur. Since the occlusion leads to more transformation complexity between the source and target, the occluded regions should be located and identified, which eases the final synthesis. Secondly, it needs to take into account different characteristics of two instances during generation. Specifically, hand is articulated and encounters self-occlusion among joints, while object is usually rigid with fine- mee-grained texture. The framework should generate realistic appearance of co-occurring instances, jointly with reasonable manipulation between them.
3.2 HOGAN
Overview. As illustrated in Figure 2, our framework first utilizes the expressive model-aware representation as the pose condition. Then we leverage the pre-defined structure in the model to build the unified space to perform occlusion-aware topology modeling. This modeling identifies the occluded regions, and provides the coarse target images and fine- aineggriied topology maps for the next stage. Finally, hand-object synthesis considers the appearance difference of these instances, and generates the target hand-object image in a split-and-combine manner.
Occlusion-aware topology modeling. We first give an overview of the utilized model-aware representation. The hand and object are represented via MANO [30] and YCB [3, 41] models, respectively. Both MANO and YCB provide triangulated meshes, which can densely depict the structure of hand and object. Specifically, the hand-object joint representation contains N,, vertices and N₂ faces. Its mesh H E R N.33 represents the vertex coordinates. The inherent topology P R x3xx2 is organized as a vertex triplet, in which each unit is recorded corresponding vertex
3

---

Source Irput
image
Source Posture
HO-Gen.
Source Image
Topology Modeling
Sampler
dbj Model
Target
"Arget Image
HO-Gem
Occlusion-Aware Topology Modeling
Hand- Hand-Object Generator
Source image
Source Posture
Impainted Background
lgg Strem
Background
Steam
Oity Topos
Objj. Input
Foreground
Torget Posture
SPADE
Hand
Hand Topo.
Hand Input
Foreground
Hand
Hand Topu. Obj Irpui
SPADE
Figure 2: Overview of our proposed HOGAN framework. It leverages the expressive model-aware representation as the pose condition, jointly with its inherent topology to build the unified surface space. Embedded with this space, explicitly model complex self- and mutual occlusion and generate the coarse image and fine-grained topology map for the next stage. Hand-object generator takes into account the different characteristics between these instances and produces the final target image.
coordinates aligning the desired plane. 8, l and u are notations for the source, target and unified surface space, respectively.
Embedded with the pre-defined topology, unravel the surface of hand and object model to build the unified space. In this space, the same representation is binded with the same mesh face, ignoring the pose configuration. This space adopted perform mapping from the source to the target, while inserting the pre-known object texture. Firstly, we leverage the source pose to map its corresponding image appearance this unified space in an occlusion-awart aware manner follows,
(x, v) P' [F" (I, w)), T Thea( v) W W
where Tucs(E, y) denotes the flow from the source image to the unified space. F", y) denotes the face index belonging to the (æ, y) location in the surface space, and (" y) represents the relative weighted position in this face. Besides, occlusion should be jointly computed to locate the visible texture as follows,
Ou. 9) (F" (zx y) # F" (Turs(xx,)) v))).
The visible texture from the source image is mapped to our unified surface space as follows,
I Warp(Tu...) I.) Ou.es
(3)
where I, aligns our surface space. and Warp(-) represents the element- wise multiplication and warping function, respectively. Notably, the object in the source image inevitably contains the occluded region, which insufficient for the target generation. Therefore, we utilize pre-stored object texture to replace the original object region in L,, resulting new texture image İ, Finally, we compute the flow performing the mapping between the unified space to the target image.
T T (2, y) w 3) P" (F (x,y)).
(4)

---

After that, the target image I is generated via sampling I, under the guidance of T as follows, Iw),
I
(5)
represents the coarse target hand-object image. Meanwhile, provide more guidance for the next stage, the fine fine-grained topology map Y, is concurrently generated as follows.
Y, (x, y) Bary(P" (F" (x, z))),
(6)
where Bary(-) computes the barycenter of the corresponding face in the surface space.
Hand-object synthesis. Considering hand and object exhibit different properties, we design the hand-object generator for target image synthesis in a split-and-combing manner. As illustrated in Figure 2, the hand-object generator consists of three streams: Background Stream, Object Stream and Hand Stream.
The Background Stream inpaints the background cropped from the source image with the hand-object foreground mask. The Object Stream takes the rendered object images as input and transfers its style to the real image domain. To make the stream aware of object structure information, we adopt the spatially-adaptiv normalization (SPADE) [26] to insert the object topology map.
The Hand Stream deals with the hand part that is not occluded by the object. We first synthesize a coarse image of the visible part in the target posture by warping the hand in the source image with the pose transformation flow, After that, the coarse image is refined by the U-structure network [31] in the hand stream. During refinement, similar the Object Stream, the hand topology map is modulated into the network with SPADE. Meanwhile, we extract the multi-scale features from the hand-object generator that reconstructs the source image, and integrates them into the generation process with the attention sampler [29].
The three streams separately process three instances with different properties, i.e., background, object and hand, and merge their results with the fusion module. With the features from the last layer of three streams, our framework further utilizes two convolutional layers to learn i.e., the hand mask MA and the hand-object mask M,, which indicate the area belonging to the unoceluded hand the hand-object foreground, respectively. Regarding these two mask, the fusion module merges the the three-stream results to the final generated results I as follows,
fusion masks,
I (L, M, I, C M2)) M, I, o (1-1M) where I, I, Ia are the generated results of the background, object and hand stream, respectively. O refers to the element-wise multiplication.
3.3 Objective Functions
We design discriminator train our HOGAN in an adversarial learning manner. The adversarial loss constrains the distribution of the generated images with that of the real images, which improves the visual performance of generated images. With the discriminator D(-), the adversarial loss is formulated as follows,
Le Ee el( - D(
CD Ex,,,.((1 (c)+ Es, [[1 +1+ ((
where the æ and x,, are the distribution of the generated and real images, respectively. c refers to the combination of target hand and object posture information.
Besides the adversarial loss, regularize HOGAN with the reconstruction loss the source image and the perceptual loss on the generated image, which is formulated as follows,
Liee || &2ll1,
Ligg Σ (1)))))))))))))))))){)))))))(( f((
(9)
where a, and a, are the generated source and target images while à. and d; are the ground-truth source and target images, respectively. fe(-) is the feature extractor from the i-th layer of a pre-trained VGG-19 network [37].
The overall loss is the summation of three objective functions as follows,
= C. + Cr + A2Cnggr
(10)
where X₁ and X2 are the weighting parameters to balance the objective functions.
5

---

Source Image
Target Posture
GesGAN OBJ
MG2T OBJ
HOGAN
Ground-truth
Figure 3: Qualitative comparison with baselines, including GestureGAN [39] + OBJ and MG2T [17] + OBJ, the HO3Dv3 and Dex YCB dataset. HOGAN exhibits superior performance, generating images without blurry, texture aliasing and false hand-object interaction.
4 Experiment
In this section, we first introduce the experiment setup, including datasets, implementation details, and evaluation metrics. Then elaborate the baseline methods and make comparison with them both quantitatively and qualitatively. After that, we conduct ablation study to highlight the important components in our framework. Furthermore, we explore more applications of our HOIG task.
4.1 Experiment Setup
Datasets. We evaluate our method on two large-scale datasets with annotated hand-object mesh representation, iee. HO3Dv3 [14] and DexYCB [6]. HO3Dv3 is captured in the real-world setting. contains 10 different subjects performing various fine-grained manipulation on one among 10 objects from YCB models [3]. The training and testing set contain 58, 148 and 13,938 images, respectively. Dex YCB is recorded in the controlled environment, with 10 subjects manipulating one among 20 objects. In our experiment, we choose the frames containing interaction between hand and object, with 33,562 and 8,554 images for training and testing, respectively.

---

Comparison with two hand-object interaction image generation basclines, i.e., GestureGAN + OBJ and MG2T + OBJ, on the HO3Dv3 and YCB dataset. and represent the higher the better, and the lower the better, respectively.
Table 1:
HO3Dv3
DexYCB
Method
FID4
LPIPS
UPR
FID4
LPIPS!
UPR1
GestureGAN [39] OBJ MG2T [17] + OBJ HOGAN
82.0
0.316
0.5
34.5
0.214
4.0
45.6
0.214
24.8
37.8
0.121
22.0
41.3
0.171
74.7
30.1
0.109
74.0
Implementation details. The whole framework is implemented on Py PyTrch and we perform experi- ments 4 NVIDIA RTX 3090. All U-Nets are trained from scratch. The hand-object generator is trained in an end-to-end manner and all the networks are being trained simultaneously. The Adam optimizer is adopted and the training lasts 30 epochs. We set the batch size in our experiment. The learning rate is set as 2e-4 for the first 15 epochs and linearly decays to 2e-6 till the end. The hyperparameter D and X2 are set 10 and 10, respectively.
Evaluation metrics. We design an evaluation protocol to measure the hand-object interaction image generation quality both quantitatively and qualitatively. In quantitative evaluation, measure both the fidelity and structure preservation of generated images. For the image fidelity, we adopt the widely-used Fréchet Inception Distance (FID) [10] and Learned Perceptual Similarity (LPIPS) [44] metrics. To evaluate the posture preservation of hand and object in the generated images, we utilize an off-the-shelf hand-object pose estimator [21] to report AUC and PA-MPJPE for hand pose, and ADD-0. ID for object pose. PA-MPJPE represents the joint mean error after Procrustes aligmment, and AUC denotes the area under the 3D PCK curve. ADD-0.1D denotes the percentage of average object vertices error within 10% of object diameter.
In qualitative evaluation, conduct user study evaluate the visual performance of method and two baselines. There are 20 volunteers participating in this study. In the study, the volunteers are asked to select the most high-fidelity generated results among method and two baselines. The percent of generated image preferred is recorded as User Preference Ratio (UPR).
4.2 Baselines
Since there exists work this task, we present two baselines from most related single-hand generation, i.e., GestureGAN [39] and MG2T [17], with few modifications. These methods are mainly modified with the object condition information involved follows.
GestureGAN + OBJ. GestureGAN translates the single-hand source image to the target posture in cycle-consistent manner. The source image is fed into the network along with the target posture represented by sparse keypoints. In GestureGAN + OBJ, we concatenate the object rendered result with the target posture and feed them into the network. The modified framework is trained with the same architecture design and objective functions as the previous GestureGAN.
MG2T + OBJ. MG2T is also a single-hand generation framework with hand prior incorporated. In MG2T OBJ, we maintain the hand processing module in MG2T and further extend the object modeling to involve the object posture information. The rendered result of the object is merged into the foreground branch to produce the hand-object translation results.
Rendering sendering-based methods should also be adopted for comparison. Among them, one main procedure get the reliable texture. However, the source image only contains the partial hand texture, which leads to the incomplete hand of the target rendered image. Therefore, the direct rendering is not applicable and we resort to MG2T OBJ, which has incorporated the refinement on rendered results.
4.3 Comparison with Baselines
We present quantitative results to evaluate the effectiveness of our method on HO3Dv3 and YCB datasets. Firstly, we study the fidelity generated images on two datasets. As shown in Table 1, our method surpasses two baselines on all metrics with a notable gain. Secondly, it is essential for hand-object interaction image generation to maintain the posture information of the target hand and object. Therefore, carefully analyze the posture preservation of generated images. With the
7

---

Table 2:
Hand-object structure preservation analysis on HO3Dv3 dataset. f and + represent the higher the better, and the lower the better, respectively.
Hand
Object (ADD-0.1D)
Method
AUC1
PAJPE.
Sugar Boxt
Bottlet
Banana t
Mugì
Power Drill
Aver:
GestureGAN [39] OBJ
74.1
13.0
6.4
0.0
14.8
1.1
0.0
4.5
MG2T [17] OBJ
76.2
11.9
36.1
65.0
21.5
48.9
10.9
36.5
HOGAN GT
76.5
11.8
52.1
85.0
26.3
55.0
34.1
50.5
76.9
11.6
76.1
100.0
30.2"
64.4
49.1
63.9
off-the-shelf network [21], we estimate the hand and object pose in the generated images and compute the error with the target pose annotation. As shown in Table 2, it can be observed that our method exhibits superior performance for hand and objects of different categories over two other baselines.
Furthermore, we perform qualitative comparisons with baselines. From Figure 3, we observe that the images generated by our method exhibit better visual performance compared with previous methods. Under the complex scenes, where the hand and object are highly interacting, our method can generate images with more reasonable spatial relationship. which significantly outperforms baseline methods. We also conduct a user study to evaluate subjective visual performance. The voting results are reported as the USP metric in Table 1. It is observed that our method is preferred by over 70 percent against two competitors.
4.4 Ablation Studies
We perform ablative experiments to highlight several important components, i.e., the hand and object topology, source feature transfer and split-and-combee combine generation.
Hand and object topology. In HOGAN, we modu- late both hand and object topology into the generator to provide detailed geometric information of hand and object. In Table we verify its effectiveness by comparing it with non-topology variants. It can be observed that HOGAN (the first row) achieves the best performance over two other variants. Without either hand or object topology, the framework suffers performance degradation, e.g. 6.0 and 6.5 on FID, 0.012 and 0.015 on LPIPS.
Table 3: Ablation study on hand and object topology HO3Dv3 dataset.
Settings
Metrics
Hand Topo.
Obj. Topo.
FID
LPIPS
41.3
0.171
47.3
0.183
X
47,8
0.187
Source feature transfer. To further enhance the refine- ment procedure of the coarse target hand image, we transfer multi-layers features from the source hand-object gener- ator according the transformation flow. In Table 4, we analyze the importance of the source feature transfer. The None variant refers the framework without source fea- ture transfer. The Bilinear and Attention variants refer to the framework with different samplers for feature trans- ferring, i.e., bilinear sampler and an attention sampler, respectively. can be seen that the source feature trans- fer provides useful cues for target image generation and the attention sampler achieves the best performance among three settings.
Table 4: Ablation study on the sampling method on HO3Dv3 dataset.
Metrics
Sampler
FID
LPIPS
None Bilinear
46.5
0.181
42,5
0.173
Attention
41.3
0.171
aplit-and-combing generation. Considering hand and object exhibit different properties. we involve split-and- combine (S&C) strategy in HOGAN. In this setting, we compare HOGAN with parameter-comparabby baseline model without separately generating hand and object. In Table observed that HOGAN outperforms its vari- ant (without S&C) by 2.9 FID and 0.036 LPIPS gain, which demonstrates this strategy benefits the quality generated images when involving instances with different properties.
Table 5: Ablation study on the split-and- combine strategy HO3Dv3 dataset.
Metrics
S&C
FID
LPIPS
44.2
0.207
41.3
0.171

---

Source Image
Edited Texture Generated Image
Source Image
Target Posture Generated Image
Object Texture Editing
(b) Real-World Generation
Figure 4: Applications of hand-on hand-object interaction image generation. (a) The object texture in hand- object generation is edited with characters, and HOGAN produces the images conditioned the edited textures. (b) We test pre-trained HOGAN for the real-world hand image. As highlighted the red box, HOGAN generates images which both maintain the hand identity in the source image and meet the target posture.
4.5 Application
In this subsection, we further explore several interesting applications on our HOGAN, i.e., object texture editing and real-world generation. In object texture editing, we alter the object texture with characters, i.e., "NeurlPS" and "2022", and generate the images conditioned on the edited textures. From Figure 4 (a), it is observed that our generated images well preserve the edited characters on the object texture. Furthermore, we take the hand image from the real scene as the source image to test pre-trained HOGAN framework. As shown in Figure (b), the generated images both maintain the source image appearance and meet the target posture condition.
These applications demonstrate the generalization of HOGAN on hands and objects for the real-world scenario and applications. For example, when a consumer is shopping online, interaction visualization will give him/her immersive experience. Furthermore, if consumers want some customization the object, e.g. adding the name on the phone, our provided application can achieve this through object texture editing. Besides, in the online shopping scenario, consumers usually do not have the object. They only need to upload a picture of their hand, and we can give them a real interaction experience via generating hand-object images with their hand identity preserved.
Based on our explored task, it is also of great importance for synthetic data creation [33]. Current HOPE methods but their performances are limited by the size of training data due to the anno- tation cost. One way to fertilize the model to utilize synthetic data. Our framework is also ready to generate images, which model real-world characteristics well and boost HOPE performance shown in Table 6. Specif- ically, we adopt the backbone [21]. "Aug" represents
Table 6: Application synthetic data ation for boosting HOPE performance.
are usually deep-learning-basee
Hand
Object
Aug
AUC
ADD-0.1D
77.2
67.6
78.0
76.8
the backbone is trained on both HO3D and synthesized data. It can be observed that "Aug" outperforms the baseline (without Aug) under all metrics, especially in the object.
Limitations and Future Work
Our framework considers the complex spatial relationship and different appearance properties between hand and object. The effectiveness our framework has been validated in Sec. There still exist limitations in our framework. Specifically, it utilizes the dense mesh representation as the pose condition, which inevitably contains misalignment with the RGB image plane even manually annotated. This issue mainly disturbs coarse image synthesis and framework mainly resorts to

---

the Hand-Objec Synthesis stage for further refinement. Another way to mitigate this issue is to adopt a more effective hand-object pose estimation method for better pose representation.
Our explored HOIG task is of broad research interest to the community. When hand pose estimation turns from the isolated hand the interacting hand-object scenario, we advocate the image generation community to draw more attention to this new HOIG task. This task involves generation of co- occurring instances under complex occlusion conditions. The advance in HOIG can also inspire other related human-centric image generation tasks. We outline the future works as follows. 1) More suitable representations can be explored to serve as a condition, jointly with their robustness analysis. 2) More applications are desirable to fertilize the HOIG task. 3) More extension to human-object interaction with the trend like PHOSA [43] can be explored.
Conclusion
In this paper, we make the first attempt to explore a novel task, namely hand-object interaction image generation. This task brings new challenges since it involves generating two co-occurring instances under complex interaction conditions. To deal with the challenges of this task, we propose the HOGAN framework. explicitly considers the self- and mutual occlusion among hand and object and generates the target image in a split-and-combing manner. For comprehensive comparisons, we present baselines adopted from related single hand generation and evaluate the generated images from multiple perspectives, including fidelity and structure preservation. Extensive experiments on two datasets demonstrate our method outperforms baselines both quantitatively and qualitatively.
Acknowledgment This work was supported by the National Natural Science Foundation of China under Contract U20A20183 and 62021001. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
References
[1] Badour Albahar, Jingwan Lu, Jimei Yang. Zhixin Shu, Eli Shechtman, and Jia-Bin Huang. Pose with style: Detail- preservie pose-guide gose-guided image synthesis with conditional stylegan. ACM TOG, 40(6):1-11, 2021.
[2] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of faces. In SIGGRAPH, pages
187-194, 1999.
[3] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Picter Abbeel, and Aaron M Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In ICAR, pages
310-517 2015.
[4] Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions
in the wild. In ICCV, pages 13417-12426 2021.
[5] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ICCV, pages
5933-5942, 2019.
[6] Yu- Wei Chao, Wei Yang. Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay. Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044-9053, 2021.
[7] Yujin Chen, Zhigang Tu, Kang, Ruizhi Chen, Linchao Bao, Zhengyou Zhang, and Junsong Yuan. Joint hand-object 3D reconstruction from a single image with cross-branch feature fusion. IEEE TIP, 00.4008-4021.202
[8] Yu Deng, Jiaolong Yang, Dong Chen, and et al. Disentangled and controllable face image generation via 3D mnitative-oontrastivee learning. In CVPR, pages 5154-5163, 2020. [9] Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J Crandall. Hope-net: A graph-based model for hand-object pose estimation. In CVPR, pages 6608 6617, 2020.
[10] DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions. JMVA,
1233::50--455 1982.
[11] Patrick Esser. Ekaterina Sutter, and Björ Ommer. variational U-Net for conditional appearance and
shape generation. CVPR, pages 8857-8866, 2018
[12] Chen Gao, Si Liu, Defa Zhu, Quan Liu, Jie Cao, Haoqian He, Ran He, and Shuicheng Yan. InteractGAN: Learning to generate human-object interaction. ACM MM, pages 165-173, 2020.
[13] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate- texture inpainting for pose-guided human image generation. In CVPR, pages 12135-12144, 2019.
[14] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3D annotation of hand and object poses. In CVPR, pages 3 3196-3206, 2020.
[15] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Mare Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In CVPR, pages 571-580, 2020.
10

---

[16] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, pages 11807-11816, 2019.
[17] Hezhen Hu, Weilun Wang. Wengang Zhou, Weichao Zhao, and Houqiang Li. Model-aware gesture-to- gesture translation. In CVPR, pages 16428-16437, 2021.
[18] Po-Hsiang Huang. Fu-En Yang. and Yu-Chiang Frank Wang. Learning identity-invarian motion represen- tations for cross-id face reenactment. In CVPR. pages 7084-7092. 2020.
[19] Kailin Li, Lixin Yang, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jicfeng Li, and Cewu Lu. Artiboost: Boosting articulated 3D hand-object pose estimation via online exploration and synthesis. In CVPR, pages 1-8, 2022.
[20] Yining Li, Chen Huang. and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In CVPR. pages 3693 3702, 2019.
[21] Shaowei Liu, Hanwen Jiang. Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3D hand-object
poses estimation with interactions in time. In CVPR. pages 14687-14697, 2021.
[22] Yahui Liu, Marco De Nadai, Gloria Zen, Nicu Sebe, and Bruno Lepri. cessurre-t-gesture translation in the
wild via sntegory-independeee
conditional maps. In ACM MM, pages 1916-1924, 2019.
[23] Supreeth Narasinhaswamy, Trung Nguyen, and Minh Hoai Nguyen. Detecting hands and recognizing
physical contact the wild. In NewrIPS. pages 7841-7851, 2020.
[24] Yuval Nirkin, Yosi Keller, and Tal Hassner. FSGAN: Subject agnostic face swapping and reenactment. In
ICCV, pages 7184-7193, 2019.
[25] Markus Oberweger. Paul Wohlhart, and Vincent Lepetit. Generalized feedback loop for joint hand-object
pose estimation. IEEE TPAMI, 42(8); 1198-199811, 2019.
[26] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, Semantic image synthesis with spatially-
adaptive normalization. In CVPR, pages 2337-2346, 2019.
[27] Dario Pavllo, Graham Spinks. Thomas Hofmann, Marie-Francine Moens, and Aurelien Lucchi. Convolu- tional generation of textured 3D meshes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, NeurlPS, pages 870-882, 2020.
[28]
Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. PIRenderer: Controllable portrait image generation via semantic neural rendering. In ICCV. pages 13759-13768, 2021.
[29] Yurui Ren. Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. Deep image spatial transformation for person image generation. In CVPR, pages 7690-7699, 2020.
[30] Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM TOG, 36(6):245, 2017.
[31]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical
image segmentation. In MICCAI. pages 234 244-24112222 2015.
[32] Dandan Shan, Richard Higgins, and David Fouhey. COHESIV: Contrastive object and hand embedding
segmentation in video. In NeurIPS. pages 3898-5909, 2021.
[33] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In CVPR, pages 2107- 2116, 2017.
[34] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In CVPR. pages
2019.
[35] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sehe. First order motion model for image animation. In NewrlPS, pages 1-11, 2019.
[36] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilicre, and Nicu Sebe. Deformable GANs for
pose-based human image generation. In CVPR, pages 3408- -118-316..222 2018
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. In ICLR, pages 1-14, 2015.
[38] Chaoyue Song, Jiacheng Wei, Ruibo Li. Fayao Liu, and Guosheng Lin. 3D pose transfer with correspon-
dence learning and mesh refinement. In NeurlPS, pages 3108-3120, 2021.
[39] Hao Tang, Wei Wang. Dan Xu, Yan Yan, and Nicu Sebe. GestureGAN for hand gesture-to-gesture translation in the wild. In ACM MM, pages 774-782, 2018.
[40] Zhenyu Wu, Duc Hoang. Shih Yao Lin, Yusheng Xie, Liangjian Chen, Yen- Yu Lin, Zhangyang Wang. and Wei Fan. MM-hand: 3D-aware multi-modal guided hand generation for 3D hand pose synthesis. In ACM MM. pages 2508-2516, 2020.
[41] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. In RSS, pages 1-10, 2017.
[42] Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In CVPR, pages 3895-3905, 2022.
[43] Jason Y Zhang. Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3D human-object spatial arrangements from single image in the wild. In ECCV, pages 51, 2020.
[44] Richard Zhang. Phillip Isola, and el al. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018.
[45] Zhen Zhu, Tengteng Huang. Baoguang Shi, Miao Yu, Bofci Wang, and Xiang Bai. Progressive posc attention transfer for person image generation. In CVPR, pages 2347-2356 2019.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We have clearly stated the contributions and scope in the Abstract and Introduction.
(b) Did you describe the limitations of your work? [Yes] We describe the limitations the Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the potential negative societal impacts in Section A of the Supplement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We include all the necessary code, instructions and environment needed in the Supplementary Materials or Github repository.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] The important training details are specified in the Section 4.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] In the Supplementary Material, we show the error bars of our main experiments experience. For the number of epochs, we keep it consistent among all methods for fair comparison. Grid search may bring better performance.
the form of variance. We choose the
by our
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We report the type of resources used in the experiments in Section 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] In Section 4.1, we cite corresponding papers for the asserts we use.
(b) Did you mention the license of the assets? [Yes] HO3Dv3 is publicly available without stating its license, and DexYCE follows the license of CC BY-NC 4.0.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] In the supplementary material, we provide the complete experiment code, the necessary commands and the corresponding environment for generating the training dataset, training the model, and testing the model.
(d) Did you discuss whether and how consent was obtained from people whose data you're 'e using/curating? [Yes] The training datasets HO3Dy3 and DexYCB publicly available. We cite the paper that provides the dataset and the model. The real-world images used in Section 4.5 are captured and granted by the authors themselves.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The datasets used in the experiments are publicly available and we have removed personally identifiable information or offensive content. The real-world image used in Section 4.5 are captured and granted from the authors.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] We include the full text of instructions given to the participants during the user study.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] Our IRB application has been granted by
12

---

our institution. The qualitative study is outsourced to the hired participants, without any author of this paper. We hire the participants person and randomly hire the participants in our college. In the user study, the participants are asked to score our experimental results. There no personal information involved and the participants remain anonymous throughout.
(c)
Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] All participants for the user study are voluntary and have estimated the hourly wage in local region and have paid them a gift of equal value. Besides, the time spent by each participant is estimated around 10 minutes.
13


Reviews:
[{"review_id": "ozBistzqtda", "reviewer": "Reviewer_aYmL", "summary": "The authors have investigated the task of hand-object interaction generation (HOIG) which is inverse problem of hand-object pose estimation (HOPE) problem. \n\nTarget image for hand, object, and the interaction between the two is created using a split-and-combine manner. \n\nHOGAN architecture based on conditional GANs is proposed for creating a unified space for hand and object. In their task, they have synthesized hand and object interaction conditioned on a target pose while preserving the appearance of the source image. This framework consists of background, object, and hand streams. Eventually, using a fusion model, all these streams are merged. \n\nThe hand-object interaction generation framework is finally tested on two large-scale datasets, HO3Dv3 and DexYCB that have annotated hand-object mesh representations. \n\nThe generated interaction hands and objects are finally evaluated both quantitatively and qualitatively under various methods for fidelity (e.g. FID), structure preservation (e.g. PA-MPJPE), as well as human subject evaluations.\n", "questions": "1- How can your model apply to non-rigid object interaction?\n\n2- How would you expand your model to hand-object interaction without an initial source?\n\n3- How does your model perform in case of non-symmetric objects vs symmetric objects?\n\n4- Does your model perform greatly independently of the object size? For example, does it do a great job when interacting with a soccer ball rather than holding a toothbrush?\n\n5- In figure 2, Inpainted Background, why didn’t you remove the entire hand? Wouldn’t the remaining of hand cause you problem further down the line?\n\n6- Line 180: please state how you choose your hyperpatameters such as LR, number of epochs, etc. Did you perform any hyperparameter tuning (e.g. Grid Search or Random Search)? Please provide the mechanism for so as well as potential results of your hyperparameter tuning. \n\n7- Have you tested your model with darker color hands? How does your model perform in such cases? Is your framework agnostic to hand texture, size, and color? For example, if someone is wearing a ring on her finger, how does your model handle the reconstruction of the ring in the target pose?\n\n\nSuggestions:\n\n1- VR / AR → AR/VR\n\nLine 68: face reenactment and sign language production → face reenactment, and sign language production\n\nLine 171: please state which VGG, e.g. VGG-16 or VGG-19 in the writing\n\nLine 175: implementation details and evaluation metrics → implementation details, and evaluation metrics\n\n", "limitations": "It is stated that the link code is provided but I cannot find it in the paper. Please provide the link to the source-code. It is also not clear to me if the code to modified baselines as well as HOGAN framework would be open-sourced in case the paper gets published. \n\nI do see the code in the supplementary materials however it is just not clear to me if the code will be released in an open-source fashion. Also, please let us if you would share the trained model checkpoints for other researchers publicly. \n\n\n", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This research work is of great importance because its direct use in AR/VR applications such as augmented shopping or mixed-reality gaming experiences. \n\nTheir framework processes complex self- and mutual occlusion for both hand and object. For example, because hand is articulated, there is self-occlusion between the hand joints. \nBased on their claims, these authors are the first to study the HOIG task which counts as novelty. Their other aspect of novelty is propose of HOGAN architecture for conditional generation of interacting hands and objects.\n\nI think it’s very impressive that the correct texture is rendered for the object in different pose despite the inherent occlusion. \n\nThe comprehensive benchmarking and evaluation methodology followed by ablation studies by author is a very strong point in this paper. Further, given that the authors are the first to define this new task and there’s a lack of baselines, they did a great job of modifying the closest baselines to their needs and use them for comparison. \n\nNot really weakness, but adding a paragraph or two about importance of synthetic dataset (not just HOI dataset) creation and its application in deep learning would be to the benefit of this paper. \n\nWeakness: Provide mechanism for hyperparamater tuning of your framework hyperparamaters. Question 3.b answer states the author have told how they have chosen the hyperparameters but it is not provided.\n\nI think there might have been a need for IRB due to use of human subjects for qualitative analysis of produced HOI. However, in Question 5.b, the answer is N/A. I would say the qualitative study should be outsourced to others not the authors because if it is done by the authors, there is a lot of bias involved. Please provide some clearance on this. \n\nIt is not clear to me how the participants for the qualitative analysis are hired. Are they hired in-person or through an online crowdsourcing platform such as Amazon Turk? In either case, please provide details of subject recruiting. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OSglgaIxkNW", "reviewer": "Reviewer_Bc9p", "summary": "The paper presents a framework to synthesize images with hand-object interactions using the 3D model of the object and the hand pose (also in 3D). The paper also claims to introduce a new task named hand-object interaction image generation. A set of experiments in two different datasets showed that the proposed framework outperformed constructed baselines in several metrics, including LPIPS, FID, AUC, PA-MPJPE, and ADD-01.D.", "questions": "1. Were all UNet trained from scratch or pre-trained using some dataset in an inpaint task?\n2. It is not clear the fusion step. Is it a learning layer? If so, is it an MLP?\n3. Is the hand-object generator training end-to-end? Are all the networks being training simultaneously, or was used a training-wise strategy?\n4. There are three independent branches in the network; therefore, it seems that each branch is unaware of the other, and the interaction between the hand and the object is not considered when training the model. How does the framework manage to explore the hand-object interaction to generate a more realistic image?\n5. At last, following my concern about the complexity and necessity of using a learning approach to solve the problem, why would not a good solution apply a rendering algorithm and after improving the photorealism of the synthetic image using a style transfer method? This kind of solution feels more practical and less complex than the proposed framework.\n", "limitations": "Yes. The limitations were adequately addressed in the Limitations and Future Work section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths: \n    - The paper is well-presented and easy to understand. No obvious typos.\n    - The technical novelty looks incremental, but the experimental results and conclusions may be interesting for the community.\n    - The proposed framework is well-validated, and the results showed clearly that it is superior to the baselines.\n\n- Weakness:\n    - There are two major weaknesses. First, it seems that the proposed framework is too complex to solve the problem. Since it is provided a 3D model for the object and the hand indicating the pose of both, why not apply a photorealistic render algorithm and compose the image using the background of the input image?\n    - The second weakness is the novelty of the technical solution feels incremental. Most of the building blocks of the proposed frameworks come from related work (e.g., UNets and SPADE). For instance, after computing a projection of the hand with the missing texture (occlusion-aware topology modeling component), the hand-object generator is basically applying a UNet to inpaint the hands and object texture and using SPADE to take into account the three-dimensional shape.\n    - Another concern refers to the baselines and the evaluation metrics, which seem they are not challenging the proposed framework. For example, why not compare against a photorealist render algorithm and not use a pixel-wise error (e.g., MSE) using the ground-truth images to evaluate the quality of image generation?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DZoHGNHgaCK", "reviewer": "Reviewer_p4XG", "summary": "In this paper, the authors propose a new task, i.e., hand-object interaction image generation. The proposed method uses occlusion-aware topology modeling, which captures the occlusion relationship between objects and hands, and warps the unoccluded region. The warped hand input and object input, together with their topologies, are fed into the hand-object generator. The hand-object generator deals with the hand and object generation separately.\n\nThe contributions lie in that 1) the paper proposes a new task; 2) the proposed method generates plausible results for this new task.", "questions": "1) Apart from proposing a new task, the author should further clarify their technical contribution.\n2) With the inputs containing rich prior information, the author should address the key technical challenges. In my understanding, with these inputs, the challenges mentioned in the paper are addressed. I wonder if there are any other technical challenges after using these complicated inputs. Please refer to my concerns about this issue in weakness (1).", "limitations": "The authors have addressed the limitations. For example, the posture representation is the dense mesh representation, which is not easy to get in real-world applications.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The proposed method is intuitive and achieves plausible results on hand-object interaction generation.\n2) The paper is well-written and easy to follow.\n\n\nWeaknesses:\n1) The proposed method takes too much information containing rich priors as inputs, which greatly eases the tasks. Specifically, with the source posture, it is easy to model the occlusion between hands and objects. With the source posture and target posture, the only thing the network needs to learn is to inpaint the warped images. As for the object generation, with the object model, the network only needs to adjust the lighting of the object. Admittedly, the task itself is challenging as suggested by the author. However, with these complicated inputs, the task is easy to solve. In my understanding, the authors tackle the problem by feeding more inputs rather than solving the problem in the technical aspect.\n2) The comparison with GestureGAN+OBJ is not fair. The input to the GestureGAN is only the source image and target pose represented by sparse keypoints. The input contains much less prior information compared to the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8icyokoPXhk", "reviewer": "Reviewer_zW9y", "summary": "This paper presents a new task of synthesizing photorealistic images of hand-object interaction. In addition, this work introduces a strong baseline method by conditioning an image-to-image translation network with hand/object topology as well as warped texture from the source image. The experiments show that the proposed approach outperforms the naive extention of existing approaches. Also the paper provides several applications such as texture editing and real hand texture transfer. ", "questions": "Related to the weaknesses above, if AR/VR, and online shopping are the primary applications of HOIG, why not proposing the task of “human”-object interaction image synthesis instead of hand-object? I can see the immediate applications in those domains if it were full-body, but not clear with only hands unless it’s used for bootstrapping pose estimation tasks.", "limitations": "The paper discusses its limitations and societal impact. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has the following strengths:\n- The paper introduces a novel task called hand-object image synthesis. As discussed in the paper, the existing literature mostly focus on scene analysis, not synthesis. \n- The proposed HOGAN presents a strong baseline for this newly introduced task. The experimental results also show that existing approaches are not suitable for this novel task. \n- The paper is well written and easy to follow.\n\nOn the other hand, this work has the following weaknesses:\n- While the paper constantly argues that HOIG is of broad interest to the community, the listed applications and demonstration in the paper are not convincing enough. In my view, there is no immediate applications in AR/VR or online shopping with texture editing or real-hand texture transfer only inside the small cropping of hand and object images. More immediate applications of this problem would be to use synthesized data to bootstrap the training of hand-object interaction analysis (e.g., pose estimation, object localization). I would highly recommend adding experiments to show that the presented task is useful for downstream analysis tasks similarly to [Shrivastava et al. 2017]. \n\nLearning from Simulated and Unsupervised Images through Adversarial Training\nAuthorsAshish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb\nCVPR 2017\n\n- Another issue of this work is the expected input seems unrealistic. More specifically, it remains challenging to obtain accurate hand and object poses in a single image. Noise perturbation in input poses would be great to have to assess the robustness of each approach in the presence of pose misalignment.\n\nWhile the paper presents a novel task and its strong baseline method, its significant remains unclear with the current state. As I mentioned earlier, providing more convincing use cases would make the paper stronger. Thus, I would recommend resubmitting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept