Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: DART is an extended-MANO with rich texture, diverse accessories and articulated wrist. A synthetic data generator based on Unity is built on top of DART to. generat 800K synthetic samples. With synthetic dataset DARTset, the paper demonstrates that it is ccomplement to FreiHAND.
strengths: The efforts for creating more realistic hand synthetic data is great.
1. Extend MANO model from 3 aspects, namely rich texture, diverse accessories and articulated wrist
2. Build a data generator to render synthetic hand data automatically
3. Perform experiments to verify the effectiveness of 
weaknesses: Main concerns lie in the experiments.
1) Sec.4.2 benchmarks 4 representative methods, but it is not very clear the main insights.
2) Sec.4.3. performs cross-dataset evaluation, although reducing errors in mixed setting, some concerns are left for:
    
    2.1) From Tab.5. and Tab.6, the cross test seems to verify that a large domain gap exists between DART and FreiHAND

    2.2) It's unclear which factor contributes to the performance, and how much these different factors matters. Are these designed rich textures important? Or the designed hand poses are crucial? Or other factors like camera pose or rich accessories influence most.

    2.3) Although extending MANO with articulated wrist, it is unclear the importance of adding it from current expriments
correctness: Most claims are sound
clarity: The paper is easy to follow.
relation to prior work: Most related works about synthethic hand dataset and articulated hand models have been discussed. Some synthetic human dataset, e.g., AGORA, HSPACE, GTA-Human, are also related. Although this paper focues on synthetic hand, synthetic human dataset performs similar pose/shape estimation task and have developed some techniques for using synthetic data.
additional feedback: 1. Will the data generator be made public?
2. I am willing to raise my rating if main concerns are addressed.
review_title: DART is an impressive synthethic hand dataset but some concerns are left for expriments 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The work introduces a generator to create synthetic datasets of photorealistically rendered hands in various poses, camera angles, backgrounds, and lighting conditions. It extended a previously developed MANO hand, morphable model. The main novel points of the work are the introduction of handcrafted realistic textures that cover variations in skin tones, nail colors, tattoos, blemishes, and other surface variations, as well as providing a set of accessories of common types — watches, rings, bracelets, and gloves. The generator is supplemented with a GUI that allows fixing some of the settings giving more precise control over the data generation.

The paper also describes a dataset obtained with the developed simulator that provides more appearance variation compared to earlier work.
strengths: I believe that the proposed dataset and data generator are a valuable development for improving hand tracking and reconstruction. It is the first work to introduce skin imperfections and accessories to the dataset of hands, taking an additional step toward bringing the gap between synthetic and real data.

Providing a GUI to control the data generation process is another strong point — it would reduce the learning effort for the users of the generator. I’d also note that using a popular base hand model is also a good choice, which makes skills and tools for working with MANO transferrable to the new generator.
weaknesses: Firstly, my main concern is the potential for gender and skin tone bias in the data, which require clarification from the authors (I discuss this further in the Ethics part of the review).

Secondly, the dataset lacks standalone documentation requested in the NeurIPS datasets and benchmarks requirements.

Thirdly, the method needs a few clarifications that I request on in other parts of the review.

The first two are my main reasons for giving the ranking of 4.
correctness: Overall, I feel that the suggested dataset and generator are constructed soundly. Extending the hand data with realistic textures and accessories is a natural direction to evolve existing synthetic datasets.

The cross-dataset comparison provides an overall demonstration that DARTSet complements FreiHAND well. However, the two datasets vary in multiple aspects — in pose distribution, skin tones (FreiHAND seems to be biased towards white skin), accessories, objects, and, likely, lighting conditions. It is hard to evaluate which aspects played a role in a performance boost. I would rather see a comparison that evaluated the work's main contributions, namely the texture details and accessories, and controlled for other factors, for example, by replicating the pose distribution of FreiHAND.

The way the authors handle accessories is not entirely clear to me. If I understood correctly, the shape of the hands could be modified using MANO-compatible shape parameters before starting the data generation. Since the shapes of accessories do not adapt to the hand shape change, how are the collisions handled? A more detailed description of the accessories' placement process is also needed. How are the allowed positions for accessories defined? These details are needed to understand the amount of variation in accessory placement and ensure the work's reproducibility.

As for the pose interpolation solution to enhance the pose distribution in DARTNet, I wonder how the authors ensured that interpolated poses are valid (e.g., do not contain self-collisions or unrealistic joint angles).

One the minor suggestions (do not affect the decision toward the paper):

- The GUI might benefit from utilizing Unity interfaces for pose manipulation instead of providing sliders for adjusting the pose.
- The naming conventions of the provided assets could be improved using distinctive features instead of numbers, e.g., glove_lace_smallpattern_white instead of lace_01_2. This approach will benefit the GUI and the overall usability of the assets.
clarity: The work delivers the intended message overall, but the explanations are unclear in several places, and some typos need fixing. Some of the things that I noticed are as follows:

- The sentence on lines 175-178 is hard to understand. I believe it should be separated into multiple, with each point providing more details.
- The sentence on lines 186-187 is strangely composed and refers to Table 1, which contains very different information.
- The following sentence (187-189) is also hard-to-read. Also, the word “stuff” is not typically used in academic writing.
- Lines 195-197. I could not understand the message at all. Do the authors provide the data loader? If yes, then what can it do? If that is an existing part of the pipeline, I ask the authors to introduce it in more detail.
- Line 219: “Exportation.” First of all, I believe it is “export.” On the other hand, judging by the provided description, it seems like something like “generation” would suit the procedure better. If I understood correctly, this feature allows generating several renders from the same pose, light setup, and camera view.
- The paper mentions “material properties” in several places. What is meant by those? How does it differ from the appearance provided by the hand textures?
- Sec. 4 of supplementary does not indicate that the licensing information is provided for the dataset and does not include licensing terms for the code and the assets (accessories, textures)

**Typos:**

Lines 90-91: “baked-in terms implied inside” → baked-in textures (?)

Line 139: detailed → detail 

Line 145: is our GPU is (?)

Line 150: “which with former arm or at least wrist contained in the image” 

Line 153: which of → “which contains” or “which consists of”

Line 165: below → above (I believe the mention was earlier)

Lines 183, 184: map → maps

Line 288: works → work

Line 293: acceptance → accepted

Supplementary, line 3: kinds of data → components (technically, GUI is not a data).
relation to prior work: The work stands apart from the prior art on synthetic hand datasets thanks to using accessories and skin imperfections, as indicated in the paper.
additional feedback: The authors chose the CC BY-NC-ND 4.0 license for their dataset, which includes the “No derivatives” condition. This clause may prohibit expanding the dataset, publishing new splits, etc., in the follow-up work. I ask the authors to confirm that they intend to restrict such derivative work.
review_title: An important development for hands datasets but has some concerns

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents DART, which consists of 325 high-resolution texture maps and accessories for the MANO hand model. The paper also introduces DARTset, a dataset containing 800k images of hands generated using DART in diverse poses. 

strengths: 1. This paper presents a dataset of high-resolution texture maps for the MANO hand, which will be of interest to those already working with the MANO hand. 

2. This paper also presents a data generation pipeline to render synthetic hands, which the authors use to generate a dataset which they claim can be scaled up to 3M+ images.

3. DARTset claims to cover a more diverse range of skin tones than prior datasets, which is important when training data-driven models with broader population groups in mind.

weaknesses: 1. Adding hand accessories is one of the claimed contributions of the work, but it is unclear from the experiments how accessories impact results. DARTset examples in Figure 5 appear to show hands w/o any accessories.

2. Section 4.3, the paper only compares against FreiHAND as a baseline on mesh reconstruction. These experiments would benefit from additional comparisons to baselines, such as those in Table 3. 

3. See Ethics section.

4. The clarity and polish of the writing could be greatly improved. See Clarity and Additional Feedback sections.

5. No code has been released yet. Even a small example demonstrating how a user may download DART and render a DART-textured hand model would be appreciated, and ease concerns of future code release.

correctness: L179-L182 describes how 3D artists were hired to create the texture maps. The paper would benefit from additional details on how the texture maps were designed, such as what these artists were instructed to do. 

clarity: The paper contains grammatical errors and awkward sentences throughout. See Additional Feedback.

relation to prior work: Table 1 includes a comparison between relevant articulated hand models. The paper describes how this work differs in NIMBLE in L96-L103, namely:

1. high-resolution texture maps for the MANO hand model, rather than define a new parametric hand model as NIMBLE does

2. skin features such as “moles, nail colors, scars, tattoos and palm prints” (L101)

3. connecting forearm to wrist, for accessories such as watches and bands

additional feedback: The paper says that DARTset may be scaled up to billions of images (L253). Please provide details on how long data generation takes, and how much disk space is required per sample (and/or in aggregate). Additionally, I would recommend that the authors avoid making this claim, since the current size of DARTset is 800k, and future work describes plans on scaling up to only 3M (L292). 

Please provide details on how Figure 4 was generated.

Table 1 provides a comparison between different articulated hand models. I think a visual comparison would be helpful as well, to assess differences in realism.

I recommend avoiding self-congratulatory language, as in:
- “ingenious simplicity” (L167)
- “unique charm” (L191)
- “superior” / “superiority”
- “major innovation” (L101-102)

---

On clarity (this list is not comprehensive, and I would recommend the authors conduct a thorough editing pass):

L44: “The Unity”, grammatical error

L46: “with above the controllers”, grammatical error

L52-54: comma splice

L58: “the deep network”, grammatical error

L134: “we separating”, grammatical error

L243: what does “embrace” mean in this context?
review_title: Official Review of Paper16 by Reviewer PcVq 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - Provided the Unity GUI which allows users to render hands with different pose, camera, background, lighting, and DART’s textures. 
- Publicly released large-scale dataset DARTset containing a sizeable number of (800K) diverse hand images paired with aligned 3D annotations.
- DART extended MANO with textures and accessories 

strengths: - Diversity in dataset in terms of appearance (skin color), blemishes, personalized make up, daily accessories
- Although not yet available, the tool would allow users to create hands interactively
- Supports a wide range of annotations photorealistic image and its corresponding pose parameters, 2D/3D joints, mesh, silhouette, and depth-map
- Also benchmarked DARTset on 4 representative methods 
weaknesses: - Texture and accessories seems to be the distinguishing factor between this dataset and others (Table 3). Why is textures i.e. appearance, blemishes, make up and accessories so important? To better evaluate the importance of these features of generalisability, perhaps the authors could train on certain subsets i.e. no accessory vs accessory and evaluate them on other benchmarks. 
- What is the effectiveness of training on DARTset? Perhaps authors could also train the same model on different datasets i.e. DARTset, FreiH, RHD, InterH. to evaluate the effectiveness of using DARTset over other datasets. 
- For Figure 4, why was the distribution compared to RHD and FreiHAND and not the other 5 datasets mentioned in Table 3. 
- The benefit of the articulated wrist allows additional wrist-accesories such as watch and bracelet. While this definitely increases diversity, I am not too sure how this will help hand reconstruction (PA-MPVPE and PA-MPJPE). Perhaps a subset with and without it could be run to evaluate the effectiveness. 

correctness: The dataset is constructed in a sound way. However, the effectiveness of training on this dataset is not evaluated or compared to other datasets. It would be interesting to compare how each diverse feature help to make it a strong dataset. 
clarity: - Overall, well written and concise.
- Section 4.3 mentioned that CMR achieves 8.9% MPVPE. However, only PA-MPVPE were evaluated in Tables 5 and 6. Results on MPVPE could be added into the table if it helps to support the claim.
- Why is PA-MPVPE not added for Integral pose and RLE? If there's a reason, it could be added into section 4.2 for clarity.
- Why is PA-MPJPE  added in Table 4 but not Table 5 and 6. PA-MPJPE seems to be a useful metric to add in. 
- Figure 4 is caption is slightly unclear. It is recommended to add the features are regarding hand pose distribution.
relation to prior work: Table 1 and 3 lacks comparison to other datasets in terms of background, viewpoints, and pose diversity.
additional feedback: - The authors have yet to publicly release the tool and full dataset at this point of review. 
- Would be a great feature to allow accessories to fit more variety in hand shapes
review_title: Paper review for "DART: Articulated Hand Model with Diverse Accessories and Rich Textures"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper describes a dataset of synthetic hands with diverse poses, textures, and backgrounds, for training hand-pose estimation models. It also promises to release the textures and accessories, and a Unity-based GUI, to generate synthetic hands data. The authors show that hand-pose inference models are able to train to high accuracy on a dataset of 0.8M synthetic-only images, and that training on a dataset with mixed real and synthetic data improves performance on real-only validation.

Note that there are really two contributions:  
* (1)  A GUI, along with assets, to generate a batch of synthetic data for a given hand pose and lighting. The process of generating a target pose and lighting is somewhat tedious but then data is generated with randomized texture and backgrounds. 
* (2) A dataset that uses bottom-up heuristics to sweep over a broad range of poses (based on A-MANO and FreiHAND), along with randomization over appearances.

strengths: This is highly relevant work, as generation of synthetic hands data has received considerably less attention than other areas such as faces and rigid objects, while accurate collection and ground-truth of in-the-wild hands data is extremely difficult.  While it has weaknesses (described below), the dataset and GUI represent extremely useful data that does not have any available alternatives in the public domain. Hence the broader research community stands to gain significant benefit from the dataset.  Also, the use of artist-generated content avoids ethical concerns with collecting real-world data or even synthetic data based on real-world scans, hence this is a significant advantage. A few bullet points as well:

* The dataset as well as the raw materials for generating new data are diverse and appear reasonably high-quality.
* There is considerable diversity in textures, including wrist and accessories, which is fairly unique in the literature.
* The paper does a good job of describing the procedure for generating synthetic images and ground truth data.
* The method for choosing a set of hand poses in DARTset is clever, by pairing each “clean” hand pose with a maximally-distant pose in FreiHand, then using Slerp to interpolate several poses in-between them. The resulting diversity in pose appears quite high.


weaknesses: There are several weaknesses in clarity, approach, and features as detailed below. While the body of work that went into the work is significant, the communication is poor so may be difficult for other researchers to make good comparisons or learn a lot about how to create their own synthetic hands datasets. This might be mitigated by the contribution of assets and a GUI to generate data. However without seeing the code it is hard to tell if this is something that is easily extensible, or if the research community is likely to be able to easily adopt and extend it. 

correctness: * There is only a bare baseline of using this data to train and test (1) a set of ML methods on the synthetic dataset, to show that models can train on this data and see how they compare with each other, and (2) comparing model accuracies when training either on synthetic-only, real-only, or a mixture.
* Any training involving real datasets is limited to FreiHAND.  It would be good to compare to other datasets such as InterHand2.6m.  

clarity: * In the description of the method for generating diverse poses in DARTset, it was not clear how a synthetic pose from A-MANO was chosen. Is there already a discrete set of realistic poses (not just realistic individual joints) provided in A-MANO?  The reference at this point is to a paper on “contact-potential-fields” for modeling hand-object interaction, which does not seem to be relevant here.
* The grammar could use significant help. The ideas are mostly clear enough that it was not impossible to understand the intent of the sentences, nevertheless the reading is a bit awkward.
* The code for GUI and asset generation are not discussed in detail so it is hard to tell if this work is easily extended

relation to prior work: Yes
additional feedback: There are areas of improvement that limit the usefulness of the data and approach. The current work does seem to be a good starting point to build off of and address these in the future however.
* Specifying and controlling dimensions of randomization:
    * For the GUI-based procedure, the randomization is only over texture and background, but doesn’t change the pose, illumination, or camera viewpoint. The latter would seem to be fairly straightforward, however as mentioned in “Limitations” section, randomization over pose would need some effort to do in a way that doesn’t create unrealistic poses so isn’t implemented here.
    * There did not appear to be any hand shape randomization (i.e., hand thickness, finger lengths, etc.)
    * Accessory placement is said to be randomized, but it is not clear how these are parameterized.  The difficulty go doing so is acknowledged in the Limitations section, but then what was actually done in the current paper?
* As mentioned in “Limitations”, there is no hand-object interaction.
* Only single-hand, not two-hand.
* No mention of accessibility concerns — e.g., missing fingers or injured hands.
* It is focused on generating single-frame poses, not animations, so no ability to use this yet for dynamic gesture recognition.
review_title: A useful dataset and GUI for hands research with weaknesses in clarity and approach.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper propose a pipeline to synthesize hand data using the 3D rendering software Unity. To simulate human hands, many features like  color/texture/lighting/background/accessories are considered, which makes the data more realistic. With this pipeline, high quality hand data can be synthesized easily with ground truth labels. A large scale dataset named DARTset, consisting 800K hand images with associated rich labels are made within this pipeline. Remarkably, the GUI to make such data is claimed to be released to the public.

strengths: 
+ Rendering hand data in CG software is a very good idea. Many comprehensive features of both hand and environment, i.e., textures, lighting and accesaries, are considered in the proposed dataset. Compared to existing hand datasets, the proposed one composites of larger amount of data and resemble real scenes more.

+ The synthesized data is shown to be useful in real cases. As shown in Table 5/6, training with the proposed dataset can improve the performance on real hand poses.

+ The authors claimed they would release the generation pipeline with a GUI. It is a big contribution in my view. With this GUI, one can collect realistic hand data easily and fast, which could burst relevant research.

weaknesses: - In this paper, the authors emphasized the importance of diversity of many factors (like textures and accessories). It is intuitively good and right. However, do these factors really have a big impact on relavant tasks? If possible, can the authors do some experiments to illustrate it? For example, to show the effect on hand color, the authors may train a model on light color hand and test it on both light and dark while keeping all other factors the same. 
Meanwhile, it is not clear whether the accessories (rings, gloves ...) are counted into hand meshes, which needs further explaination 

- This paper lacks detailed statistics on data distributions, e.g., hand shape (beta), background types, lighting parameters or RGB statistics..., of DARtset compared to existing datasets. Specifically, a comprehensive illustration or clear quantitative values on the *pose diversity* are not provided. Since hand poses shown in Figure 5 are in quite similar angles, it may lead the readers to worry whether all data in DARTset are like those. I think the authors could show more details on advantages of their dataset.

- Few things can be learned from the Benchmark part. Ideally, I think it should contain more analysis on performance and shortcomings of different methods instead of just a benchmark, if possible.
correctness: The dataset is constructed in a sounding way, where hand data are rendered from Unity with correct settings.
clarity: The paper is easy to read and to follow.
relation to prior work: The paper contains a comprehensive related work section, but still need to show more details on difference of data distritbutions.
additional feedback: If the GUI/dataset is released, and/or the authors provide more details, I will definitely improve the score.
review_title: A good pipeline to synthesize hand data

Meta-review: metareview: Four reviewers recommend acceptance, and two recommend rejection. Some reviewers, including an ethics reviewer VcHW, initially brought up issues with bias in skin tone, but they felt that these issues were largely addressed in the revision. Reviewer UtnZ recommended rejection. They raised concerns about the domain gap with FreiHAND, the analysis, and the utility of the articulated wrist. Reviewer xPmt initially recommended rejection based on concerns related to the cross-dataset comparison and handling of accessories, but recommended acceptance after the author response. Reviewer PcVq recommended rejection. They praised the diversity and size of the dataset, but had concerns about writing, claims around skin tone. They did not post a final review. Reviewer Pb7s recommends acceptance, based on the usefulness of the dataset and GUI, but had concerns about the writing it is unclear if these were addressed since there is no final review. Reviewer qE3p initially was concerned about diversity and analysis, but felt that these issues were addressed in the author response, and recommended acceptance. Reviewer GNor recommended acceptance, though had concerns initially about evaluation and about clarity. On balance, the AC recommends acceptance. However, the reviewers must address the concerns raised by the reviewers, including the issues around writing raised by PcVq (which are not completely addressed in the current revision) and the clarity issues raised by GNor. 
confidence: 3: The area chair is somewhat confident
Decision: Accept