Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a dataset with human interactions recorded in an ACM MM workshop. It involves multiple overhead videos, audio and wearable sensors with privacy-preserving. The author group has also provided continuous key points, speaking statuses and F-formation annotations. The dataset support person/keypoint detection, speaking status detection, and F-formation detection. In a broader context, this dataset can be useful for many applications such as security surveillance and social signal processing.


strengths: - The paper provides a comprehensive description of how data is collected, prepared and benchmarked for multiple tasks.
- It involves a tremendous amount of work for visual annotations (if done properly).
- The dataset includes a higher number of subjects, better video quality, and wearables signals with wireless hardware synchronisation.
- Both individual-level and group-level statistics were provided.
- Useful for many applications in a broader context.
- Privacy preserved for human subjects.
- Interesting and challenging for SOTA.
weaknesses: - Limited emotional expressions in social interactions.
- Lack of benchmarking results on recent SOTA methods as the paper only included very limited number of methods in the benchmark
- Lack of background and context diversities

correctness: Yes
clarity: Yes
relation to prior work: The paper included a comprehensive comparison with prior datasets of free-standing people in the wild.
additional feedback: NIL
review_title: Useful dataset for understanding social interactions

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces the Conflab dataset, a multi-modal dataset of social interactions in-the-wild. Data from 15-45 minutes of 48 individuals interacting at a conference from 10 overhead cameras, as well as a badge-borne accelerometer and audio recordings. Multiple hardware advances are made to allow for synchronization across cameras and devices, as well as inclusion of multiple datastreams on the badge. The authors’ also collect survey data on participants with information about their senority and conference interests. The raw data is annotated with body keypoints, speaking status, and F-formation identity, a form of social grouping. Several experimeriments are performed on (1) keypoint estimation from overhead cameras (2) turn-taking from keypoint and accelerometer data (3) F-formations detection from different keypoint subsets. The dataset is well documented, with a complete datasheet and discussion of privacy. 

This manuscript describes a problem that I feel is of broad interest to the community, and I believe describes multiple novel advances. However I think the manuscript is undermined as a dataset by the quality of the video tracking and somewhat cursory annotations of the data described below. Because of these weaknesses, I am not sure that it will find broad utility in the field, and as a result may be better suited to a specialized journal. 

strengths: This manuscript demonstrates a new technical approach to a complex, multimodal machine learning problem. The data collection contains multiple innovations and the benchmark experiments included span multiple dicsiplines. There is a detailed methodological supplement and ethical discussion. 
weaknesses: The weaknesses of the manuscript are in the quality of the data annotations. The manuscript clarity could also be improved:

(1)	Quality of the video tracking. The authors’ motivate using overhead views to preserve privacy, but ultimately this leads to poor pose estimation, with up to 50% occlusion for each keypoint. This occlusion makes interpretation of the following results difficult. If part of the motivation of the data is as use for keypoint estimation from overhead video, then I would hope the quality of the ground truth would be higher, eg including ground truth data from the side view cameras, multiple annotator labels, estimates of MPJE etc. 

Because of the occlusions, much of the data is imperfect or even removed (‘L 1079 in the supplement If there are more than 50% keypoints missing for a person, we remove the person bounding box and keypoints from the ground-truth.) As a result the utility of having in the wild data has to be balanced against the fact that the data is missing salient events. This issue affects all downstream tasks as well, e.g. for F-formations it is unclear how much the accuracy is impaired by noisy or missing keypoints.

Overall, it seems like a superior approach would be to conduct either semi-natural recordings in a lab with superior ground truth, or to use side view cameras to do the pose annotations but only release the overhead if privacy is truly the concern. It isn’t clear how the side-view cameras are used. 

(2)	Quality of annotations and inter-rater reliability. It is difficult to interpret the efficacy of some of these ML approaches, because there is no sense of the inter-annotator variability to gauge performance. Is 80% precision in vocal turn taking good? It is hard to assess without understanding what fraction arises from annotator variability. It also seems the audio data would be useful here for labeling? Similar holds to an ever larger degree for keypoint data as mentioned above and for F-formations. 

Minor points:
Minor
(3)	In the supplement it mentions the cameras were calibrated, but I don’t see any use of 3D pose data.
(4)	At some times there were 8 cameras mentioned, other times 14, but only a subset were analyzed.
(5)	I found the description of how different camera views were reasoned across for F-formation detection confusing. 
(6)	The authors emphasize that the data was synchronized and thus could be analyzed at high framerates, but the speaking status is only annotated at 1 Hz and I believe the F-formations 
(7)	Use of low-frequency audio – was this data used in any of the experiments? Or the Bluetooth proximity data? It seems relevant to the audio turn taking tasks and F-formation respectively.
(8)	There is lots of tangential information that could be removed eg: L200, context of using the mobile phone, L283 discussion of conversation floors
(9)	How were identities from the midge associated with visual keypoint data?
(10)	Elaborate on definitions for F-detection, as this is a relatively specialized topic that may not be familiar to many readers. 
(11)	Table 1 – is the dataset 45 minutes long or 15 minutes from multiple modalities? 
L294 – development typo

correctness: See above -- I have reservations about the dataset quality. 
clarity: The clarity is okay. Overall the manuscript felt a bit unfocused, in part because it is trying to do so many things (a new dataset, a new dataset concept, multiple technical advances as well as benchmarks, across several different fields). eg	There is lots of tangential information that could be removed eg: L200, context of using the mobile phone, L283 discussion of conversation floors. 

I would suggest streamlining the manuscript, removing the discussion of the new dataset collection concept. 


relation to prior work: Yes. 
additional feedback: See above.
review_title: Multiple technical advances, but limited by data quality

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a dataset focusing on free-standing interactions in an international conference. To address the problem of capturing data such as pose annotations with extensive social interactions, the authors propose a data collection design named ConfLab.

strengths: [S1] ConfLab captures high-quality datasets (body keypoints in high sampling rates) and enables multi-sensor data from different modalities. I can sense that the authors put a lot of effort into constructing this dataset.

[S2] The discussion of the relevant methods is thorough. The comparison in Table 1 shows that this dataset has merit.
weaknesses: In addition to the detection of speech states, I wonder if the authors could provide more benchmarks for this multimodal setup. I think the potential of this dataset in multimodal tasks based on social interaction (detection, prediction) is of interest.
correctness: The construction of the dataset and evaluating methods are technically sound. The description of data acquisition in the paper is detailed.
clarity: Yes, the paper is overall well written and easy to understand. 
relation to prior work: The discussion about prior works are clear.
additional feedback: N/A
review_title: A Multimodal Multisensor Dataset of Social Interactions

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, the authors propose a multisensor dataset to understand human social interactions. This dataset is collected using aerial-view cameras capturing the interactions of attendees during the conference. The dataset supports applications including person and keypoints detection, speaking status detection and F-formation detection.
strengths: 1. Compared to existing datasets, ConfLab involves detailed annotations of 17 keypoints and speak status annotations and hardware synchronized camera and multimodal wearable signals.

2. The paper is well-written and the dataset is well documented, which will benefit the potential users.

3. Privacy and ethical issue are well discussed and the dataset collection process is under strict restrictions.

4. The authors provide reproducibility checklist and well organized reproduction procedures.
weaknesses: 1. The authors claim that pre-trained SOTA methods struggle with a privacy-sensitive top-down perspective of person and keypoints detection, which motivates this application. I am a little bit doubtful about the usefulness of this application from a top-down perspective. 
correctness: I don't have concerns about the correctness of the data collection process.
clarity: The paper is well written and easy to follow.
relation to prior work: The authors carefully and comprehensively discussed the prior works.
additional feedback: No additional feedbacks.
review_title: ConfLab: A Rich Multimodal Multisensor Dataset of Free-Standing Social Interactions In-the-Wild

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper makes three contributions.
(1) It introduces ConfLab as a concept to collect multimodal free-standing social interaction data using multiple sensors in semi-controlled settings, such as organized social gatherings. 
(2) It introduces a dataset, gathered during the first ConfLab run at an international conference.
(3) It introduces a benchmark, derived from (2), for the tasks of (a) person and keypoints detection, (b) speaking status detection, and (c) F-formation detection.
strengths: ConfLab as a data collection concept that allows capturing rich social interaction data in a realistic, yet privacy-preserving setting in semi-controlled venues constitutes a valuable addition to the methodological toolkit of "research on the community by the community" (and human behavior research more generally).

The dataset and benchmark will facilitate progress in computer vision research on (and likely also beyond) the benchmark tasks.

The dataset (and interaction data captured in a ConfLab more generally) could also prove useful for other subcommunities in data mining, machine learning, and human interaction research; for example, the derivable social interaction networks are much richer than most readily available social interaction networks, which often determine interaction only by sensor proximity.
weaknesses: The overall ConfLab setup appears to be costly (in terms of the financial, temporal, computational, and human resources needed), which limits the replicability of the setup to well-funded institutions or venues.

The work does not discuss potential negative societal impacts in the main paper, but such a discussion (in a spot more prominent than the Appendix) seems warranted, as the work might contribute to the development of privacy invasion and surveillance technologies in a more direct fashion than other datasets and benchmarks. 
See also Point 2 in the "Ethics" section.

Also, "in-the-wild" is over-advertising.
correctness: The data collection concept is sound. 
The data underlying the dataset was adequately captured, and it was annotated in an overall reasonable way (but see the related questions in "Additional Feedback").
The tasks selected for the benchmark are natural, interesting, and challenging. 
The baselines, training process, chosen evaluation metrics, and overall experimental setup are reasonable.
clarity: The paper is structured well and written well.
See "Additional Feedback" for potential micro-level improvements.
relation to prior work: The "Related Work" section situates ConfLab in the relevant literature and highlights what distinguishes the work from previous approaches.
additional feedback: Content
- Title: "In-the-Wild" seems to be a bit over-advertised (also everywhere else it appears); the setup is more like "In-the-Zoo". You might want to eliminate "In-the-Wild" (if you keep it, it would be "in the Wild" when not used to describe a noun or a verb directly), and perhaps add "Realistic" or "Unscripted" before "Free-Standing" if you feel that this adjective does not convey enough wild(er)ness. However, more generally, the title, which mentions the dataset only, is a bit at odds with your actual contributions: dataset, benchmark, and data collection concept. The latter seems to be most important to you (in particular when judging from the abstract and the conclusion), while the former two might be most relevant for the audience of the NeurIPS Datasets and Benchmarks track – so maybe all three should go into the title? Something in the direction of "ConfLab: A Data Collection Concept, Dataset, and Benchmark for Capturing, Annotating, and Analyzing Free-Standing Social Interactions"? 
- l. 21/22: I'm afraid I don't follow the reasoning in this sentence. (1) What are "interactive social systems"? (2) Why is "studying social human behavior in real-life situations" a "crucial challenge" in their "development"? This might be just a problem of phrasing, but your first sentence is currently one of the least clear sentences in the paper (at least to me).
- l. 122: How is the data from the 4 cameras that are not shared to preserve privacy used in the dataset generation process?
- l. 170–173: How do you catch annotation errors or inaccuracies? Can you explain why you deemed it sufficient to have each segment annotated by _one_ annotator only (e.g., why not 2, + 1 for reconciliation)? Was there any kind of quality control? What would be the impact of uncaught inaccuracies in the annotation process?
- Section 5 (related to ConfLab as a Data-Collection Concept): What about participant demographics? You distinguish between newcomers and veterans, but from, e.g., a social science perspective, it would be extremely interesting to see how conversation groups relate to demographic features.
- Figure 6c could become a panel of Figure 7 (it appears a bit out-of-place in Figure 6)
- Figure 2 in the main paper, Figure 11 in the Appendix, Figures in Sample Participant Report: Pie charts are never the answer to any visualization task, unless the task is to visualize the division of a circular pie (or pizza). You are visualizing a 1D statistic in 2D, the size relationship between the pieces now depends on the chosen radius of the pie chart, too, people are bad at judging angles and circularly arranged proportions, ...even worse: the pie chart with an enlarged slice stuck out for emphasis (Sample Participant Report Figures 1, 2, and 5 (a)). I would recommend to use bar charts instead – both for the parts you can still change in the paper and for the participant reports going forward.
- Main paper and Appendix: You seem to be using red, pink, and indian red as colors for emphasis/links. Settle for one, and perhaps make it a shade of blue. Reds and pinks are extremely aggressive to the eye (see also the next point).
- Appendix B: Please change the color of the headers of the datasheet questions and headers from what appears to be "indian red" (#E7485D) to something that is not overwhelming for people with a very sensitive visual cortex. Frankly, there is no reason to use color for emphasis _at all_: The main questions are already bold, the follow-up questions could be typeset in Italics (if you feel the need to distinguish them from your answer by more than the start of a new paragraph), and the block backgrounds could become black or some shade of gray. This is also better for printing (some people might still do that).
- Appendix ll. 801 et seq.: A tabular overview of the costs involved in a ConfLab run (broken down by the steps of the data lifecycle), including an estimate of the total cost, would be appreciated, as this will help others decide whether they can afford it.
- Appendix ll. 832–838: Which platform was used for recruiting crowd workers? What is known about their demographics (esp.: how does your average hourly payment compare to the minimum wage paid in their country of origin)? What was the qualification task?
- Appendix l. 1014: I would recommend using an institutional email address not bound to a person's name here. That way, you can always forward to the currently responsible person without (having to remember) updating the data sheet.

End-User License Agreement (EULA)
You require the signee of the EULA to have a permanent position at their institution, effectively requiring even senior researchers employed as PostDocs to run the form by their formal bosses. 
Could you elaborate on the reasoning behind this?

Typos etc. (just those which I found passing by)
- l. 6: "conversations" -> "interactions" (?)
- l. 51: "sample-rates" -> "sampling rates" (?)
- l. 99: comma after "In this section"
- l. 161: comma after "issues"
- l. 177: eliminate "who are" (community is singular)
- l. 200: "the mobile phone" -> "a/their mobile phone" (?)
- l. 229: "For further insight performed" -> "For further insight, we performed"
- l. 244: comma after "task"
- l. 260: "as main evaluation metric" -> "as our main evaluation metric"
- l. 294: excess "e" in "development"
- l. 297: "open the floodgates for" -> "open the door to" (floods have a negative connotation)
- l. 300: excess space after [66]
- l. 300: no need for the "etc" (you already say "such as" in the previous line)
- l. 301: "Data-Collection" -> "Data Collection"
- l. 302: comma after "e.g."
- l. 305: "Regarding maximizing" -> "To maximize"
- l. 310: "long-term goal for" -> "long-term goal of"
- Appendix l. 828: Full stop missing after "(see Appendix C)".
- Appendix l. 1109: space missing after MSPN
review_title: Valuable data collection concept, dataset, and benchmark for capturing, annotating, and analyzing free-standing social interactions.

Meta-review: metareview: The paper proposes a dataset with human interactions recorded in an ACM MM workshop. It involves multiple overhead, privacy-preserving videos, audio and wearable sensors. The author group has also provided continuous key points, speaking statuses and F-formation annotations. The dataset supports person/keypoint detection, speaking status detection, and F-formation detection. In a broader context, this dataset can be useful for many applications such as security surveillance and social signal processing.

The reviewers found the paper to be well written and consider the dataset to be somewhat useful to the scientific community. One of the reviewers is concerned about the size and quality of the dataset to be helpful in driving novel AI/ML approaches in the NeurIPS community. 

The reviewers appreciate the authors' responses to their reviews and the changes implemented in the revised version of the manuscript. In particular, there has been a discussion related to potential ethical implications of this work. The reviewers seem to be satisfied with how the authors have addressed the ethical concerns. Hence, I recommend to accept this paper.


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept