Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a novel dataset of depth maps and stereo images called SurDis, it exemplifies the issue of surface discontinuity in the urban areas of Klang Valley, Malaysia.

The dataset contains: 

1- more than 17000 depth maps generated from 200 sets of stereo image sequences, 

2- annotations of surface discontinuity in the depth maps, and 

3- bitmap stereo image pairs corresponding to the depth maps in (1).
strengths: The paper and the presented dataset address an important topic to society of assistive technology. It shows superiority over the cited existing datasets. The dataset is accessible via Github and well explained.   The dataset has claimed several potential usages like developing wearable assistive devices, integrating with other datasets, and evaluating the level of hazardous. The reviewer agrees on all of them. 
weaknesses: The main weakness of the presented work is that it does not show a comprehensive analysis and experimentation to show the need to and the importance of the proposed dataset. Such experimentation support the potential usegs. Even though reviewer agrees on these potentials but they are not proven and supported experimentally. Generally, there is no practical motivation to use the dataset by researchers in future. 
correctness: The dataset is constructed and annotated correctly in a sound way. 
clarity: The paper needs a couple of improvements here and there. For example, the Introduction section does make a strong and clear claim of the contribution leaving the reader guessing about what to read. Figure 4 does not convey any information leaving questions about why to fill this much space. Figure 5 is not readable. Equation 1 is not strongly connected to the text and not written in a compact way. 
relation to prior work: The dataset makes an advanced step beyond the existing datasets. It is well connected to the prior work.
additional feedback: The dataset is interesting. The main comment is that several demos need to be experimented with, in order to motivate the usage of the data, in addition to the experimental comparison of the previous datasets.
review_title: An interesting social support to BLV community.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a dataset of labeled surface discontinuities in depth maps and stereo images. The dataset can be used to create assistive technology for blind and low vision people. The authors provide substantial detail on the specific data collection procedure and artifacts of the dataset.

strengths:  - The authors design the dataset around feedback from potential users of the assistive technology. Using 3d printed models as aids for identifying the appropriate discontinuities to target was an excellent addition.
- The specific data collection procedure is outlined in detail
- The data is permissively licensed, easily accessible, and well documented

weaknesses: - The authors did not implement a baseline model to evaluate the usefulness of the dataset as the training set for an ML application.
- The paper does not evaluate how sensitive an assistive system would be to the camera configuration and environment, although the authors do acknowledge that this is a limitation

correctness: The claims are correct to my knowledge
clarity: The paper is well written
relation to prior work: Prior work is discussed in the intro and the authors do a good job explaining how their dataset fills a gap in the existing work
additional feedback: I think this is a great dataset. I would suggest the authors create a baseline model, a well-defined evaluation procedure, and a leaderboard in order to encourage others to use the data.

The authors could also enlist the help of the community to collect more data in their area using the same prototype data collection tool. Crowd sourcing the next iteration of the dataset seems critical to expanding its size, which is necessary for future adoption.
review_title: A great dataset for assistive technology

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors introduce SurDis, a dataset of surface discontinuities captured by a chest-mounted depth camera. The dataset and models trained using it could be useful for improving navigation for blind and visually-impaired people by automatically detecting discontinuities (hazards) in front of them. In contrast to other relevant datasets, SurDis was collected in a unique urban environment where such surface discontinuities are more prevalent.

strengths: This paper has several strengths, and I appreciate the problem that it is trying to address. Blind navigation is a difficult problem, especially in less-developed urban environments.
-it is clear the technology and collection platform was developed in collaboration with blind people and experts in the area. The chest-mounted device can be used in conjunction with existing navigation apparatus (e.g., cane). The researchers also consulted O&M experts and used 3D tactile models to inform data collection with blind people.
-as the authors state, the dataset is collected in a unique environment where accessibility challenges are perhaps more widespread (urban Malaysia). A model or technique that succeeds under these conditions could perhaps be more useful in practice.
-the dataset's focus on depth map sequences as its primary modality could support diverse areas of ML research: traditional vision-based approaches (e.g., image/video object detection/segmentation) as well as potentially point-cloud models.

weaknesses: My main concern with this work is that no experiments were performed to validate the usefulness of the dataset. While the focus of the paper is the dataset itself, I would like to see some evidence (e.g., preliminary/baseline models that suggest it's possible to detect surface discontinuities using this dataset). Otherwise, it is unclear if the data is too "noisy" to be useful for ML approaches. I believe it is possible to address these concerns during the rebuttal period and I am open to raising my score if they are.
-lacking empirical experiments: the paper lacks any preliminary experiments that show the data is useful for learning detection models. Such experiments would also be useful for delineating the set of tasks that this dataset could be used to support. A simple object detector model trained on the depth maps could be a good start, since the annotations are already in the COCO format. 
-missing some details: while some parts of the data collection were thorough, others were not. for example, how many 3D models were used in the pre-collection stage with blind participants? Of these, how were the eventual 5 classes chosen? It was also difficult to find info about the relative frequency of these 5 classes throughout the dataset.
-minor writing problems: the paper contains some minor writing errors that could easily be fixed with an automated grammar checker.

correctness: As I mentioned earlier in my review, I appreciate the collection procedure where the authors consulted with blind people and experts in the field. That leads me to believe that the types of scenarios, discontinuity types, and collection apparatus are ecologically valid. However, I believe more evidence is needed to show that the dataset is diverse, large, and clean enough to support machine learning in this space.

clarity: besides a few minor writing errors, the paper was easy to follow. Some details are missing and should be included (see weaknesses).

relation to prior work: The paper does discuss its relation to prior work, namely other datasets used for blind navigation. Differences include the collection environment and the unique collection apparatus that was built for this data collection. One tangentially related work that I happen to know of in this area (Project Sidewalk (https://sidewalk-sea.cs.washington.edu)) was not and probably should be included.

additional feedback: * Post rebuttal comments:
The authors have thoroughly addressed my initial concerns and I believe the paper stands much stronger now. I would argue for its acceptance.
review_title: This work tackles an important problem but it is hard to gauge how useful the dataset is without any experiments.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The main contribution of this work is a collection of labeled depth maps of 10 locations that contain different surfaces. It is the first dataset that considers surface discontinuity while creating mobility tools for blind or low vision (BLV) people. 
strengths: The creation of mobility tools for BLV people seldom considers problems with surfaces, assuming their continuity. The set of depth maps from this collection would be helpful to address this issue if the surface discontinuity poses a serious problem in the area of application. Instead of using typical hardware, the authors build a wearable sensor and used it to capture the data.  
weaknesses: The number of works in the literature on the subject of surface discontinuity for wearable sensors for BLV people is limited, confining the number of researchers interested in the usage of the dataset. The mentioned in the paper way of integrating the dataset with other datasets is not shown and requires further investigation on the possible problems with implementation. The dataset is collected in a specific environment which for many application domains would not be relevant (buildings, even pavements) or is not representative enough (country road, a footpath in the park). It is not certain if applications developed with depth maps created with the specific wearable sensor would work with other depth sensors.    
correctness: The submission seems correct. However, it lacks information on the diversity of classes. Is the dataset balanced? Also, there are only five classes. In my opinion, examples of flat surfaces or surfaces assessed by a specialist as non-hazardous should be present in the dataset. If a “normal” case (the sixth class) is not present, a surface classifier will always assign one of those classes to input depth maps. This requires clarification and explanation. Furthermore, it should be explicitly ensured in the text that depth maps from testing and training subsets are disjoint.  
    

clarity: The quality of the paper is acceptable. Please proofread it.  
relation to prior work: The dataset is compared with data collections designed for a similar purpose. 
additional feedback: 1.	It is not clear how printed 3D replicas would support potential users of the dataset. 
2.	The limitations imposed by the wearable sensors should be discussed. Researchers would create their own sensors and datasets instead of using a dataset and have problems with sensor construction with parts that may not be available in the nearest future. 
3.	The integration of different sensors or depth maps, as mentioned, should be investigated. What would the extension of the dataset look like? Readers are encouraged to do it but would they know how? Are the depth maps from different sources easily merged or used together in the same software? 
4.	Are the selected ten locations enough to cover the variability of the surfaces in urban environments? How do we know they are representative?
5.	A performance evaluation of a typical machine learning method on this dataset would be a nice addition to show the usability and difficulty of the dataset.
6.	At https://github.com/kuanyewleong/surdis it is written that “it is reccommended that when you are training your model using this dataset, you may slightly expand the bounding boxes (i.e. add a few units of pixel to all edges).” This would make the repeatability of the results for different researchers more difficult. Why was the suggested addition not performed?
review_title: A surface discontinuity dataset for navigation in urban environments with a wearable technology

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a novel dataset that focuses on detecting surface discontinuities to help blind people navigate in urban areas. The dataset is collected from 200 sets of stereo image sequences from 10 different locations in Klang Valley, Malaysia. 
strengths: 1 The paper is well motivated. In real world, identifying discontinuous surface areas is important to blind people. Previous works primarily focus on obstacles detection. The proposed SurDis can facilitate future research on discontinuous surface detection.

2 The images are collected from Malaysia and might be more challenging than images collected from developed countries.

3 The stereo pairs can provide valuable depth information about the scene.
weaknesses: 1 It's beneficial to evaluate recent frameworks for object detection on SurDis. The examples in Fig. 9 seem to be easy. Thereby my major concern is the SurDis might be overwhelmed by existing frameworks even without depth information.

2 Selecting 10% of each sequence as test set might leak label information, because the images within a sequence are quite similar with each other.

3 In Sec 2.1, how many 3D replicas take part in the survey? (including the 3D replicas that are viewed as safe or easy by volunteers)




correctness: Selecting 10% of each sequence as test set might leak label information, because the images within a sequence are quite similar with each other.
clarity: This paper is well written and easy to understand.

relation to prior work: Related works have been discussed in detail.
additional feedback: It will be great to evaluate recent frameworks for object detection on SurDis.
review_title: Review for SurDis

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors spared much effort to collect a set of Real path surface discontinuity data, aiming to cultivate and help researchers to assist the Blind and Low Vision people’s daily activities. The work is of good motivation and tends to solve real-world problems, which is worth encouragement.
In the data collecting process, the authors followed strict and rigorous procedures, from in-person consultation, and instrument development to data sampling, and afterward data quality analysis and preprocessing, making such resources reliable to support future work.
A dataset with long-term value usually requires a well-arranged maintenance plan, the author has done good collecting work, and some details for future management will be better. Besides, if the author could explore its usage in the real-world application using provided dataset would be better.

strengths: The dataset is ground-breaking in the blind and low vision problem solutions. It has the potential to boost such areas’ application development and research and to better serve people with visual impairments. 

Judging from the data collection procedures, the dataset is well structured and source-reliable, It’s possible to broader the research community and arise studier's awareness of public service.

Also, the provided dataset could help in realistic wearable equipment designs.

weaknesses: The work mainly collected and introduced a pure dataset and its collection procedures. However, its real value on how exactly the data would help is not demonstrated, if the author could show a study case to reveal the data’s potential.

There is no example code for the usage of the dataset, if the author could demonstrate a piece of toy code, it will help users to get started easily. 

correctness: The dataset is constructed in a sound way with reliable conducting procedures.
clarity: The paper is written relatively clearly, but some minor typos should be examined. Such as: Typo problem at the beginning of paragraph 5, Introduction part “VIsual”. and the Reference first line, an extra line should be deleted.
relation to prior work: The paper demonstrates related previous published work, with a clear explanation and introduced the proposed work's uniqueness.
additional feedback: Please see my comments above.
review_title: New dataset but lack of sample codes.

Meta-review: metareview: All reviewers recommend that the paper be accepted. The AC sees no basis to overturn reviews, and recommends that the paper by accepted. The authors should attend to main points in the reviews when preparing the paper's final version. This includes Reviewer S4AE's suggestions about addressing diversity and balance, and Reviewer CSu3's suggestion about evaluating recent object detectors, and Reviewer Vbjb's questions about examples.
confidence: 3: The area chair is somewhat confident
Decision: Accept