Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a dataset to train AI models on computer vision tasks such as optical flow estimation, depth estimation, and self-motion estimation. The dataset is gathered by rendering camera views in virtual environments. The stereo cameras are designed to have similar attributes to insect eyes. The dataset includes synthetic visual views, motion flow, position, and orientation of the cameras. Cameras are designed to follow the visual features similar to insect eyes (4mm stereo baseline). The dataset gathering is inspired from insects but it does not provide any bio-inspired network architecture designs.
strengths: I really like the general direction of this paper, which provides some insights about how insects fly in complex environments. I indeed appreciate the deep knowledge of authors in the bio field, and how they tried to incorporate this knowledge in a virtual environment. In general, I am interested in combining the knowledge in various fields, and I believe authors’ general research direction will be really fruitful. I completely agree with the authors that using optical flow and wide field of view cameras can indeed improve navigation systems. Moreover, the dataset acquisition procedure is well described in detail. 
weaknesses: While I appreciate the authors’ efforts and their deep knowledge in the bio field, I have some suggestions/concerns related to the vision field that might help improve the work, as described in the following. 

**1. [Self-contained data?]:** In the supplementary (page 2), it is written that the dataset is self-contained. However, in the github page, only a sample of data is provided, which contains an example trajectory in a simple scene. As mentioned in the supplementary (page 5), authors do not own the rights to the 3D assets they used. Therefore, it seems that for each virtual scene, the code should be used to render and generate the data. Based on the paper (line 182-184), rendering time might take over a period of a few months. Authors should clarify whether they will provide the final rendered data or future users must render the scenes to generate the data. Without providing the final data, it will be really hard to use the dataset.

**2. [No Experiments]:** No experiments or evaluations are provided to support that this dataset can really improve the claimed applications in the computer vision domain: optical flow estimation, depth estimation, and self-motion estimation. Authors mentioned that this dataset is deliberately designed to help avoid over-fitting but no evidence is provided. In particular, the total number of virtual scenes used to create this dataset is 9 (2 outdoor and 7 indoor scenes), and I am not sure this is sufficient to generalize well on unseen real world scenes. To clarify the usefulness of this dataset, a comparison between 1) a network (e.g,, an optical flow estimator) trained using this dataset with 2) a network trained using existing datasets (or combination of this data with other ones) will be really useful.

**3. [No Training]:** In the conclusion (line 339-340), authors mentioned that “Our current work uses these data to train novel bio-inspired computer vision algorithms”. In the paper, I could not find any training procedure. This paper only provides a bio-inspired dataset without training it on any existing or proposed bio-inspired network architectures. Readers will benefit if authors clarify what they mean by training novel bio-inspired computer vision algorithms.

**4. [Trajectory designs seem independent to the environment]:** What makes insects interesting is their interaction with complex environments and their ability to fly and navigate in complex scenes with respect to objects in their surroundings. However, it seems that the camera trajectories are designed without considering the interaction with the environment. As far as I understood, some predefined camera trajectories are rendered in different virtual environments. This dataset can be more useful, if one could use this data to imitate the insect fly navigation (path planning) with respect to objects in a complex environment.
  
**5. [Why dynamically meaningful self-motion?]:** Authors might clarify why using dynamical meaningful motions is more effective than using any arbitrary motions in the self-motion estimation. Why does using insect motions help improve self-motion estimation in general applications? If the final goal was path planning or imitating flying paths of insects, it will be obvious that we must generate motion data similar to those of the insect motions. However, when the final goal is the self-motion estimation, maybe even using more various camera path shapes (e.g., splines with various shapes and moving cameras on them with different velocities in the range of insect or drone physical limitations)  provides a richer dataset, which covers a wider range of possible camera motions. To prove the claim that dynamically meaningful self-motion is useful for self-motion estimation, an experiment might be needed to compare a self-motion estimator trained using arbitrary paths against the one trained on dynamically meaningful self-motion paths.  

**6. [Application]:** Readers will benefit if authors can be more specific about the applications of this dataset in studying the insects vision system. Moreover, I think this data will benefit drone navigation if that drone can be as small as an insect or can have the same camera setup (2 cameras with 4 mm distance). However, I am not sure about its feasibility, considering current drone sizes and camera systems. If no drone exists with a similar camera setup, I am not sure how using 4mm stereo baseline can benefit drone self-motion estimation or path planning. Moreover, I am not sure why it is important to have 340 degree coverage similar to insects for drone navigation if having a 360 view field is possible for drones using one 360 VR camera.
 
correctness: The data collection procedure seems technically correct. 

clarity: It can be improved by further clarifying the applications of this dataset. In general, why this dataset should be used instead of existing ones for the claimed computer vision tasks (e.g., optical flow estimation) or how this dataset will improve the results of claimed tasks if it is aggregated with current datasets for training. 
relation to prior work: While the paper tries to faithfully discuss the relation to prior studies, it will benefit readers if authors further clarify the differences between this dataset and the ones in Table 1 with the checked stereo feature (column 6). Moreover, there are many studies in the computer graphics literature (especially TOG/SIGGRAPH) related to aerial/drone motion path planning and drone trajectory design, which are missing. These studies might help designing better drone trajectories (used in section 3.3.2) or even using their drone dynamical models to generate synthetic drone trajectories without a need for flying in a motion capture system. These dynamical models also consider physical limitations of the drone, similar to a real experiment. 
additional feedback: - In Table 2, for the blowfly Calliphora, 12,700 frames are provided with the frame per second rate of 16,620. This means that the whole data duration is less than 1 second for 36 different insect trajectories. Is this short amount of time enough for providing data that is a good representative of insect fly dynamics?

- If the purpose is self-motion estimation, drones have onboard sensor data that work quite well for estimating camera motion. What is the advantage of using this data to train a self-motion estimator over using onboard drone sensors? 

- Line 339:  “featuresextraction” should be changed to “feature extraction”
review_title: Interesting work with room for improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this manuscript, Leroy and Taylor describe both a generator of scenes using Blender, suitable to create pictures to train drones and robots for navigation using machine learning, and a dataset of 42,475 images containing depth information. Their dataset, FlyView, is calibrated to the FOV of the blowfly Calliphora stygia. The generator is impressive, allowing the use of custom trajectory with rotation and translation. One drawback of the current dataset is the low picture quality (1700x900 px), with new cameras having >4K resolution and the annotation of the different images in terms of object types (class/category), locations (3d space) and composition (environmental effects such as weather?) that is not presented in the manuscript. Also, the section comparing this dataset to the similar datasets  (section 4.2) could be improved. Nevertheless, the manuscript and developed Blender pipeline could be of in different training of neural networks, given that some information could be provided by the authors in modifying their pipeline to accommodate for other camera types and location. 
strengths: 1) Replicable code and creating of high fidelity (photo-realistic) images using Blender 
2)  Application of the pipeline to render images with different motion attributes and FOV. 
3) Good description of the Flyview dataset
4) The use of optical flow 


weaknesses: 1) Unclear which annotations is provided in the dataset, reading the manuscript. This is clarified in the Suppl. Information, but should be clearly stated in the manuscript.
2) The manuscript is centered around a set of images (The FlyView dataset) which might not be generalized to many applications. 
3) The authors should bonify section 2 and 4.1 with specific application in biological vision and use-case for the dataset. 
4) The rendering pipeline including the inputs and outputs should be described in more details.
5) There is, beside the depth information, no annotation of object in the dataset (bounding boxes or other), limiting its application in various Machine Learning training application.
6) Unclear if some environmental conditions (e.g. weather, flare) are included. This is important since in real-world applications, cloud covers or other factors will change the illumination of surfaces.
7) While training of novel computer vision algorithms and benchmark is mentioned in the conclusion, a Table of compatible drone or robotic platform could be provided to enhance the usefulness of this dataset in computer vision applications. 

correctness: 1) Please use the correct taxonomic identification for the blowfly. 
2) It is unclear what is the importance of figures A.1, B.1 and B.2, relating to the dataset. While those figures demonstrate the diversity of motions and colors in the datasets, a 3D view of the different drone or simulated trajectories such as Figure 2C or a trackview of the velocities/motion would better complement the dataset.
3) While the authors describe in great details the set up used to get the trajectory and motion capture, and the accuracy of this work using MoCap, no metric and error comparing real vs simulated dataset is provided.

clarity: The authors described the dataset and provided a datasheet. The text is well written, however, some information (see above) would complement this manuscript.
relation to prior work: Section 3.1 give a good description of the dataset with Table 1 and section 2. 
additional feedback: The ideas presented in this manuscript are interesting. However, the dataset per se seems limited in the research application and use of the dataset. A better overview of the use-case scenario for this dataset might be provided by the authors. 

Otherwise, a schematic view of the rendering process including input/output and rendering time per frame could be provided in the main manuscript. There is clearly a need for similar datasets in agriculture and automated driving or pipeline for such images generation. However, since different camera placement, type and characteristics will change the FOV and the resulting images, and since training DL networks required some fixed parameters, this part of the manuscript should be improved.
review_title: A pipeline to render photo-realistic  using Blender, inspired by the blowfly

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: **Summary of contribution**:
- This paper is generating a dataset that could be used for training deep learning models of optical flow estimation for self-motion estimation. 
- The paper mentions that there does not exist a dataset with panoramic view that provides optical flow ground truth for estimating "self-motion" -- where the "motion" is the flying pattern of an insect or a drone (micro-air vehicle). 
- To this end, they propose a dataset with panoramic views that mimics how insects and drones fly in different environments. 
- Quantitative improvement benchmarking on optical flow estimation or self-motion estimation tasks using this dataset is missing.


**Dataset information**:
- They generate the **synthetic data** using Blender and place the **three camera views** to cover **340 degree views** of the environment.
- They also provide intrinsics to convert between different camera views based on the application. Although authors mention this, I would like to know more about this claim -- for example: it would be really useful for the community if the authors could provide a streamline way of converting from panoramic view camera model to the generic pinhole camera model so that it could be used in different applications.
- RGB images, Forward-Backward motion flow, Self-motion (trajectory of the cameras), and Depth Map are provided for all the views.
- Dynamically meaningful camera trajectories are provided which could mimic how the insects and drones fly in the real world.
- Some comparative datasets are missing in the related works. 
strengths: 
**Large-scale dataset**: 

The major strength of this work is the generation of a simulated large scale optical flow, camera trajectory, RGB, and depth ground dataset with panoramic views. The camera trajectories in this dataset are dynamically meaningful since they mimic the trajectories of insects and drones. 

**Generalization ability**:

Moreover, since this dataset is covering varied environments, one strength of this approach is that the authors have taken into consideration what data splits are relevant keeping the generalization ability in mind if this dataset is used to train deep neural networks.

**Aerial vehicle research**: 

Since the camera trajectories mimic the trajectories of drones and insects, this dataset could find usage in the aerial vehicle community where they could train optical flow estimation or self-motion estimation tasks in simulation and replicate them in the real world.
 
weaknesses: **Quantification of improvement**:

One of the areas where this submission could be improved is in the task of showing how much value does this dataset actually bring to the table. The authors did not show any results on how much improvement can this dataset bring in the tasks of optical flow or self-motion estimation compared to earlier datasets. One simple way to do this would have been to pick any state-of-the-art (SOTA) methods for optical flow estimation or self-motion estimation, run them on existing datasets, and compare the same SOTA method using the proposed dataset. This metric could be very useful for the reader to understand where this dataset is bringing value or providing something which was not available in the earlier datasets.


**Related works**:

Although it is understandable that the authors are trying to formulate the problem as a panoramic view dataset mimicking the flying vehicles or flying insects, it would be useful for readers if the authors add the following related works, for completeness.
1. Yogamani, Senthil, et al. "Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
2. Artizzu, Charles-Olivier, et al. "Omniflownet: a perspective neural network adaptation for optical flow estimation in omnidirectional images." 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.
3. M. Yuan and C. Richardt, “360° optical flow using tangent images,” in Proc. BMVC, 2021
4. A. R. Sekkat et al., “SynWoodScape: Synthetic surround-view fisheye camera dataset for autonomous driving,” IEEE Robotics and Automation Letters, 2022.

correctness: The claims made in the submission are accurate and the dataset is constructed in a sound way. The datasheet is also provided along with the dataset in the supplementary materials which further covers the details of the dataset.
clarity: The paper is well written and easy to follow. There are no major issues with the way the dataset is documented and the datasheet submitted along with the dataset covers all the details relevant to the usage of the dataset.
relation to prior work: Please refer to the **Weaknesses** section.
additional feedback: It would be great to know what the authors respond to the questions stated in **Weaknesses** section. 


**Also, a question to the authors concerning drone research:**

How well do you think a drone trained on the simulated dataset would perform in the real world? Domain adaptation from sim-to-real has been an area of interest by many drone researchers and I wonder if this dataset could bridge that gap. Since many deep neural networks are running on the i.i.d. assumption, many times the domain change from simulation to real brings with itself the complicated problem of out-of-distribution samples. How well do you think the simulated data work in such scenarios? Are there any good precedents in the drone research where a system trained on simulated data transfers the knowledge reasonably well in the real world data?
review_title: This paper provides a panoramic view dataset, inspired from how biological insects view a scene. This dataset provides optical flow, self-motion, and RGB view ground truth data. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work presents a novel optical flow dataset with panoramic stereo images to estimate self-motion in flight. The authors reveal the importance of downstream tasks (such as self-motion estimation) for insects' navigation, so this bio-inspired optical flow dataset is designed to supplement the lack of such research in academia.

The main contributions of this paper are:
1.FlyView includes special panoramic RGB images modelled on the visual field of blowfly, which is of bionic significance and brings new challenges to both biology and computer vision.
2.FlyView is the first optical flow truth dataset to include motion dynamics relevant to flight.
3.FlyView offers an almost complete field of view for visual images, motion flow, and depth maps in diverse indoor and outdoor environments, and these data have rich research value for visual navigation.
strengths: significance of the contribution:
1.FlyView builds a bio-inspired optical flow truth dataset, that mimics insects to navigate by using optical flow with compound eyes. 
2.FlyView includes dynamically meaningful self-motion, and is the first optical flow truth dataset for motion flow and self-motion estimation.

relevance to the broader research community:
1.FlyView offers an almost complete field of view for visual images, motion flow, and depth maps in diverse indoor and outdoor environments, and these data have rich research value for many tasks such as optical flow estimation, depth estimation, and self-motion estimation.
2.FlyView includes special panoramic RGB images modelled on the visual field of blowflies' compound eyes, which is of bionic significance and brings new challenges to both biology and computer vision.

accessibility and accountability:
The dataset will be uploaded to Zenodo and HuggingFace, and download links will be shared on the Github repository.

ethical and social implications:
1.The dataset is self-contained and does not rely on any other dataset.
2.The dataset does not contain data that might be considered confidential
3.The dataset does not contain data that might be offensive, insulting, threatening, or might
cause anxiety, and is not related to people.
weaknesses: 1.The dataset is not currently implemented on a specific task or lacks a baseline model.

2.The font in the figure 5-7 is too small.

3.Appendix B can give more explanations about motion flow and scene frames.
correctness: The dataset was generated using Blender 2.93 LTS and its cycle engine for ray-tracing. Because visual images were generated using ray-tracing in Blender, the pixel intensity generated have to be post-processed to remove potential noise.
clarity: The paper basically describes the the motivation and methods of dataset establishment. The theoretical starting-point of this paper is logical. The writing quality is good.
relation to prior work: Yes, the dataset has several key features that are missing from existing optical flow datasets, including:
1. panoramic cameras with a monocular and binocular field of view matched to that of a fly’s compound eyes;
2. dynamically meaningful self-motion modelled on motion primitives, or the 3D trajectories of drones and flies;
3. complex natural and indoor environments including reflective surfaces.
additional feedback: First, although the dataset can be used to optical flow estimation, depth estimation, and self-motion estimation, the paper could discuss how the dataset are implemented on these tasks, especially in self-motion estimation.

Second, there are few experiments in this paper, which prove the rationality and availability of the dataset or prove the advantages of bionic optical flow dataset.
review_title: FlyView is a novel and valuable work for visual navigation.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors present an efficient optical flow truth dataset derived from panoramic stereo images to visual navigation. This paper attempts to highlight the capability of the proposed dataset, such as camera rendering method.
strengths: 1- The contributions of this study are clear.
2- Attempt at leveraging large-scale, stereo and complementary sources of information.
weaknesses: (1) Page 1, Lines 1 and 17: “complex natural and indoor environments” is conceptually unclear. Shadows and occluded areas, vegetation covers, dense areas, and large-scale variation are some challenges in natural and indoor environments. Please explain this fact.
(2) Experimental benchmarking analysis (numerical and graphical) should be added to learn about the accuracy of the proposed dataset. Experimental benchmarking should provide the readers with sufficient detail about the related methods to be able to reproduce the study if so desired. The authors should be aware that in any case, benchmarking studies are normal practice when proposing a new dataset.
correctness: There is no description of numerical results on state-of-the-art methods. The authors should be evaluating their dataset using several different comparisons.
clarity: Yes
relation to prior work: Recently, a new optical flow dataset, called SAMA-VTOL, from drone images was done by [1] in 2021. The related works and Table 1 can benefit from SAMA-VTOL.
[1] https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XLIV-M-3-2021/1/2021/ 
additional feedback: Due to the proposed dataset being very interesting for aerospace and remote sensing researchers, a demo code with a tutorial video should be posted on github for readers.
review_title: A new optical flow truth dataset that contains panoramic RGB images rendered in both monocular and binocular views

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors propose a bio inspired simulated dataset for optical flow based egomotion estimation.

The dataset has been generated from 9 static scenes and contains 42'475 images triplets (each triplet containing 3 cameras). Different motion models have been used including basic motions like pure drifts or pure rotations, as well a natural motions of an UAV and a fly trajectory.
strengths: Overall the paper is well motivated (e.g. the problematic has direct applications and the use of large field of view cameras is justified by the authors when they explain that it ensure the motion vector is present in the field of view of the camera), and well written (i.e., pleasant to read). The dataset has no equivalent in prior art and would be useful in practice.
weaknesses: The major weakness of this work is that all scenes are static (see my comment under Correctness for more details).

Two other points that bother me, is that no evaluation of any kind is conducted on the dataset and no real data is acquired.

The authors claims on line 34 that the relevant metrics should be on downstream tasks. While getting the ground truth optical flow for real images is unpractical, generating the ground truth for a downstream task, e.g. egomotion, should be easier. This should enable the generation of real testing data.

If such real test data was generated, it would be possible to show the benefits of the proposed dataset for training, which is not done at the moment.

Overall, since this paper is submitted to a data specific track, these weaknesses are not game breaker, as the proposed dataset still have clear benefits over the state of the art, but they definitely makes that paper borderline.


correctness: The authors used simulation to create their dataset, claiming that real acquisition methods are complex and unprecise. I mostly agree with that statement, but there is one details of utter importance for which I disagree: this is that using a technique like the one used for the Kitti dataset for static scene when subpixel accurate optical flow is not required is still sound and valid. 

Now the point is, the proposed dataset contains only static scenes, and the targeted application (i.e. egomotion) do not require subpixel accurate optical flow. In my opinion this is the major weakness of this work. It would be highly desirable that the authors add at least one scene with rigid moving object and one scene with non-rigid moving objects, at least for evaluation purposes.
clarity: Yes
relation to prior work: The relevant literature has been covered and the differences between previous works and this work have been highlighted.
additional feedback: I find the comparison to the brain of a fly, used on lines 3 to 5 and again on line 21 to be completely uninformative for computer scientists. For comparison with artificial neural networks (which, as far as I know, being software do not weight anything), it might be better to report the number of neurons in a fly's brain (which is apparently 100'000, if I believe the claim in "A Complete Electron Microscopy Volume of the Brain of Adult Drosophila melanogaster [ https://www.cell.com/cell/fulltext/S0092-8674(18)30787-6 ]"), as the number of neurons can be compared with the number of "neurons" in an artificial neural network.

A reference might be needed to support the sentence between lines 21 and 23 about the use of optical flow for drone egomotion but this is a minor issue.

One line 61 the authors claim that their dataset is exceptionally large: this is wrong. While the dataset rank second in therm of number of images, it has a limited number of scenes (only 9, even if the kind of scenes seems more diverse than in other works). I would reformulate that statement to something a bit more objective, underlining the fact that the dataset has a large number of images, spread across a limited number of diverse scenes. It is also important to underline that the scenes are realistic, which is not always a given for simulated datasets.

Finally I was wondering if the authors used the cycles or eevee rendering engines, and if both would be fine or if certain limitations impose the use of cycles ? Eevee is generally much, much faster which is highly beneficial when generating a large amount of simulated images.

Overall, I feel that this work is clearly borderline paper. Some limitations are pretty severe but the dataset is well motivated, has no strict equivalent in pre-existing datasets and would be useful in practice. I would surely upgrade my review to something more positive if the authors were to add at least one scene with moving rigid objects and one scene with moving non-rigid objects, those would be really useful for evaluation and are a cases where the use of simulated data is far more justifiable.
review_title: Review for FlyView

Meta-review: metareview: Accept(Poster), The paper received reviews leaning towards accept. The rebuttal by authors addressed a number of concerns, it would be useful to see improvements to the documentation of the dataset in the final version of the paper.
confidence: 3: The area chair is somewhat confident
Decision: Accept