SpaceSet: A Large-scale Realistic Space-based Image Dataset for Space Situational Awareness

Authors: Rangya Zhang, Jiaping Xiao, Yuhang Zhang, Lu Bai, Qianlei Jia, Mir Feroskhan
Primary area: datasets and benchmarks
Keywords: space situational awareness, object detection and tracking, space image dataset, high resolution image

Abstract:
Space situational awareness (SSA) plays an imperative role in maintaining safe space operations, especially given the increasingly congested space traffic around Earth. Space-based SSA offers a flexible and lightweight solution compared to traditional ground-based SSA. With advanced machine learning approaches, space-based SSA can extract features from high-resolution images in space to detect and track resident space objects (RSOs). However, existing spacecraft image datasets, such as SPARK, fall short of providing realistic camera observations, rendering the derived algorithms unsuitable for real SSA systems. In this research, we introduce SpaceSet, a large-scale realistic space-based image dataset for SSA. We consider accurate space orbit dynamics and a physical camera model with various noise distributions, generating images at the photon level. To extend the available observation window, four overlapping cameras are simulated with a fixed rotation angle. SpaceSet includes images of RSOs observed from $19 km$ to $63,000 km$, captured by a tracker operating in LEO, MEO, and GEO orbits over a period of $5,000$ seconds. Each image has a resolution of $4418 \times 4418$ pixels, providing detailed features for developing advanced SSA approaches. We split the dataset into three subsets: SpaceSet-100, SpaceSet-5000, and SpaceSet-full, catering to various image processing applications. The SpaceSet-full corpus includes a comprehensive data-loader with $781.5GB$ of images and $25.9MB$ of ground truth labels. We also benchmark detection and tracking algorithms on the SpaceSet-100 dataset using a specified splitting method to accelerate the training process.

Decision: reject
Reviewer scores: 6, 5, 5, 10

Reviews and discussion:
<thread>
creation date:
1730030897804

thread_type:
official_review

<thread_object>
creator: reviewer_ajkv

<content>
summary:
Given my limited expertise in the field of space observations, I find myself ill-equipped to provide a comprehensive evaluation of this paper. The specific challenges and nuances within this domain are not within my area of specialization. Therefore, I recommend that you consult with a diverse group of reviewers who possess a deeper understanding of space-related research to ensure a thorough and informed assessment of the paper's content and significance.

strengths:
Given my limited expertise in the field of space observations, I find myself ill-equipped to provide a comprehensive evaluation of this paper. The specific challenges and nuances within this domain are not within my area of specialization. Therefore, I recommend that you consult with a diverse group of reviewers who possess a deeper understanding of space-related research to ensure a thorough and informed assessment of the paper's content and significance.

weaknesses:
Given my limited expertise in the field of space observations, I find myself ill-equipped to provide a comprehensive evaluation of this paper. The specific challenges and nuances within this domain are not within my area of specialization. Therefore, I recommend that you consult with a diverse group of reviewers who possess a deeper understanding of space-related research to ensure a thorough and informed assessment of the paper's content and significance.

====

**Post-Rebuttal**: I think it is a good topic in AI4Science. However, my expertise does not extend into the realm of space observations. Consequently, I am not in a position to accurately assess the quality of the data or the principal challenges it presents. Therefore, I must express my recommendation for this paper with a degree of caution and limited confidence, acknowledging the limitations of my own knowledge in this specific area.

questions:
Given my limited expertise in the field of space observations, I find myself ill-equipped to provide a comprehensive evaluation of this paper. The specific challenges and nuances within this domain are not within my area of specialization. Therefore, I recommend that you consult with a diverse group of reviewers who possess a deeper understanding of space-related research to ensure a thorough and informed assessment of the paper's content and significance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730531131637

thread_type:
official_review

<thread_object>
creator: reviewer_qnxp

<content>
summary:
This paper introduces SpaceSet, a large-scale, realistic image dataset designed to enhance space situational awareness (SSA) for tracking and monitoring resident space objects (RSOs). Unlike previous datasets, SpaceSet incorporates accurate orbital dynamics and a physical camera model with photon-level noise distributions to produce realistic space-based images. Simulated from multiple orbital perspectives (LEO, MEO, GEO), the dataset covers distances from 19 km to 63,000 km and provides high-resolution images (4418 × 4418 pixels) suitable for advanced SSA methods. It includes three subsets—SpaceSet-100, SpaceSet-5000, and SpaceSet-full—addressing various image processing needs, along with a benchmark evaluation for detection and tracking algorithms.

strengths:
The main contribution of this paper is a large-scale, realistic image dataset designed to enhance space situational awareness (SSA) for tracking and monitoring resident space objects (RSOs). The dataset has a few merits compared to existing ones:

+ The proposed dataset incorporates realistic orbital dynamics and a camera model with photon-level noise, enhancing its applicability to real-world SSA tasks.

+ The image resolution in the proposed dataset is high (4418 × 4418 pixels). It covers multiple orbital altitudes (LEO, MEO, GEO) and observation distances (19km to 63,000km)

weaknesses:
My major concern is the limited contribution. A pure dataset contribution may not align with the topic of the ICLR conference (Learning Representations). This paper benchmarks many existing detection and tracking algorithms on the proposed subset (SpaceSet-100). However, no new algorithms regarding learning representations are provided. 

Another concern is the dataset setup. I understand the proposed one is already closer to real settings than previous ones, but it is still a synthetic dataset. I wonder what the gap is between the proposed synthetic and realistic settings. For example, the dataset uses a fixed camera setup, four overlapping cameras, and a fixed rotation angle. Is this always the real application scenario? Will any missions require a different number of cameras with other relative pose settings (e.g., unstructured)?

Is there any available real dataset that could be used to evaluate the quality of the synthetic data or the synthetic-to-real generalization ability of an algorithm? 

Some illustrations might be clearer. For example, from Fig.1, I cannot figure out why (a) shows the realistic exposure with noise distribution, how exposure is reflected, what the noise distribution is, and why they are realistic. (b) shows a picture from an existing dataset, SPARK, but there isn't an image of the proposed dataset for comparison. (c) is almost black and I do not know where to put my focus.

questions:
Plz refer to my comments in "Weakness".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720812405

thread_type:
official_review

<thread_object>
creator: reviewer_egtv

<content>
summary:
To improve Space Situational Awareness (SSA), the authors introduced SpaceSet, a comprehensive, large-scale dataset of high-resolution spaced-based images, designed to address the limitations of existing simulated datasets. This datasets consists of images generated with accurate orbital dynamics and a physical camera model with various noise distributions, capturing observations from altitudes between 19 km and 63,000 km. In experimental section, the authors provided the object detection and tracking benchmarks. The benchmark indicates that a series of YOLO-8 can show the better performance in the computational overhead and the accuracy.

strengths:
First of all, the motivation of the paper seems to be meaningful and pragmatic in the perspective of addressing limitations in the existing dataset. 

SpaceSet contains high-resolution scenes and more RSOs than existing datasets and also contains multi-camera images that consider various physical factors including noise, object properties, camera models, and locations.

To validate its effectiveness as a benchmark for object detection and tracking, the authors selected the most widely used models and conducted meaningful experiments.

weaknesses:
As a benchmark paper, it lacks many details and analysis on dataset itself. Except for data generation, the information such as the properties of RSOs such as size and orientations, the positional distribution of frequently appearing locations in the images, and criterion for image splits is needed. Furthermore, there is a lack of detail regarding the benchmark experiments such as whether all models used input images of the same size or why do existing SOTA object detection and tracking methods not perform as well as expected on SpaceSet-100?

Another concern is the completeness of the paper. Although the completeness is an essential aspect of a paper, this paper contains awkward sentences, typos (e.g., L.140, 210, obit), grammatical errors (e.g., L.216 box are -> is). Regardless of whether these issues hinder understanding of the content, the lack of attention to such things is problematic.

Lastly, the content in A.5 intended to validate the proposed dataset’s effectiveness in real-world data is not clearly conveyed. It would be beneficial to emphasize in the main paper why these metrics provided in Table.8 and 9 are meaningful.

questions:
I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper. If the rebuttal period resolves the above-mentioned concerns, I will gladly raise my score. Also, there are many vague sentences and grammatical errors in the paper. I recommend that the author will revise the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730810288688

thread_type:
official_review

<thread_object>
creator: reviewer_ngxj

<content>
summary:
This paper proposes a large-scale realistic space-based image dataset for space situational awareness. The dataset contains 20k images with 673 objects. The paper describes the detailed pipeline of data curation and data annotation. Additionally, the authors construct an object detection and tracking benchmark based on the proposed dataset and provide several baseline results.

strengths:
1. This paper proposes a new large scale dataset for space situational awareness. Comparing to previous public avaiable dataset available in this filed, the proposed dataset is the first realistic image dataset at the photon level.
2. This paper constructs object detection and tracking benchmark on the proposed dataset, and implemented several baselines methods on the dataset.

weaknesses:
1. Considering the relatively small number of target objects (673) in the dataset, it is crucial for the authors to perform multiple experimental runs for object tracking tasks to obtain statistical mean values and variances. The authors should ensure that the variation in evaluation metrics across different runs (e.g. 3 or 5 runs) remains below a specified threshold (e.g., 0.5) to establish the reliability and stability of the results.
2. Given that object scale is a critical factor influencing detection performance, the authors should provide comprehensive statistical analysis of object scales in the dataset, such as the distribution of object sizes (e.g., small, medium, and large objects). This information would help readers better understand the dataset characteristics and evaluation results.

questions:
1. Please provide multi-run results with mean and variances on the object detection and object tracking benchmarks to ensure the variances of multiple-runs are small.
2. The authors could include baseline results of specialized small object detection methods, such as approaches [1] that have shown promising performance on the VisDrone [2] benchmark.

[1] QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection.

[2] VisDrone-DET2019: The Vision Meets Drone Object Detection in Image Challenge Results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732696945075

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Rebuttal by Authors

comment:
We would like to thank all the reviewers (ngxj, EGtv, qNXp, and ajKv) for their valuable suggestions and positive evaluations of our contributions to "a new large-scale dataset for space situational awareness", "the motivation being meaningful and pragmatic in the perspective of addressing limitations in the existing dataset", and "a few merits compared to existing ones". The main concerns mentioned are summarized here, and the reviewers' comments have been addressed one by one. We sincerely hope that all the concerns have been addressed, and we are willing to answer any questions raised during the discussion period. We have attached the revised paper containing new Figures and Tables referenced in our following responses.

>**More dataset analysis**

We have conducted more in-depth analysis (suggested by ngxj and EGtv) beyond the existing SNR, average RMS contrast, and the distribution of observation ranges, including the size distribution of the whole dataset categorized by the classes of LEO, MEO, and GEO in Figure 6. The positional distribution of the observed objects in the image coordinate is also analyzed in Figure 5.

>**More experiments and comparison**

We have included more runs to obtain the mean and standard deviation of the performance (suggested by ngxj). The results show the robust performance of different detection and tracking algorithms. We have also tested one more baseline method [1] as suggested by Reviewer ngxj. The rationale and experimental details (suggested by EGtv) are further explained in the revised paper Section 5.2.

>**Dataset  uniqueness and validity**

We have included more details and explanations of the validation of our dataset. Currently, there are fewer than six datasets in this field, all simulations, as NASA has not made its database public, but our dataset provides the first large-scale realistic space-based image dataset from the photon level, which bridges the gap from simulated to real-world data. Most existing datasets, such as BUAA-SID-share 1.0 [2], SPARK [3], and the Space Target Dataset [4], are generated for satellite pose estimation and space target classification in ideal simulation without noise. These datasets typically focus on capturing targets from close distances and multiple angles to highlight a single target's characteristics. Their approach differs significantly from our dataset, where targets are captured at various distances based on realistic space-based camera specifications and physical models. Compared to the images from the ground-based telescope and the space-based camera in [5], the characteristics of objects and noise distributions highlight the effectiveness of our dataset. We have further explained the realistic characteristics and significant contributions of our dataset for learning representations, as we replied to Reviewer qNXp.

We have carefully proofread our paper and made revisions to avoid any typos and grammatical errors (suggested by EGtv). 

[1] Yang, Chenhongyi, et al. "QueryDet: Cascaded sparse query for accelerating high-resolution small object detection." In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 13668-13677. 2022.

[2] Haopeng Zhang, Zhengyi Liu, Zhiguo Jiang, Meng An, and Danpei Zhao. "Buaa-sid1.0 space object image dataset". Spacecraft recovery & remote sensing, 31(4) (2010):65–71.

[3] Musallam, Mohamed Adel, Vincent Gaudilliere, Enjie Ghorbel, Kassem Al Ismaeil, Marcos Damian Perez, Michel Poucet, and Djamila Aouada. "Spacecraft recognition leveraging knowledge of space environment: simulator, dataset, competition design and analysis." In 2021 IEEE International Conference on Image Processing Challenges (ICIPC), pp. 11-15. IEEE, 2021.

[4] Zhang, Zipeng, Chenwei Deng, and Zhiyuan Deng. "A diverse space target dataset with multidebris and realistic on-orbit environment." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 15 (2022): 9102-9114.

[5] Dignam, Aishling, et al. "In-Space Situational Awareness: Developing Spaceborne Sensors for Detecting, Tracking and Characterising Space Debris." Proc. 2nd NEO and Debris Detection Conference. 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary
The paper introduces SpaceSet, a large-scale realistic space-based image dataset for space situational awareness. It contains 20k images with 673 objects, and includes a pipeline for data curation and annotation. The dataset addresses limitations in existing simulated datasets and provides high-resolution images suitable for advanced Space Situational Awareness methods.

## Strengths
* Proposes the first realistic image dataset at the photon level for space situational awareness, and constructs an object detection and tracking benchmark
* Implements several baseline methods to validate its effectiveness.
* Features high-resolution scenes, more RSOs, and multi-camera images considering various physical factors.
* Enhances applicability to real-world space situational awareness tasks.

## Weaknesses
* The need for detailed information on the dataset, including properties of RSOs, positional distribution of frequently appearing locations, and criterion for image splits.
* The paper's completeness is questioned due to awkward sentences, typos, and grammatical errors.
* The content in A.5 intended to validate the proposed dataset’s effectiveness in real-world data is not clearly conveyed.
* The paper's contribution may not align with the topic of the ICLR conference (Learning Representations), as it benchmarks many existing detection and tracking algorithms on the proposed subset (SpaceSet-100).
* The proposed dataset setup is questioned, with questions about the gap between the proposed synthetic and realistic settings.
* The paper also lacks an available real dataset to evaluate the quality of the synthetic data or the synthetic-to-real generalization ability of an algorithm.

## Conclusions
Two reviewers has not enough expertise to assess the quality of the paper although one of them mentioned that the paper would fit in a AI4Science. The other reviewers have more concerns about the paper although there was not almost no discussion during the rebuttal. Based on the reviews and author’s feedback, the paper should have a more extensive experinents and comparisons before being accepted. For instance, how the size of the slices affects the methods, the qualititive difference between Object detectors, apply SOTA algorithms in OD and tracking.