Satformer: Accurate and Robust Traffic Data Estimation for Satellite Networks

Authors: liang  qin
xiyuan  liu
wenting  wei
chengbin  liang
huaxi  gu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:41:11

Abstract: The operations and maintenance of satellite networks heavily depend on traffic measurements. Due to the large-scale and highly dynamic nature of satellite networks, global measurement encounters significant challenges in terms of complexity and overhead. Estimating global network traffic data from partial traffic measurements is a promising solution. However, the majority of current estimation methods concentrate on low-rank linear decomposition, which is unable to accurately estimate. The reason lies in its inability to capture the intricate nonlinear spatio-temporal relationship found in large-scale, highly dynamic traffic data. This paper proposes Satformer, an accurate and robust method for estimating traffic data in satellite networks. In Satformer, we innovatively incorporate an adaptive sparse spatio-temporal attention mechanism. In the mechanism, more attention is paid to specific local regions of the input tensor to improve the model's sensitivity on details and patterns. This method enhances its capability to capture nonlinear spatio-temporal relationships. Experiments on small, medium, and large-scale satellite networks datasets demonstrate that Satformer outperforms mathematical and neural baseline methods notably. It provides substantial improvements in reducing errors and maintaining robustness, especially for larger networks. The approach shows promise for deployment in actual systems.

Main paper: 
 introduction: 
 As a potential complement to terrestrial networks, satellite networks are envisioned to provide broadband connectivity with seamless coverage and in a cost-effective manner. Internet service and content providers are interested in satellite networks due to their wide international coverage and lower entry costs in rural and underdeveloped areas [1] .Traffic engineering [2, 3, 4] and topology engineering [5] of satellite networks, such as access control, routing and congestion control, are key to achieve efficient control of satellite networks, which rely on real-time perception of global traffic data [6] . Timely and accurate traffic measurements beyond basic metrics are undoubtedly beneficial for such applications. However, it is troublesome and costly to collect massive traffic data by measuring all transmission pairs directly [7] , since traffic data is naturally distributed throughout the entire network. In order to support the network operation of the emerging mega-constellations, there is an urgent need to explore cost-effective traffic measurement methods. Traffic data estimation is a feasible approach for large-scale satellite networks, where the global traffic data can be estimated according to partial traffic sampling and measurement [8] .Due to inherently dynamic natures of spatial distances and orbital positions in satellite networks, traffic volumes and patterns vary over time [9] . Emerging mega-constellations networks typically involve numerous satellites, so the dynamic traffic data between satellite pairs can be represented as high-dimensional matrices or tensors. This complexity makes it difficult to capture the complicated relationships within the data [10] . Furthermore, the instability of inter-satellite and satellite-ground links often leads to the loss of traffic data during transmission. It should also be noted that not all satellite pairs have constant communication demands. As a result, the measured traffic data is often sparse and incomplete, making traffic data estimation complex [1] . Therefore, the primary challenge in accurately and robustly estimating satellite network traffic data lies in effectively capturing the complex and nonlinear spatio-temporal correlations while maintaining robustness for varying sequence lengths [11] .Indeed, most efforts in traffic data estimation focus solely on low-rank linear decomposition, which cannot effectively capture the nonlinear spatio-temporal correlations among large-scale and dynamic traffic data, leading to inaccurate estimations. Therefore, developing a novel approach is crucial for enhancing traffic estimation performance to effectively extract and utilize the complex and nonlinear spatio-temporal correlations among inter-satellite traffic data.For large-scale and highly dynamic satellite network traffic data, we propose Satformer, a new neural network architecture designed for accurate and robust traffic estimation. Satformer systematically constructs encoder-decoder components with stacked spatio-temporal modules to effectively capture complex spatio-temporal correlations in traffic data. Within each module, an adaptive sparse spatiotemporal attention mechanism (ASSIT) is adopted to extract key features from numerous sparse inputs by focusing on specific local regions. This enables Satformer to capture nuanced traffic patterns essential for accurate estimation. This is particularly useful in satellite networks where traffic may be concentrated in certain areas due to regional demand or satellite coverage. Additionally, ASSIT is more robust to sparsity as it can identify and focus on areas with higher data density, which may contain more informative traffic features, rather than being overwhelmed by overall sparsity. Simultaneously, we utilize a graph embedding module to effectively process non-Euclidean data through a Graph Convolutional Network (GCN). These components in spatio-temporal module enhance Satformer's ability to capture and exploit the nonlinear and complex information present in traffic data. Furthermore, a transfer module is incorporated to disseminate global context information throughout the model.Our contributions are as follows:• We designed ASSIT, which adopts a multi-head self-attention structure. It can learn the correlation representation of traffic data at different spatial and temporal scales. We added a sparsity threshold to the attention matrix to efficiently process a large number of sparse inputs. By dynamically adjusting the threshold value, ASSIT adapts to the sparsity levels of various datasets, thereby enhancing the model's inference efficiency. Additionally, ASSIT allows the model to dynamically allocate computational resources to regions of interest, making the model operate more efficiently and enhancing its scalability.• To process non-Euclidean structured data, we introduce a graph embedding module within each module via GCN. Since the graph embedding module learns the relationship between nodes and neighbors adequately, it can extract the local and global information of nodes from non-Euclidean structured data. It improves the ability of the model to extract nonlinear spatio-temporal correlation.• We add a transfer module to the Satformer framework, which can blend and reshape the traffic representation learned by the previous modules, conveying a global temporal and spatio perspective, while also helping to strengthen the generalization ability of the model on different types of datasets This paper is structured as follows. Section 2 surveys relevant research. Section 3 explains our proposed Satformer methodology. Section 4 presents experimental verification and comparisons. Section 5 makes a discussion and concludes this paper. 
related works: 
 We provide a review of the existing work on network traffic estimation. Existing traffic data estimation methods can be mainly divided into matrix completion based, tensor completion based and neural network based methods.Matrix Completion (MC) methods have found widespread application in the estimation of traffic data. Some algorithms, such as the convex relaxation method based on minimum nuclear norm approximation [12] and matrix factorization-based methods [13] , leverage the linear spatiotemporal characteristics of traffic data to infer missing values. However, these methods are often too simplistic, which can lead to inaccurate estimations when applied to large-scale traffic data.As an extension of matrix completion, the goal of tensor completion aims to reconstruct low-rank tensors based on sparse observations of their entries. Several studies have adopted tensor completion, including recent works [14, 15, 16] .To achieve higher accuracy in traffic data estimation, these works propose the use of tensor completion methods, which can more comprehensively capture spatio-temporal features in traffic data, effectively. A typical work of such a method is LTC [17] , which leverages the strong local correlation of the data to identify and complete each subtensor with low rank. However, many traffic estimation algorithms based on tensor completion rely on CANDECOMP/PARAFAC (CP) or Tucker decompositions, commonly using inner products as interaction functions. This approach can often reduce estimation performance to some extent due to its limited ability to capture both linear and nonlinear correlations in traffic data.In recent years, deep learning methods have shown notable advancements in traffic network analysis.Notably, research such as NTF [18] and [19] have explored the application of deep learning models, including Recurrent Neural Networks (RNNs), to achieve adaptive grouping and prediction of traffic tensors within large-scale networks. Noteworthy among these efforts is CoSTCo [20] , which incorporates two convolutional layers to extract features from stacked embeddings, enhancing awareness of network dynamics through the acquisition of complex spatio-temporal features. Recent studies [21, 22] employ meta-learning and other algorithms, alongside attention mechanisms, to dynamically adapt to rapid changes in traffic patterns within the network. However, current deep learning models may focus more on global features, while neglecting the local and hidden spatiotemporal correlations in traffic data, which may lead to suboptimal estimation effects 3 Estimation Model: Satformer 
system model & problem definition: 
 In satellite networks, inter-satellite traffic data can be modeled as a time-space matrix, which reflects the data volume to be transmitted between all node-node pairs over satellite networks. For the problem statement of traffic estimation over satellite networks, we introduce the following symbols: N : Number of satellites, T : Discrete time steps, we define the inter-satellite traffic matrix Y ∈ R I×J×T , where Y ijt represents the data transmission from satellite i to satellite j at time step t. The t-th layer of this matrix represents a discrete time step.Considering the influence of spatio distance and transmission delay in satellite networks, we can adjust the inter-satellite traffic by introducing a weight matrix. Let W ∈ R N ×N be the weight matrix representing spatial distance and transmission delay, where W ij denotes the weight from satellite i to satellite j. Thus, the adjusted inter-satellite traffic data matrix can be represented as Ẋ = Y W, where denotes element-wise multiplication. Taking into account these factors, the mathematical modeling of inter-satellite traffic data can be expressed as follows:EQUATIONwhere i, j = 1, 2, . . . , N , and t = 1, 2, . . . , T . This model considers the spatio distance and transmission delay between satellites, allowing the traffic data matrix to more accurately reflect the actual communication scenarios in the satellite networks. In the process of sampling and recovering inter-satellite traffic data, we begin by introducing the sampling matrix S, the sampled data X , and the nonlinear estimation function F . The sampling process can be expressed using mathematical notation: X = Ẋ S.This process retains elements in the inter-satellite traffic matrix Ẋ where the corresponding positions in the sampling matrix S are 1, while setting other positions to zero, resulting in the sampled data matrix X . To recover complete traffic data from the sampled data, we introduce a nonlinear estimation function F . This function involves a complex nonlinear mapping to better estimate actual traffic data.EQUATIONwhere X represents sampled data, and X is the recovered data obtained through the non-linear estimation function F . We design Satformer, a tensor completion model designed for the accurate and robust estimation of global traffic data in satellite networks. As illustrated in Fig. 1 , Satformer is structured as an encoderdecoder architecture, with both components featuring multiple spatio-temporal modules. Residual connections interlink these modules to prevent neural network degradation. Each spatio-temporal module comprises a Graph Embedding Module and a Satformer Block. The key of Satformer to improve the estimation accuracy is that it can extract features efficiently and accurately from a large number of sparse satellite network traffic data. This is achieved through adaptive sparse spatiotemporal attention inside each Satformer block, facilitating the estimation of traffic data. A transfer module facilitates the seamless transmission of features from the encoder to the decoder. The encoder encodes the input traffic information, while the decoder is tasked with estimating the missing traffic data. The subsequent section provides a detailed description of each module.Satformer utilizes spatio-temporal modules to extract spatio-temporal features from input tensors; this module primarily consists of graph embedding components and Satformer blocks.Graph Embedding: The Spatio-Temporal module serves the goal of extracting spatio-temporal features from input tensor. Considering the inherent high sparsity of observed traffic data in realworld, it becomes imperative to represent tensors as low-dimensional vectors. Through the learning of embedded representations for nodes, the model inherently captures both structural and semantic information of nodes within the graph. This capability enables the model to comprehend relationships between nodes more effectively, facilitating the extraction of meaningful features from the X ∈ R I×J×T . Each Origin-Destination (OD) pair corresponds to an origin node, a destination node, and the traffic of the OD pair. To address the non-Euclidean nature of the data, particularly the spatio relationships within each OD pair, we employ Graph Embedding through Graph convolutional neural network (GCN), which has been widely used in many works [23, 24] . This approach allows the model to effectively handle non-Euclidean data, enhancing its capacity to capture and utilize the structural information present in the tensor X ∈ R I×J×T .In Satformer, each Spatio-Temporal module contains a GCN model. A GCN model contains two layers of convolutional layer, the feature propagation rule can be stated as follows:EQUATIONEQUATIONwhere, H (l) signifies the node embedding matrix for layer l, A represents the adjacency matrix and I represents the self-connections matrix of A. Ã ∈ R I×J represents the adjacency matrix with self-connections. D ∈ R I×I denotes the degree matrix, which is a diagonal matrix with each element on the diagonal representing the sum of the corresponding row in Ã. The weight matrix for layer l is denoted as W (l) ∈ R I×M , and H (l+1) ∈ R I×J×M represents the node embedding matrix for layer l + 1. W (0) ∈ R K×L denotes the weight matrix from the input layer to the hidden layer, and W (1) ∈ R K×L denotes the weight matrix from the hidden layer to the output layer. Both σ(•) and ReLU are activation functions employed in the model. Z ∈ R I×M ×K represents the output embedding tensor.Satformer Block: As shown in Fig. 1 (b), in each Satformer block, we use a layer normalization at the beginning to normalize the input embedding tensor. We then apply an ASSIT mechanism and a 2-layer MLP module for sparse spatio-temporal feature modeling and per-location embedding, respectively.In the domain of communication network tensor completion, the spatio-temporal relationships among traffic data are complex, and it is necessary to model these relationships effectively. Traditional attention mechanisms, with their intensive nature, may encounter challenges related to high computational complexity and difficulties in capturing global relationships in such intricate scenarios. Several works proposed different sparse attention mechanism to mitigate such issue either relay on static patterns or skip computations in specific regions. As shown in Fig. 1 (e), in this work, we explore an adaptive, sparse spatio-temproal mechanism. Detailed descriptions are as follows:Given an input embedding feature tensor Z ∈ R I×M ×K . First we divide the tensor slice into several local regionsZ div , each of which has a size of D × D. In our module, Q, K and V respectively represent query, key, and value, which are used to calculate attention weights and generate the final output [25] . Then we calculate Q, K and V tensor with linear projections for each region:EQUATIONwhere W q , W k and W v are projection weights for query Q, key K and value V respectively. We then consider introducing a local attention mechanism when calculating the attention score α to make the model pay more attention to each local region in the input tensor. This improvement is designed to sharpen the model's attention specifically on local regions within the input sequence. The goal is to augment the expressiveness and robustness of the model by enabling it to capture and leverage more nuanced details and patterns present in localized segments of the input data. The implementation involves incorporating a position-related weight when calculating attention scores. The local attention in each region is operationalized through the use of a two-dimensional mask matrix Ψ ∈ R D×D , wherein, elements inside a defined center window H are retained, while elements in other positions are set to zero. The size of the center window H is a hyperparameter of the model, and its optimal value is determined through experiments on different datasets. The calculation of attention score α i for each region can be denoted as follows:EQUATION)EQUATIONwhere is an element-wise product and C is scaling factor. And the final output α t are computed as:EQUATIONTo regulate the sparsity of the attention scores and channel the model's focus onto specific portions of the input, an adaptive sparse regularization term is introduced. This involves applying L1 regularization to each element of the attention score matrix. The utilization of ReLU operations ensures that the attention scores remain non-negative. Thus the sparse mask can be denoted as: ReLU(1 -W r α i ).Finally, we apply the weighted Value to the attention score, while introducing additional learnable parameters to allow the model to adaptively learn the weighted sum of each position, resulting in the final output α t , as shown in Eq. 9. Where, W r is the weighted matrix of L1 regularization, W s is the scaling matrix, both W r and W s are trainable parameters. The conventional information transfer between the encoder and decoder typically relies on the output of the last layer of encoder. However, this approach may fall short in adequately conveying global context information, particularly when dealing with input tensors spanning a large number of time slices. The accumulation of errors over time can become a challenge. Consequently, it is necessary to add a module between encoder and decoder to effectively transfer the information. Satformer incorporates a self-attention-based transfer module between the encoder and the decoder. This module leverages Self-Attention, enabling the seamless transfer of globally contextual information learned in the encoder to the decoder. This augmentation empowers the decoder to more comprehensively consider information from the entire input sequence when generating output for each time slice, thus enhancing the estimation accuracy of missing values. Moreover, the transfer module enables the model to integrate spatio-temporal information in a more fine-grained manner, improving its adaptability to patterns across different temporal and spatio scales. The mathematical description of the Transfer Module is as follows:Suppose the encoder outputs an eigenvector e t for each time step t input x t , where t = 1, 2, . . . , T , then the output sequence of the encoder is E = e 1 , e 2 , . . . , e T . The goal of Transfer Module is to convert the output of the encoder E to a new set of feature vector D = d 1 , d 2 , . . . , d T , where each d t is a feature enhanced representation corresponding to time step t. This process is achieved through the following self-attention mechanisms:• Calculate Query, Key, and Value: The query vector Q = EW Q represents the query of future time points against past time points. The key vector K = EW K represents the encoding of a past point in time. The value vector V = EW V represents the specific characteristics of past time points. W Q , W K , and W V are learnable weight matrices. • Calculate attention weight: Calculate the attention weight α t,t-1 of each time step t and consider the effect that past attention scores exert on the present. C t is scaling factor at time step t, i belongs to 1 to t -1, p and q are parameters to control the effort of past time attention scores.EQUATION• Generate transformation feature vector: Based on weights, attention to each time step t to generate a new feature vector d t .EQUATIONThis process enables the Transfer Module to accurately measure the relationship between each future time point and all past time points, and to generate a new set of features that represent valuable information for future predictions.During the training stage, the primary objective is to minimize the discrepancy between the actual and predicted traffic data. To achieve this, the loss function employed by Satformer is the mean square error (MSE), as expressed in Eq. 12. Additionally, to curtail the growth of model weights and mitigate the risk of overfitting, a penalty term is incorporated into the loss function.EQUATIONwhere Ā denotes the set of observed traffic data, χ ijk and χijk are the truth and estimated traffic data respectively, θ represents all trainable parameters in Satformer, λ is weight decay coefficient. 
experimental settings: 
 Datasets. To assess the performance of Satformer, we employ it on three real-world satellite networks: Iridium, Telesat, and Starlink, thereby evaluating its capabilities across varying network scales: smallscale, medium-scale, and large-scale environments. Given the ongoing construction and utilization of many satellite networks, acquiring actual traffic data proves to be challenging. Thus, we generate corresponding traffic datasets using real satellite parameters and ground station coordinates. Similar methods have been used in many previous studies, and the specific details of this process are explained in the Appendix A. The traffic data collection interval was 1 second for all three datasets.• Iridium [26] : The Iridium constellation comprising a total of 66 satellites uniformly distributed across 6 orbital planes. For our experimentation, we focus on the initial six periods, encompassing 36,000 time slices. • Telesat [27] : It collects traffic data from the Telesat constellation which has a total of 298 satellites distributed in 26 orbital planes. We select the first five periods about 31500 time slots in our experiment.• Starlink [28] : The traffic data recording originates from the Starlink constellation, comprising 1584 Low Earth Orbit (LEO) satellites evenly dispersed across 72 orbital planes. The first six periods about 32400 time intervals in our experiment.For all three datasets, we divided the original dataset into a training set and a test set in an 8:2 ratio using the time slice partitioning method. We then used the training set for model training and the validation set for model validation and tuning. Subsequently, we constructed the test set by randomly masking portions of the training and validation sets that were not used for training. This approach ensures that the model is trained and validated on distinct segments of the data, which can help prevent overfitting and improve the model's ability to generalize to new, unseen data.Baselines. For comparative analysis against our Satformer model, we select the following baseline models: three mathematical tensor completion models, namely HaLRTC, LATC and LETC, and four state-of-the-art neural network-based tensor completion models, CoSTCo, DAIN, SPIN and STCAGCN.• HaLRTC [29] : A prototypical high-accuracy low-rank tensor completion algorithm utilizes the Alternating Direction Method of Multipliers (ADMMs) to attain precise outcomes, effectively managing dependencies among various constraints.• LATC [30] : It introduces a novel regularization term, integrating temporal variation, into a third-order tensor completion model. • LETC [31] : a Laplacian enhanced low-rank tensor completion framework for large-scale traffic speed kriging. • CoSTCo [20] : An innovative Convolutional Neural Network (CNN)-based model developed for tensor completion to overcome the limitations associated with traditional low-rank tensor factorization approaches.• CDSA [32] : A novel cross-dimensional self-attention approach for imputing missing values in multivariate, geo-tagged time series data.• DAIN [33] : This method explicitly crafted to enhance the accuracy of neural tensor completion methods when predicting missing values within sparse, multi-dimensional datasets.• SPIN [34] : An attention-based architecture using spatiotemporal graphs and autoregressive models for effectively reconstructing missing data in sparse, multivariate time series. • SAITS [35] : a self-attention-based method for multivariate time series imputation that uses joint-optimization and diagonally-masked self-attention blocks.• STCAGCN [36] : A graph-based deep learning method for traffic volume estimation by utilizing a graph attention-based speed pattern-adaptive adjacency matrix and a customized temporal attention mechanism.Evaluation Metrics. Two widely employed metrics are applied to evaluate the estimation performance of Satformer. The calculation equations for these metrics are presented as follows:• Normalized Mean Absolute Error (NMAE):EQUATION• Normalized Root Mean Squared Error (NRMSE):EQUATION)where χ ijk and χijk represent the truth value and estimated value, Ā denotes the set of unobserved traffic data. For both two metrics, the smaller they get to 0, the better the estimation performance of the model.Compare Satformer with mathematical baselines. 38 .66% and 32.32%, respectively. These results highlight Satformer's effectiveness in handling large-scale datasets, suggesting potential deployment in real-world satellite networks. The limitations of CoSTCo are evident due to its exclusive reliance on two-dimensional convolution for spatial feature extraction without explicitly modeling temporal features. DAIN falls short by not explicitly modeling interactions between entities, which limits information utilization, despite its combination of information for data augmentation. SPIN's ability to handle sparsity or irregularly sampled data might be limited, which could affect the accuracy of traffic estimation in satellite networks where data is often incomplete. STCAGCN captures time-asynchronous correlations may not fully account for the complex temporal dynamics in satellite network, leading to less accurate estimations. The architecture of STCAGCN cannot ensure the information learned at earlier stages is preserved and utilized in later stages. Although CDSA also utilizes the self-attention mechanism, its dimension-wise processing may limit its ability to capture complex interactions. RNN-based models of SAITS are generally inferior to Transformer architectures in terms of handling long-distance dependencies and efficiency. In contrast, Satformer excels by explicitly incorporating both spatial and temporal features within each module. The graph embedding captures nonlinear information, the Satformer module integrates the ASSIT, and the transfer module seamlessly transmits global contextual information. This comprehensive design enables Satformer to deliver exceptional performance in inter-satellite traffic data estimation, effectively addressing the challenges of large-scale, sparsely populated datasets. 
conclusion and discussion: 
 This paper proposes Satformer, a novel traffic data estimation algorithm for large-scale satellite networks, aiming at fast and accurate estimating global traffic matrix from partial sampling in a cost-effective manner. Motivated by this, we design a region-aware sparse spatio-temporal attention mechanism to concentrate on specific local regions of the input tensor, where the input tensor is embedded in a graph convolutional neural network. Thus, spatio-temporal features from the traffic matrix are effectively extracted with computational efficiency and robustness.Extensive experiments with datasets of varying scales-small, medium, and large have shown that Satformer has significant advantages on both accuracy and efficiency for traffic estimation compared with baselines, particularly in larger networks. Moreover, we analyze the robustness of Satformer under different conditions and further verify the role of each module through ablation studies. The results demonstrate the potential of Satformer for deployment in actual systems.Despite Satformer is effective adopted for traffic estimation, deep learning models for traffic estimation remain mostly black boxes. It is quite important to understand the reasons behind inferences in the satellite networking domain. In addition, although Satformer is cost-effective, it is necessary to further reduce its computational complexity, considering the limited computational resources of existing satellites.Future works should prioritize enhancing computational efficiency. It is also important to explore interpretability and decision basis of our deep learning model for traffic estimation. For example, explanation techniques, such as a local interpretable model-agnostic explanation (LIME) [37] , are able to make a visual analysis of the model and analyze the internal working mechanism from specific examples. Additional explanatory tools, such as feature importance analysis, will help users in understanding the model's workings. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The existing assets used in this paper are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To the best of our knowledge, there is no publicly available inter-satellite traffic data or literature describing the traffic distribution for any existing satellite networks. This includes long-established networks like Iridium, as well as those currently under development, such as Starlink. The characteristics of traffic load borne by satellite networks are intricate. Satellites predominantly communicate with terrestrial terminals via ground stations. Spatially, the distribution of ground stations is uneven due to factors such as topography, economic considerations, and geopolitical influences. In regions with extreme environmental conditions or economic underdevelopment, such as oceans, deserts, and polar areas, the received traffic is significantly lower compared to more favorable environments, contributing to an uneven spatial distribution of traffic in the satellite network. Furthermore, the global distribution of earth stations spans various time zones, resulting in non-stationary traffic generation at different times. This temporal variability leads to significant traffic variations among stations. Additionally, to ensure link quality between satellites and ground stations and to mitigate the impact of frequent satellite handoffs, ground stations must consider multiple factors, including elevation angle, service time, and signal strength, when selecting communication satellites. This complexity adds to the challenges associated with managing traffic in satellite networks. temporal dynamics. The temporal variations in traffic load within non-geostationary orbit satellite networks predominantly arise from two factors: the diurnal fluctuations induced by regional local times and the geographic variations in daily traffic patterns influenced by global time zones. In the foreseeable future, the satellite optical network is expected to handle a traffic load comparable to that of the ground network, either matching, proportionally scaling, or exhibiting similar patterns. To ensure accuracy and effectiveness in generating traffic scenarios, this paper employs the four-month average daily traffic change trend from the GEANT network to characterize daily traffic variations. Fig. 4b depicts the normalized cumulative traffic load over a 24-hour period, with the highest peak value normalized to 1. Notably, the flow intensity peaks around 12 noon, gradually diminishes, and then experiences a subsequent rise around 5 AM the following day. In addressing geographical variations, for the sake of model simplicity, the local time of a ground station within its respective time zone is incremented by one hour for every 15 degrees of longitude eastward from Greenwich Mean Time. This adjustment is contingent on the specific time zone associated with each ground station.satellite-ground access. The capability of a ground station to establish satellite-ground communication links with multiple satellites within a given time window is influenced by factors such as satellite density and the coverage area of an individual satellite. Different satellite-ground access methods introduce varying effects on the characteristics of traffic load. Utilizing visibility analysis outcomes obtained through the Satellite Tool Kit (STK) and considering conditions for establishing satellite-ground links, the service time offered by all satellites visible to a ground station in each time window is computed. The satellite offering the longest service time is then selected for access, allowing for the flexibility to choose alternative or custom-designed satellite-ground access methods as needed. The ground station selects the destination node, following a uniform distribution. The traffic density at the ground station is jointly determined by the local time within the time zone and the spatial distribution of ground stations across each time zone.Assuming the network time is based on GMT (Greenwich Mean Time) and considering the generation interval ∆t(s) of the traffic matrix, the computation process for the traffic matrix sequence {F t ter |t ∈ N * } between ground stations is as follows: For any time t, the total traffic D t sent by all ground stations in the whole network is calculated as Eq.15.EQUATION)where, of f erload is the network traffic load, B is the maximum bandwidth of the inter-satellite link, and n ter is the total number of ground stations.The latitude and longitude coordinates of any ground station i is (x i , y i ), then the local time t m (h) of the time zone m of any ground station i can be calculated by Eq.16:EQUATIONCombined with the normalized cumulative traffic load in 24 hours in Fig. 4b , the traffic intensity weight of the time zone m where the ground station i is located can be calculated:w t m = w tm w total , 0 ≤ m ≤ 23&m ∈ N * (17)where, w total is the total traffic load and w tm is the load at time t corresponding to time zone m.The traffic (f t m ) z that a ground station i located in time zone m needs to send at time t can be calculated by Eq.18 as follows.EQUATION)where, n m denotes the number of ground stations in time m, and23 m=0 n m = n ter .Since the destination node is selected by the ground station according to uniform distribution, the traffic sent from the ground station i to the ground station j at time t can be calculated by Eq.19:EQUATIONwhere, F (i, j) denotes the traffic from the source ground station i to the destination ground station j, and U (0.1, 1) represents the uniform distribution from 0.1 to 1.By traversing all the ground stations in the network at each time interval, the traffic matrix sequence {F t ter |t ∈ N * } between the ground stations can be calculated. According to the inter-satellite visibility analysis results provided by STK, as shown in Fig. 5 , combined with the satellite-to-ground access method described previously, the inter-satellite traffic matrix sequence {F t sat t ∈ N * } can be obtained.Lemma 1. Convergence of GCN Layer-wise Propagation.Statement: For a multi-layer GCN with layer-wise propagation defined asH (l+1) = σ(D -1 2 ÃD -1 2 H (l) W (l)), the node embeddings H (l) converge as l increases, under mild conditions on the activation function σ and weight matrices W (l) . In our work, Lemma 1 supports the use of two-layer convolutional layers in GCN to effectively propagate features, thereby validating the capability of GCN to capture and utilize the structural information present in the input tensor.Proof. Consider a GCN with L layers. The layer-wise propagation is given by:H (l+1) = σ(D -1 2 ÃD -1 2 H (l) W (l) )where H (l) ∈ R N ×d l is the node feature matrix at layer l, D is the degree matrix, Ã = A + I is the adjacency matrix with self-loops, and W (l) ∈ R d l ×d l+1 is the weight matrix.Assume the activation function σ is Lipschitz continuous with Lipschitz constant L σ , i.e., for all x, y ∈ R,|σ(x) -σ(y)| ≤ L σ |x -y|Assume the weight matrices W (l) are bounded, i.e., there exists a constant M such that W (l) ≤ M for all Define the propagation operator Φ : R N ×d l → R N ×d l+1 as:Φ(H) = σ(D -1 2 ÃD -1 2 HW )To show Φ is a contraction, consider two node feature matrices H 1 , H 2 ∈ R N ×d l . We need to show that:Φ(H 1 ) -Φ(H 2 ) ≤ k H 1 -H 2 for some 0 ≤ k < 1.Compute the difference:Φ(H 1 ) -Φ(H 2 ) = σ(D -1 2 ÃD -1 2 H 1 W ) -σ(D -1 2 ÃD -1 2 H 2 W )Using the Lipschitz continuity of σ:Φ(H 1 ) -Φ(H 2 ) ≤ L σ D -1 2 ÃD -1 2 (H 1 -H )WApply the sub-multiplicative property of norms:Φ(H 1 ) -Φ(H 2 ) ≤ L σ D -1 2 Ã D -1 2 H 1 -H 2 WSince D and Ã are derived from the graph structure and W is bounded by M :D -1 2 Ã D -1 2≤ λ max where λ max is the largest eigenvalue of the normalized adjacency matrix.Combining these, we get:Φ(H 1 ) -Φ(H 2 ) ≤ L σ λ max M H 1 -H 2For Φ to be a contraction, we need:L σ λ max M < 1 If L σ λ max M < 1,then Φ is a contraction mapping.By the Banach fixed-point theorem, every contraction mapping on a complete metric space has a unique fixed point.Therefore, the node embeddings H (l) will converge to a fixed point as l increases.Statement: The attention mechanism with a mask matrix focusing on the center region of the input data remains stable and does not degrade performance, provided the mask matrix is appropriately designed. Lemma 2 supports the introduction of a local attention mechanism in the adaptive sparse spatio-temporal attention mechanism, enabling the model to better capture and utilize details and patterns in localized regions of the input tensor.Proof. The mask matrix M is designed such that M ij = 0 for elements outside the central region and M ij = 1 within the central region.This implies that only the attention scores corresponding to the central region are retained, effectively reducing the complexity of the attention mechanism by filtering out less relevant data.Mathematically, M acts as a sparsity-inducing regularizer:A (X) = softmax(M (XW 1 W 2 X ))By focusing on the central region, M ensures that the attention mechanism does not overfit to peripheral noise, enhancing generalization.The central region often contains the most informative parts of the data, as observed in empirical studies (e.g., Fig. 9(a) ).By applying M , the attention mechanism A (X) prioritizes the computation of attention scores within this region, thus capturing critical local patterns:A (X) ij = exp((M (XW 1 W 2 X )) ij ) k exp((M (XW 1 W 2 X )) ik )This prioritization ensures that the most relevant features are emphasized, leading to improved prediction accuracy.To show that the deviation introduced by M is bounded, consider the difference between the original and masked attention mechanisms:∆A = A(X) -A (X)Since M ij ∈ {0, 1}, it acts as a binary mask, thus the modification is limited to setting some attention scores to zero while retaining the rest:∆A F = softmax(XW 1 W 2 X ) -softmax(M (XW 1 W 2 X )) FGiven that softmax is a Lipschitz continuous function with constant 1, the Frobenius norm ∆A F is bounded by the norm of the difference in the inputs:∆A F ≤ (1 -M ) (XW 1 W 2 X ) FSince M zeros out peripheral entries, the difference is confined to the less relevant regions, ensuring that the overall deviation remains controlled. Satformer and the neural network-based baselines are implemented in PyTorch, while the mathematical baselines are implemented using Numpy. We evaluated Satformer against the baselines on a server equipped with an NVIDIA RTX 2080Ti GPU, 128 GB DDR4 RAM, and an Intel Xeon Silver 4208 CPU, running the Ubuntu 18.04 operating system. All models are trained for a range of 50 to 300 epochs with the first 5 epochs designated for warmup, and early stopping is adopted during the training process. The Adam optimizer [38] is used to optimize our model. A grid search strategy is applied to determine the best learning rate, epochs, and batch size. Based on the results of the grid search strategy, the optimal hyperparameters for the Satformer model and the neural network baselines are presented in Table 2 , with the best weight decay determined to be 0.00001. We run each model 10 times with the same parameters and record the mean results in Table 1 .Impact of module number. The module number indicates how many spatio-temporal modules should be contained in Satformer. It significantly impacts the computational efficiency and accuracy of Satformer. We adjust the number of modules from 1 to 20 and record the NMAE and NRMSE for each dataset, set the sample ratio to 2% and maintain the other hyperparameters constant. As shown in Fig. 6 , we observe that the error of Satformer continually decreases as the number of modules increases, up to a point (10 for all three datasets), after which it begins to increase. The reason is that increasing the number of modules enhances Satformer's ability to extract spatio-temporal features, Consequently, we select a module number of 10 for all three datasets.Impact of centre window size. The centre window size indicates how many elements should be retained in a mask matrix. It also significantly influences the accuracy of Satformer. It is another crucial hyperparameter of Satformer. The results in Fig. 6 show that, for Iridium and Telesat, the estimation performance of Satformer continues to improve until it starts to decrease at a certain point (16 for Iridium and 32 for Telesat). Conversely, for Starlink, the estimation performance continues to increase. It can be observed that the window size is proportional to the dataset's scale. For small and medium-scale datasets, the model may extract useful information with a small center window size, thereby improving computational efficiency. However, for large-scale datasets, a larger center window size enables the model to extract more features while maintaining high accuracy. Accordingly, we set the center window size to 16 for Iridium, 32 for Telesat, and 64 for Starlink. To thoroughly evaluate the robustness and performance of Satformer, we conducted a comprehensive comparison of its Normalized Absolute Error (NMAE) and Normalized Root Mean Squared Error (NRMSE) metrics against all six baseline models across three diverse datasets. These datasets encompass varying numbers of time slices, ranging from 100 to 1500, providing a broad spectrum of temporal scales for analysis. As the number of time slices of the output increases, the accuracy will inevitably decrease, mainly because the complexity of the data increases exponentially with each additional dimension, making it harder to model and predict accurately [39] . In addition, In the process of tensor completion, especially over many time slices, there can be a loss of information that is critical for making accurate predictions. Each step in the prediction process may introduce slight inaccuracies, which accumulate over time [40] . The results from our experiments unequivocally demonstrate the superior and consistent performance of Satformer across all datasets, irrespective of the number of time slices involved. Notably, as the number of input time slices increases, Satformer consistently outperforms other models in terms of reliability, maintaining consistently low NMAE and NRMSE indicators. Importantly, even with the escalation of the temporal dimension and the expansion of dataset sizes, Satformer exhibits remarkable stability in its results. These findings underscore the robustness and scalability of Satformer, rendering it not only valuable in theoretical contexts but also highly applicable in real-world engineering scenarios. To assess the impact of our graph embedding module (GE), ASSIT module, and transfer module on Satformer's performance, we conducted ablation experiments, systematically removing these key components to create various model variants. Specifically, we evaluated the following variants: 1) Satformer-GE omits the graph embedding module; 2) Satformer-AT removes the adaptive sparse spatio-temporal attention mechanism module; 3) Satformer-TM does not incorporate the transfer module.Experimental results consistently demonstrate the superior performance of the complete model as compared to the variant models from which crucial components have been removed, across three datasets that have varying sampling rates. Notably, the absence of the graph embedding module hinders the effective capture of topological structure information, underscoring its pivotal role in augmenting overall model performance. The significance of the ASSIT module becomes evident in addressing sparsity issues within spatio-temporal data; when it is absent, the model experiences a noticeable decline in performance, validating its effectiveness in uncovering temporal and spatial dependencies. Furthermore, the transfer module plays a crucial role in facilitating the conversion of information between different feature spaces, thereby enhancing the model's capability in feature representation. The adaptive sparse spatio-temporal attention mechanism allows Satformer to focus on specific local regions of the input tensor, which is particularly beneficial for handling the large-scale and highly dynamic nature of satellite networks. We visualize the attention mechanism in Satformer to verify whether the functionality is achieved, as shown in Fig. 9 .Fig. 9 (a) presents the initial traffic matrices for Iridium, Telesat, and Starlink satellite constellations. Each matrix's dimension is determined by the number of satellite nodes, denoted as N, with each point representing the volume of traffic between respective node pairs. Fig. 9 (b) illustrates the traffic matrices after applying a 10% sample rate to the three datasets, demonstrating a clear reduction in data density. Fig. 9 (c) displays the attention map, which are derived from the Satformer module using the attention scores, α s , as defined by Eq.9. The attention map's dimensions are R (D × D×heads), which is then averaged across the head dimension and scaled to N × N in the D × D dimensions for visualization purposes. This scaling process is designed to be intuitive without altering the inherent relationships within the tensors. Finally, Fig. 9 (d) represents the estimated traffic matrices, which are the reconstructed traffic data from the sampled data.Upon examining the initial traffic matrix for Iridium constellation, a notable volume of traffic is evident within the red-boxed area. After sampling, only sparse data points remain, yet the attention map successfully captures the significance of this high-traffic area, as indicated by the high attention scores. The estimated traffic matrix aligns well with the actual traffic in this region. Similar observations can be made for the Telesat and Starlink datasets, where the attention mechanism effectively identifies and emphasizes critical traffic areas, leading to accurate estimations within the reconstructed traffic matrices.Although Satformer was developed to address traffic data estimation in satellite networks, the core strengths of its methodology endow it with the potential to be applied to other tensor completion Figure 9 : Visualization of adaptive sparse spatio-temporal attention mechanism tasks requiring the handling of large-scale, sparse, and complex spatio-temporal characteristics. This includes, but is not limited to, social network analysis, environmental monitoring, bioinformatics, and other domains that necessitate the reconstruction and analysis of multidimensional data. We use the Foursquare [33] and PeMS-Bay [41] tensor dataset to evaluate Satformer's performance on general tensor completion tasks. The Foursquare dataset is a point-of-interest tensor defined by user, location, and timestamp. Utilizing this dataset allows researchers and developers to gain insights into the dynamics of geographic social networks and to develop innovative applications and services based on these insights. The PeMS-Bay data collect this traffic volume data from 325 loop sensors in the San Francisco bay area, ranging from January to March 2018 with a 5-min time interval.The test results on two general datasets demonstrate the applicability of the Satformer method to other general tensor completion tasks. These positive results indicate that Satformer can serve as a powerful tool across various fields that involve complex spatio-temporal data, including social network analysis, traffic flow environmental monitoring, and others. Naturally, additional adjustments and optimizations may be necessary for different application scenarios to achieve optimal performance.Satformer is primarily used in emerging large-scale satellite communication networks. Currently, optimizing satellite network traffic engineering [6] , [10] or topology engineering [11] relies on real-time collection of global traffic data. However, due to the limitations of satellite networks, real-time collection of comprehensive traffic status information incurs significant costs and overhead, making it nearly impossible to achieve, which further hinders the deployment of these solutions. Satformer addresses this issue by recovering global traffic data from a low sample rate (2%), which significantly reducing costs and overhead, thereby enabling the deployment of these solutions in real satellite networks. Additionally, Satformer helps improve network administrators' awareness of network states, optimizing network operations and maintenance. Beyond satellite communication networks, Satformer has the potential to be extended to other scenarios, such as transportation networks, the Internet of Things (IoT), and image processing. We have validated this in Appendix I. However, Satformer also presents interpretability risks. We recommend that researchers focus on interpretability when using Satformer, enhancing the transparency of application decisions related to traffic scheduling, routing, and congestion control.Answer: [Yes] Justification: We have provided the code and data of this paper in supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed both potential societal impacts in Appendix J. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper received 5/7/7/4 ratings. Reviewers acknowledge the clear presentation and thorough experimental evaluations. Reviewer 4's major concern came from the novelty of the proposed method. The AC recognize the application in satellite networks analysis is still valuable to the community, thus recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents to make use of the transformer for spatio-temporal data imputation, and the application is for satellite networks.
strength_weakness: strength_weakness: -  The spatial characteristics of the data in satellite networks are not presented. Many other spatial-temporal data are also high-dimensional and non-linear. Many existing approaches have been developed for those data, such as SPIN, CDSA, SAITS. The authors only compared with SPIN.
- The authors explained that their model outperforms SPIN because “SPIN’s ability to handle sparsity or irregularly sampled data might be limited”. However, SPIN is designed for sparse data, and hence the intuition on why the proposed method is better is not provided.
- Applying L1 to the attention score matrix cannot make the output to be sparse. 
- There are quite a few approaches to obtain graph embedding (or just a node embedding), why the authors choose this specific GCN for embedding?
questions: questions: Please see weaknesses
limitations: limitations: Please see weaknesses
paper_summary: paper_summary: This paper introduces novel network designed for satellite traffic data estimation. Proposed approach includes several novelties and the experimental results demonstrate that the propsoed methods outperformed several alternative mehtematical and neural baseline methods.
strength_weakness: strength_weakness: No major weakness but on the write up and method description can be improved:
questions: questions: * The significance of solving this traffic prediction problem should be made clearer in the introduction. 
* You need a figure for the transformer module (Section 3.4). 
*  Please provide references for Eq. (5) and (4) or describe the rational behind them. 
* In Page 5, why do you need the mask matrix? Why the focus on the center region? How is this not impacting performance?
* In Page 5, "the implementation involves incorporating a position-related weight when calculating attention scores". What is the mathematical symbol for this weight matrix in the equations?
* Page 6, C_{t} is not in equation (10) ?
* Add a figure dsecribing the transfer module.
limitations: limitations: Limitations were discussed.
paper_summary: paper_summary: The paper introduces Satformer, a novel method for accurate and robust traffic data estimation in satellite networks. It addresses the challenges of large-scale and dynamic nature of satellite networks by proposing an adaptive sparse spatio-temporal attention mechanism that focuses on specific local regions of the input tensor, enhancing the model's sensitivity to details and patterns. Satformer incorporates a graph convolutional network for processing non-Euclidean data and a transfer module for disseminating global context information, demonstrating superior performance over mathematical and neural baseline methods, especially for larger networks. Extensive experiments on simulated satellite network datasets, including Iridium, Telesat, and Starlink, show Satformer's significant improvements in reducing errors and maintaining robustness. The paper concludes that Satformer shows promise for deployment in actual systems and potential application in other tensor completion tasks requiring handling of complex spatio-temporal data.
strength_weakness: strength_weakness: 1. The paper does not present any theoretical results or proofs to support its claims, focusing instead on empirical results.
2. The paper primarily focuses on the application of Satformer in satellite networks. While it mentions potential applications in other areas, it does not provide extensive testing or results in those domains, which may limit the perceived generalizability of the model.
3. The paper may rely on certain assumptions that are not fully discussed or tested under various conditions. For example, the model's performance might vary under different network configurations or traffic patterns that were not part of the evaluation.
questions: questions: 1. What is the unique challenges of traffic data estimation for satellite networks, compared with other network types?
2. The research motivation is not convincing when there is few relevant literatures.
limitations: limitations: 1. Only simulated datasets are used and it is difficult to collect real-world datasets to validate the proposed method.
paper_summary: paper_summary: This paper proposed a satellite network traffic data estimation method which is called Satfomer.  The proposed method uses the adaptive sparse spatio-temporal attention mechanism to capture nonlinear spatio-temporal relationships. The proposed method is assessed on different satellite network datasets and compared to different existing methods.
strength_weakness: strength_weakness: 1. The overall idea of the proposed is based on the attention mechanism, which is not novel in the field of spatio-temporal forecasting.
2. The motivation of this work is unclear, what is the main difference between the satellite network traffic data and other spatio-temporal data, e.g., traffic flow/speed data?
3. NMAE and NRMSE are used as the metrics to gauge the prediction performance of the methods in this work, what about MSE, MAE, and MAPE? 
4. Some important baselines are missing, e.g., GraphWavenet, AGCRN, and GMAN, etc.
questions: questions: See Weaknesses.
limitations: limitations: The novelty of the proposed method is limited.

Comments: title: Reply to Reviewer 78MM
Reply to Reviewer LZHh
Reply to Reviewer 6cUM's Comments
reply to rebuttal
Reply to Reviewer RAyE
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
Continue
comment: Thank you for your previous suggestions and positive reply. We are pleased to know that our responses have been acknowledged by you.
Thank you very much for your positive comments and score improvement on our paper. We are pleased to know that we responded adequately to your questions.
The authors have addressed my previous concerns.
The authors have addressed all my questions, and they explicitly compare with imputation methods in traffic data, and hence I raised my score.
Thank you for your reply. While we acknowledge that adaptive learning of spatio-temporal correlations has been applied to time series modeling in other fields, this does not mean that the novelty of our work is limited. Indeed, our main contributions focused on cost-effective traffic measurement over satellite networks.

Different from terrestrial networks, satellite networks have unique challenges, such as spatial and temporal dynamics, sparsity and incompleteness, and limited computational resources. Our model, Satformer, effectively addresses these challenges through the ASSIT mechanism. This mechanism is specifically designed to focus on key data regions, enhancing the model's ability to capture complex, nonlinear spatiotemporal patterns essential for large-scale, highly dynamic environments like satellite networks. To the best of our knowledge, these challenges have not been sufficiently explored in existing literature, including the references you suggested.

Satformer is theoretically convergent and stable, and extensive experiments demonstrate its accuracy and robustness with low overhead. This represents a significant contribution to the field, addressing the high overhead and costs associated with measuring global traffic data and meeting the urgent needs of satellite network management, operation, and maintenance. Unlike existing models such as AGCRN, Satformer specifically caters to the intricacies of satellite networks. In addition, the other three reviewers also acknowledged our novelty.

The advanced method you recommended, Pristi, requires multiple inputs, including observed spatio-temporal data, geographic information, interpolation data, and noise information, which limits its application on satellite networks. In contrast, our method, Satformer, only requires observed spatio-temporal data as input, and has a lower time complexity of $O(NK)$ compared to Pristi's $O(Nkd)$. 

We initially intended to compare our method with Pristi, but since the author has closed the source code link and there is only one day left for discussion, it is impractical to reproduce Pristi. However, this does not impact the conclusion of our experiment, as our Satformer has been extensively compared with SOTA methods from 2021, 2022, and 2023 across multiple datasets.
Thanks the authors for the comments. I would say the the main contributions of this paper are application-focused (satellite networks). Adaptive learning spatiotemporal correlations is not so novel in this field of  time series modelling, e.g., AGCRN and many other existing works (maybe with slight differences). Thus, the novelty of the paper is limited.  


Moreover, in terms of spatiotemporal imputation, one could also consider exploring more advanced techniques, for example: 
Liu, Mingzhe, Han Huang, Hao Feng, Leilei Sun, Bowen Du, and Yanjie Fu. "Pristi: A conditional diffusion framework for spatiotemporal imputation." In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 1927-1939. IEEE, 2023.
Thank you for your suggestions, we will include these responses in the camera-ready version if this paper is accepted. Thank you again for your valuable comments.
Thanks for the informative and detailed reply. I suggest including these responses in the paper as appropriate.
## Limitations 1:
_Only simulated datasets are used and it is difficult to collect real-world datasets to validate the proposed method._

## Response L1:

Thank you for your suggestion. As you said, it is difficult to collect real datasets, on the one hand, satellite networks are still in the early stages of construction, on the other hand, they are limited by privacy issues, proprietary data policies and other factors. However, it is gratified that there are also researchers devoted to the study of satellite network measurement [1][2], and provide some feasible methods and ideas. Based on this, we believe that in the near future, we will certainly be able to obtain the real datasets of satellite network.

In many cases, it is common to use simulated datasets to verify the performance of the solution, such as CV[3], NLP[4], etc. Although our dataset is simulated, we focus on the geographical distribution of the ground stations, the time zone change and the visibility between the satellite and the ground stations when generating this data set, which can maximally reflect the traffic characteristics in the real satellite network and effectively verify the traffic estimation scheme Satformer in this paper. In addition, the simulated datasets also have the benefit of allowing the tuning of various parameters to test the robustness and accuracy of the proposed method under different conditions, which may be constrained in real datasets. 

While we have not yet tested our solutions on real satellite network datasets, we have used the real datasets Foursquare [5] and Pemsbay [6] to evaluate Satformer and baseline models on general tensor-completion tasks. The evaluation results are detailed in Response W2, and it prove that our method Satformer is also effective in real datasets.

The dataset generation method has been made public, and we hope to encourage more researchers to participate in the study.

**References:**

[1] Izhikevich, Liz, et al. "Democratizing LEO Satellite Network Measurement." ACM SIGMETRICS Performance Evaluation Review 52.1 (2024): 15-16.

[2] Pan, Jianping, Jinwei Zhao, and Lin Cai. "Measuring a low-earth-orbit satellite network." 2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC). IEEE, 2023.

[3] Ge, Yunhao, et al. "BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Long, Lin, et al. "On LLMs-driven synthetic data generation, curation, and evaluation: A survey." arXiv preprint arXiv:2406.15126 (2024).

[5] Oh, Sejoon, et al. “Influence-Guided Data Augmentation for Neural Tensor Completion.” Proceedings of the 30th ACM International Conference on Information &amp; Knowledge Management, 2021.

[6] Lee, Hyunwook, et al. "An empirical experiment on deep learning models for predicting traffic data." 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 2021.
##  Question Q5:

*In Page 5, "the implementation involves incorporating a position-related weight when calculating attention scores". What is the mathematical symbol for this weight matrix in the equations?*

## Response Q5:

I apologize for the confusion caused by the misrepresentation. Actually, the weight matrix associated with the position is the 2D mask matrix $\Psi$. It can be understood in terms of weighting, where $\Psi$ will assign a larger weight to the center region and a weight close to 0 to the inattentive region. This 2D mask matrix plays a crucial role in the attention mechanism by ensuring that the model focuses on the most relevant parts of the input data, thereby enhancing its ability to capture significant patterns.

To elaborate further, the mask matrix $\Psi$ is integrated into the calculation of the attention score $\alpha$ to modulate the influence of different regions. This integration helps in refining the attention mechanism by providing positional context, which is essential for accurately capturing the dependencies in the data. By applying $\Psi$, the model effectively differentiates between regions of varying importance, assigning higher weights to the central, more informative areas, and lower weights to the peripheral, less relevant areas.

Additionally, the visualization of the calculated attention score $\alpha_s$ is shown in Figure 8(c). This figure demonstrates how the attention mechanism, influenced by the mask matrix $\Psi$, highlights different regions of the input data. The attention scores are visibly concentrated on the center regions, confirming the role of $\Psi$ in guiding the model's focus towards the most pertinent sections of the data.

## Question Q6:

_Page 6, $C_{t}$ is not in equation (10) ?_
## Response Q6:

I apologize that the presentation of that formula in the paper confused you. We double-checked the submitted paper  and confirmed that $C_{t}$ is indeed in the formula and represents the scaling factor. We will further investigate the relevant parts of the document to ensure the accuracy and rigor of the formula.

The equation (10) in the paper reads:
$$
 \alpha_{t,t-1} = \text{softmax}\left(p \sum_{i=1}^{t-1} \frac{Q_i K_i^T}{\sqrt{C_i}} + q \frac{Q_t K_t^T}{\sqrt{C_t}}\right) V_t 
$$
This equation shows the computation of the attention weight $\alpha_{t,t-1}$ for each time step $t$ considering the impact of past time steps. Here, $C_t$ is explicitly included as a scaling factor to normalize the dot product of query and key vectors at time $t$. The symbol $C_i$ for $i$ belonging to $1$ to $t-1$ indicates the scaling factors for the previous time steps. These factors are used to normalize the dot products for each of the past time steps individually, ensuring that the contributions of the past time steps to the attention weight are properly scaled and balanced.

## Question 7:

_Add a figure describing the transfer module._
## Response Q7:

Thank you for your constructive suggestions, a more visual presentation is a real necessity. I will show the figure of transfer module on **Author Rebuttal** Block which is highlighted in orange at the top of the webpage.
Our current experiments have not found that focusing on the center region negatively affects performance. In fact, by ensuring that the attention mechanism prioritizes the most important parts of the data, this typically improves the effectiveness of the model.  The improvement was about 30% relative to using the unimproved attention mechanism and more than 50% relative to not using the attention mechanism. The latter is demonstrated in the results of our ablation experiments in Appendix E. As an added benefit, this strategic sparsity helps to reduce computational complexity (the advantages of our computational performance can be seen in Table (3).

We give a proof of the stability of attention mechanism with masking to support our claim.

**Lemma: Stability of Attention Mechanism with Masking**

**Statement:** The attention mechanism with a mask matrix focusing on the center region of the input data remains stable and does not degrade performance, provided the mask matrix is appropriately designed.

**In our work**: Lemma  supports the introduction of a local attention mechanism in the adaptive sparse spatio-temporal attention mechanism, enabling the model to better capture and utilize details and patterns in localized regions of the input tensor.

**Proof of Lemma**

The mask matrix $M$ is designed such that $M_{ij} = 0$ for elements outside the central region and $M_{ij} = 1$ within the central region.This implies that only the attention scores corresponding to the central region are retained, effectively reducing the complexity of the attention mechanism by filtering out less relevant data.Mathematically, $M$ acts as a sparsity-inducing regularizer:
$$
A'(X) = \text{softmax}(\Psi \odot (XW_QW_K^\top X^\top))
$$
By focusing on the central region, $M$ ensures that the attention mechanism does not overfit to peripheral noise, enhancing generalization.

The central region often contains the most informative parts of the data, as observed in empirical studies (e.g., Figure 8(a) in the original paper).

By applying $M$, the attention mechanism $A'(X)$ prioritizes the computation of attention scores within this region, thus capturing critical local patterns:

$$A'(X) = \\frac{\\exp((\\Psi \\odot (XW_QW_K^\\top X^\\top))}{\\sum_k \\exp((\\Psi \\odot (XW_QW_K^\\top X^\\top)))}$$

This prioritization ensures that the most relevant features are emphasized, leading to improved prediction accuracy.

To show that the deviation introduced by $M$ is bounded, consider the difference between the original and masked attention mechanisms:

$$
\Delta A = A(X) - A'(X)
$$
Since $M_{ij} \in \{0, 1\}$​, it acts as a binary mask, thus the modification is limited to setting some attention scores to zero while retaining the rest:
$$
\| \Delta A \|_F = \| \text{softmax}(XXW_QW_K^\top X^\top) - \text{softmax}(\Psi \odot (XXW_QW_K^\top X^\top)) \|_F
$$
Given that $\text{softmax}$ is a Lipschitz continuous function with constant $1$, the Frobenius norm $\| \Delta A \|_F$ is bounded by the norm of the difference in the inputs:


$$
\| \Delta A \|_F \leq \| (1-\Psi) \odot (XXW_QW_K^\top X^\top) \|_F
$$
Since $M$ zeros out peripheral entries, the difference is confined to the less relevant regions, ensuring that the overall deviation remains controlled.
## Question 3:

_Please provide references for Eq. (5) and (4) or describe the rational behind them._
## Response Q3: 

Thank you for your constructive questions, and I will then further explain the references and rationale for Eqs. (5) and (4). The original formulas are as follows

**Equation (4):** $\tilde{A} = A + I$

**Equation (5):** $Z = f(X, A) = \sigma(\tilde{D}^{-1/2} \tilde{A} \tilde{D}^{-1/2} \text{ReLU}(\tilde{D}^{-1/2} \tilde{A} \tilde{D}^{-1/2} X W^{(0)}) W^{(1)})$​

The two formulas are based on the theory of graph convolution, and the references and rationale are stated in the following three points.

1. **Graph Convolutional Networks (GCNs):** The rationale for Equations (4) and (5) is rooted in the theory and application of Graph Convolutional Networks (GCNs) by Thomas N. Kipf and Max Welling [1]. GCNs process data structured as graphs, using spectral graph theory to define convolution operations. The process involves normalizing the adjacency matrix $ A $ with the degree matrix $ D $ and adding self-loops (identity matrix $ I $) to preserve the node's own features during convolution. Additionally,  Chung's work on spectral graph theory [2] provides the mathematical framework for understanding the importance of the graph Laplacian in GCNs, which is fundamental to the normalization and augmentation steps described in Equations (4) and (5).
2. **Equation (4):** Defines the augmented adjacency matrix $\tilde{A}$. Adding the identity matrix $I$ to the adjacency matrix $A$ incorporates self-loops, ensuring the node's own features are included in the aggregation process. This step is essential for preserving the node's identity during the convolution operation, as it prevents the node's features from being lost or diluted when averaging with its neighbors. 
3. **Equation (5):** Represents a two-layer graph convolutional network, 
   - **Augmented Adjacency Matrix:** The adjacency matrix $A$ is augmented with self-loops to become $\tilde{A}$. This ensures that each node's own features are considered during the convolution process.
   - **Normalization:** The normalized adjacency matrix $\tilde{D}^{-1/2} \tilde{A} \tilde{D}^{-1/2}$ ensures balanced feature propagation by accounting for the varying degrees of nodes in the graph. This normalization helps to stabilize the learning process and improves the model's performance.
   - **Feature Transformation:** The input feature matrix $X$ is transformed using weight matrices $W^{(0)}$ and $W^{(1)}$ through a ReLU activation function. This transformation allows the model to learn complex representations of the graph's structure and the node features. 

The difference between the two convolutional layers in Equation (5) is the use of different activation layers and input feature matrices. Figure 1(d) in the original paper illustrates this structure. I will consider revising the original article based on the inclusion of references and specific elucidations. Thanks again for the question. It makes a lot of sense!

**References**

[1] Kipf, Thomas N., and Max Welling. "Semi-supervised classification with graph convolutional networks." *arXiv preprint arXiv:1609.02907* (2016).

[2] Chung, Fan R. K. "Spectral Graph Theory." *CBMS Regional Conference Series in Mathematics* 92 (1997).

## Question 4:

_In Page 5, why do you need the mask matrix? Why the focus on the center region? How is this not impacting performance?_
## Response Q4:

**Why is the Mask Matrix Needed?**

Thank you for your question. In summary, we introduced the mask matrix based on two considerations.
(i) Satellite traffic data are highly dynamic and large-scale. In our organization and observation of the dataset, as shown in Fig. 8(a) of this paper, the distribution of the satellite traffic matrix is characterized by high dynamics, and with aggregation.
(ii) Satellite networks have limited computing resources.  Therefore, we tend to reduce the amount of unnecessary computation without compromising performance.

We then tried to focus on the central region of the input data using a mask matrix, and found a significant increase in prediction accuracy while achieving a reduction in computational complexity.

**Why Focus on the Central Region?**

Focusing on the central region is adapted to the distribution of the satellite network traffic matrix, which allows the model to focus computational resources on the most informative parts of the data, thus improving overall performance and efficiency. This approach helps the attention mechanism to capture important local features without being overwhelmed by less relevant peripheral data. 

**Impact on Performance**

A reasonable speculation is that the presence of the mask enhances the model's ability to capture local patterns and relationships, which is crucial for accurate prediction of spatio-temporal data.
**References:**

[1] Wang, Zhaohua, et al. "Examination of WAN traffic characteristics in a large-scale data center network." Proceedings of the 21st ACM Internet Measurement Conference. 2021.

[2] Orlowski, Sebastian, et al. "SNDlib 1.0—Survivable network design library." Networks: An International Journal 55.3 (2010): 276-286.

[3] Xu, Xiongxiao, et al. "Surrogate Modeling for HPC Application Iteration Times Forecasting with Network Features." Proceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation. 2024.

[4] Roy, Arjun, et al. "Inside the social network's (datacenter) network." Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 2015.

[5] Chan, Vincent WS. "Optical satellite network architecture [Invited Tutorial]." Journal of Optical Communications and Networking 16.1 (2024): A53-A67.



## Questions 2:
_The research motivation is not convincing when there is few relevant literatures._

## Response Q2:

Thank you for your question. Network management and operations depend on real-time traffic data. However, current network measurement technologies face significant challenges, including high measurement overhead and difficulty in collecting large-scale traffic data. Traffic estimation methods can provide complete global network traffic by collecting only a small amount of data, thereby reducing collection overhead and improving real-time performance. Although previous research has primarily focused on traffic estimation for terrestrial networks [1-5], there is few relevant literatures on traffic estimation for satellite networks. 

With the rise of satellite networks in recent years, we have noticed that access management, flow control [6], routing [7] and congestion control [8] of satellite networks also require accurate and robust traffic estimation methods. However, satellite networks pose some unique challenges to traffic estimation due to their highly dynamic nature, low link quality and limited computational resources. To fill this gap, we present Satformer, an accurate and robust traffic estimation method for satellite networks. To the best of our knowledge, this work is the first to study satellite network traffic estimation.

**References:**

[1] Xie, Kun, et al. "Neural Network Compression Based on Tensor Ring Decomposition." IEEE Transactions on Neural Networks and Learning Systems (2024).

[2] Qiao, Yan, Zhiming Hu, and Jun Luo. "Efficient traffic matrix estimation for data center networks." 2013 IFIP Networking Conference. IEEE, 2013.

[3] Li, Xiaocan, et al. "A Light-Weight and Robust Tensor Convolutional Autoencoder For Anomaly Detection." IEEE Transactions on Knowledge and Data Engineering (2023).

[4] Qiao, Yan, Kui Wu, and Xinyu Yuan. "AutoTomo: Learning-Based Traffic Estimator Incorporating Network Tomography." IEEE/ACM Transactions on Networking (2024).

[5] Miyata, Takamichi. "Traffic matrix completion by weighted tensor nuclear norm minimization." 2023 IEEE 20th Consumer Communications & Networking Conference (CCNC). IEEE, 2023.

[6] Zhong, Xiaoqing, et al. "Link Topology and Multi-Objective Mission Flow Optimization for Remote Sensing Satellites With Inter-Layer Links and Satellite-Ground Links." IEEE Transactions on Vehicular Technology (2024).

[7] Mao, Bomin, et al. "On an intelligent hierarchical routing strategy for ultra-dense free space optical low earth orbit satellite networks." IEEE Journal on Selected Areas in Communications (2024).

[8] Yang, Wenjun, et al. "Mobility-Aware Congestion Control for Multipath QUIC in Integrated Terrestrial Satellite Networks." IEEE Transactions on Mobile Computing (2024).
## Weakness 3:
_The paper may rely on certain assumptions that are not fully discussed or tested under various conditions. For example, the model's performance might vary under different network configurations or traffic patterns that were not part of the evaluation._
## Response W3:

Thank you for your comment. I'm sorry for the trouble caused by our lack of clarity. An important assumption in this paper is that the orbit height of each node in the satellite network remains unchanged and the bandwidth of each link does not change. This assumption is reasonable. While the orbit height may degrade and require adjustments, these changes are minor and do not significantly affect topology in the short term. Additionally, satellite networks are equipped with same optical inter-satellite links. Given this assumption, our datasets are primarily influenced by offerload and network scale.

The offerload represents the proportion of traffic sent per second relative to the total network bandwidth, which mainly affects the sparsity of traffic data. Sparse traffic data proposes a significant challenge for deep learning models [1][2]. It reduces the model's ability to learn useful feature representations, slows down or prevents model convergence, and reduces computational efficiency. To evaluate the model's performance with sparse traffic data, all experiments in this paper use traffic matrix sequences with an offerload of 0.1. This means the traffic injected into the network accounts for only 10% of the total network bandwidth per second, indicating a very low network load [3][4]. Our method, Satformer, demonstrates excellent accuracy and robustness compared to other models across all three datasets with an offerload of 0.1. As offerload increases and the traffic matrix becomes denser, tuning the model's performance becomes easier, resulting in improved performance for both Satformer and other methods.

To represent different network scales, we selected three real-world satellite networks: Iridium (66 nodes), Telesat (298 nodes), and Starlink (1584 nodes), corresponding to small, medium, and large-scale satellite networks, respectively. Our test results have shown that our model Satformer performs well enough for networks of thousands of satellites.

**Reference:**

[1] Zhang, Wenhao, et al. "Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning." Proceedings of The Web Conference 2020. 2020.

[2] Yuan, Yuan, et al. "Spatio-Temporal Few-Shot Learning via Diffusive Neural Network Generation." The Twelfth International Conference on Learning Representations. 2024.

[3] Wang, Ruibo, Mustafa A. Kishk, and Mohamed-Slim Alouini. "Reliability analysis of multi-hop routing in multi-tier leo satellite networks." IEEE Transactions on Wireless Communications (2023).

[4] Qin, Liang, et al. "Orchid: enhancing HPC interconnection networks through infrequent topology reconfiguration." Journal of Optical Communications and Networking 16.6 (2024): 644-658.

## Questions 1:
_What is the unique challenges of traffic data estimation for satellite networks, compared with other network types?_

## Response Q1:
Thank you for your question. Other networks, such as WANs [1][2] and HPC/DC interconnect networks [3][4], are static with infrequent changes in node connections. Additionally, these networks typically have high link quality, resulting in complete and reliable for collected traffic data. Moreover, they usually have rich computational resources, allowing for the deployment of complex models. In contrast, satellite networks have highly dynamic nodes, leading to frequent changes in connections and poor link quality [5]. Limited computational resources further complicate traffic estimation in satellite networks, presenting unique challenges summarized as follows: 

1. Spatial and temporal dynamics: Satellite networks experience highly dynamic traffic patterns due to the constant movement of satellites, varying regional demands, and differing satellite coverage areas.

2. Sparsity and incompleteness: Traffic data in satellite networks is often sparse and incomplete due to the selective and intermittent nature of communication links and Space weather, radiation, and other environmental factors, making it difficulity to the parameter adjustment of deep learning models.

3. Limited computational resources: Satellites have limited computational resources and power, which can restrict the complexity of the traffic estimation models that can be deployed on them.
**Proof of Lemma 2**

The mask matrix $M$ is designed such that $M_{ij} = 0$ for elements outside the central region and $M_{ij} = 1$ within the central region.This implies that only the attention scores corresponding to the central region are retained, effectively reducing the complexity of the attention mechanism by filtering out less relevant data.Mathematically, $M$ acts as a sparsity-inducing regularizer:
$A'(X) = \text{softmax}(\Psi \odot (XW_QW_K^\top X^\top))$
By focusing on the central region, $M$ ensures that the attention mechanism does not overfit to peripheral noise, enhancing generalization.

The central region often contains the most informative parts of the data, as observed in empirical studies (e.g., Figure 8(a) in the original paper).

By applying $M$, the attention mechanism $A'(X)$ prioritizes the computation of attention scores within this region, thus capturing critical local patterns:

$A'(X)_{ij} = \frac{\exp((\Psi \odot (XW_QW_K^\top X^\top))_{ij})}{\sum_k \exp((\Psi \odot (XW_QW_K^\top X^\top))_{ik})}$


This prioritization ensures that the most relevant features are emphasized, leading to improved prediction accuracy.

To show that the deviation introduced by $M$ is bounded, consider the difference between the original and masked attention mechanisms:

$\Delta A = A(X) - A'(X)$
Since $M_{ij} \in \{0, 1\}$​, it acts as a binary mask, thus the modification is limited to setting some attention scores to zero while retaining the rest:
$\| \Delta A \|_F = \| \text{softmax}(XXW_QW_K^\top X^\top) - \text{softmax}(\Psi \odot (XXW_QW_K^\top X^\top)) \|_F$
Given that $\text{softmax}$ is a Lipschitz continuous function with constant $1$, the Frobenius norm $\| \Delta A \|_F$ is bounded by the norm of the difference in the inputs:

$\| \Delta A \|_F \leq \| (1-\Psi) \odot (XXW_QW_K^\top X^\top) \|_F$
Since $M$ zeros out peripheral entries, the difference is confined to the less relevant regions, ensuring that the overall deviation remains controlled.`
## Weakness 2:

_The paper primarily focuses on the application of Satformer in satellite networks. While it mentions potential applications in other areas, it does not provide extensive testing or results in those domains, which may limit the perceived generalizability of the model._
## Response W2:

Although Satformer was developed to solve the problem of traffic data estimation in satellite networks, the core strengths of its methodology give it the potential to be applied to other tensor completion tasks that need to deal with large-scale, sparse, and complex spatio-temporal characteristics. This includes, but is not limited to, social network analysis, environmental monitoring, bioinformatics, and other areas that require the reconstruction and analysis of multidimensional data.

We used Foursquare tensor dataset [1] and PeMS-Bay dataset[2] to evaluate Satformer's performance and generalization ability. Foursquare is a point of interest tensor defined by (user, location, timestamp); PeMS-Bay data collect this traffic volume data from 325 loop sensors in the San Francisco bay area, ranging from January to March 2018 with a 5-min time interval.

The test results are shown on the _TABLE I_ in pdf at the **Author Rebuttal** Block highlighted in orange. These positive results show that Satformer can be used as a powerful tool in a variety of fields involving complex spatio-temporal data, such as social network analysis, traffic flow forecasting, environmental monitoring, and more. Of course, for different application scenarios, further adjustments and optimizations may be required to achieve the best performance. 

[1] Oh, Sejoon, et al. “Influence-Guided Data Augmentation for Neural Tensor Completion.” Proceedings of the 30th ACM International Conference on Information &amp; Knowledge Management, 2021.

[2] Meng, Chuishi, et al. “City-Wide Traffic Volume Inference with Loop Detector Data and Taxi Trajectories.” Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2017.
## Limitations 1:
_The novelty of the proposed method is limited._

## Response L1:

Thank you for your kindly comment and for recognizing the effort that went into our work. Your concern about novelty is valuable.

First of all, satellite networks experience long link delays, high dynamics, and limited computing resources, resulting in high overhead and costs for measuring global traffic data. Thus, the primary focus of this paper is to reconstruct the remaining traffic data from a small sample (2%) of the satellite network data, rather than predicting future traffic. These traffic estimation methods encounter unique challenges due to the highly dynamic nature of satellite networks and the sparse and incomplete nature of traffic data. To the best of our knowledge, this is the first work on satellite network traffic estimation.

Follow your suggestion, we restate the novelty. Our methodology, named "Satformer," differs from existing methods. We propose an Adaptive Sparse Spatio-Temporal Attention Mechanism (ASSIT), which dynamically adjusts the sparsity threshold of the attention matrix to enhance the model's inference efficiency when handling large-scale sparse inputs. This mechanism enables the model to adaptively allocate computing resources based on the sparsity level of different datasets, thereby improving operational efficiency and scalability.

Additionally, a Graph Convolutional Network (GCN) is introduced as a graph embedding module to handle data with non-Euclidean structures. This module effectively captures local and global information between nodes, enhancing the model's ability to extract nonlinear spatio-temporal correlations.

Furthermore, we introduce a transfer module to ensure no omissions occur in the long-term transmission between the encoder and decoder. This module employs a self-attention mechanism to comprehensively consider the entire input sequence when generating the output for each time slice, thereby improving the accuracy of missing value estimations.

In summary, our work introduces innovations in several aspects, with its effectiveness and superiority verified through mathematical proofs and experiments.
### 3. AGCRN [3]

Input:
The input of AGCRN includes historical time series data and dynamic graph structure. Specifically, the input data is of the following form:

Historical time series data: denoted as $X = (X_{t1}, X_{t2},... , X_{tP}) \in \mathbb{R}^{P \times N \times C}$, where P is the number of historical time steps, N is the number of nodes, and $C$ is the number of features. Each $X_{ti}$ represents the graph node feature matrix at the $i$ time step.

Dynamic graph structure: adaptive graph convolution is used to capture the dynamic spatial dependencies, and an adaptive adjacency matrix $\mathbf{A}_{\text{adaptive}}$ is used, which is learned from data and represents the time-varying relationship between nodes.

Output: The output of AGCRN is the predicted future time series data, which is expressed as:

$\hat{Y}=(\hat{X}_{t_{P+1}},\hat{X}_{t_{P+2}},... ,\hat{X}_{t_{P+Q}})\in\mathbb{R}^{Q\times N\times C}$

where, $Q$ is the number of time steps to predict, and $\hat{X}_{t{P+1}}$ is the graph node feature matrix at the first future time step to predict.
### Our model (Satformer)

Our research is not purely historical traffic data interpolation, but real-time interpolation and recovery for traffic data, which has significant differences in processing. Our core task is to deal with missing traffic data so that the data is more complete and reliable for further analysis. Our research objectives and methods include:

Input: Traffic flow matrix with missing data $\mathbf{X} \in \mathbb{R}^{T \times N \times N}$, where $T$is the number of time steps and $N$is the number of satellites.

Output: The complete padded flow $\mathbf{\hat{X}} \in \mathbb{R}^{T \times N \times N}$.

Model structure: Interpolation model based on spatio-temporal graph neural network, using adjacency matrix and time series information for data recovery.

In summary, our research goal is fundamentally different from GraphWavenet, AGCRN, and GMAN. While the former focuses on spatio-temporal traffic prediction, we focus on real-time interpolation and recovery of flow data to ensure data integrity and reliability. It is hoped that these detailed mathematical explanations and comparisons can help clarify how our study differs from other models.

### Comparison of other state-of-the-art models

However, in order to further verify the effectiveness of our research results, we added other models (CDSA [4], SAITS [5]) for comparison, and the results are shown on the _TABLE II_ and _TABLE III_ in pdf at the **Author Rebuttal** Block highlighted in orange.

Satformer is based on the Transformer architecture, which is very powerful in handling long-distance dependencies and suitable for handling time series data with variable patterns. Although CDSA also utilizes the self-attention mechanism, its dimension-wise processing may limit its ability to capture complex interactions. Rnn-based models of SAITS are generally inferior to Transformer architectures in terms of handling long-distance dependencies and efficiency.

**Reference**

[1] Zheng, Chuanpan, et al. "Gman: A graph multi-attention network for traffic prediction." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 01. 2020.

[2] Wu, Zonghan, et al. "Graph wavenet for deep spatial-temporal graph modeling." arXiv preprint arXiv:1906.00121 (2019).

[3] Gao, Xicai, et al. "An AGCRN Algorithm for Pressure Prediction in an Ultra-Long Mining Face in a Medium–Thick Coal Seam in the Northern Shaanxi Area, China." Applied Sciences 13.20 (2023): 11369.

[4] Ma, Jiawei, et al. "CDSA: cross-dimensional self-attention for multivariate, geo-tagged time series imputation." arXiv preprint arXiv:1905.09904 (2019).

[5] Du, Wenjie, David Côté, and Yan Liu. "Saits: Self-attention-based imputation for time series." Expert Systems with Applications 219 (2023): 119619.
## Weakness 3:
_NMAE and NRMSE are used as the metrics to gauge the prediction performance of the methods in this work, what about MSE, MAE, and MAPE?_

## Response W3: 

Thank you for your comment. Although MSE, MAE and MAPE are also commonly used evaluation metrics, they may not be suitable to evaluate satellite network traffic estimation methods. The primary evaluation metrics used in this work, NMAE and NRMSE, as well as MSE, MAE and MAPE, are calculated by the following equations, where ${{\\chi _{ijt}}}$ and ${{\tilde {\\chi} _{ijt}}}$ represent the truth value and estimated value, ${\bar {\rm A}}$ denotes the set of unobserved traffic data.

$NMAE=\\frac{ \\sum\\nolimits_{(i,j,t) \\in \\bar {\\rm A}} {| {{\\chi _{ijt}}} - {{\\tilde {\\chi} _{ijt}}} |} } {\\sum\\nolimits _ {(i,j,t) \\in \\bar {\\rm A}} {| {{\\chi _{ijt}}}|}}$

$MAE=\\frac{ \\sum\\nolimits_{(i,j,t) \\in \\bar {\\rm A}} {| {{\\chi _{ijt}}} - {{\\tilde {\\chi} _{ijt}}} |} } {t}$

$NRMSE=\\sqrt {\\frac{ \\sum\\nolimits_{(i,j,t) \\in \\bar {\\rm A}} {| {{\\chi _{ijt}}} - {{\\tilde {\\chi} _{ijt}}} |}^2 } {\\sum\\nolimits _ {(i,j,t) \\in \\bar {\\rm A}} {| {{\\chi _{ijt}}}|}^2} }$

$MSE=\\frac{ \\sum\\nolimits_{(i,j,t) \\in \\bar {\\rm A}} {| {{\\chi _{ijt}}} - {{\\tilde {\\chi} _{ijt}}} |} ^ 2} {t}$

It can be seen from the equations that NMAE and NRMSE are MAE and RMSE normalized by the truth value range, and RMSE is the root of MSE. Compared with MAE and MSE, it can not only effectively measure the estimation error, but also take into account the size of the error and its variability in different traffic data estimation scenarios in satellite networks.

$MAPE=\\frac{  100\\% \\times \\sum\\nolimits_{(i,j,t) \\in \\bar {\\rm A}} {| \\frac { {{\\chi _{ijt}}} - {{\\tilde {\\chi} _{ijt}}} }{ {{\\chi _{ijt}}}}|} } {t}$

From the equation of MAPE, it can be seen that its truth value cannot be 0; however, the truth data collected from the satellite networks may be 0, so it cannot be used as an evaluation metric in this work.

## Weakness 4:
_Some important baselines are missing, e.g., GraphWavenet, AGCRN, and GMAN, etc._
## Response W4:

Thank you very much for your comments. Our research objectives and application scenarios are fundamentally different from GraphWavenet, AGCRN, and GMAN. The main task of GraphWavenet and GMAN is to predict the traffic state at future moments, while our work focuses on interpolating the existing flow data so as to recover the complete flow data for further analysis and use. Although these two types of tasks both involve the processing of spatio-temporal data, the focus and specific application scenarios are different, so taking these models as baseline models is not suitable for our study. Here's a more detailed explanation:

### 1. GMAN [1]
The core task of GMAN is spatio-temporal traffic prediction. It achieves the prediction of future traffic conditions by encoding and decoding historical traffic data. Mathematically, the basic flow includes:

Input: Historical traffic data $X = (X_{t1}, X_{t2},... , X_{tP}) \in \mathbb{R}^{P \times N \times C}$, where P is the number of historical time steps, N is the number of sensors, and $C$ is the feature dimension.

Output: predict the future traffic data $\hat{Y} = (\hat{X}_{t{P+1}}, \hat{X}_{t{P+2}}, ..., \hat{X}_{t{P+Q}}) \in \mathbb{R}^{Q \times N \times C}$, $Q$ is predicted time steps.

### 2. GraphWaveNet [2]
Input: The input of GraphWaveNet model is a multi-dimensional spatio-temporal traffic data sequence and related graph structure information. These include:

Historical traffic data series:
Representation: $\mathbf{X} \in \mathbb{R}^{P \times N \times C}$
Where $\mathbf{X}$ is a 3D tensor, $P$is the historical time step, $N$ is the number of sensors, and $C$ is the feature dimension (usually traffic flow, speed, etc.).

Graph structure information:
Adjacency matrix: $\mathbf{A} \in \mathbb{R}^{N \times N}$
Description: $\mathbf{A}$ represents the adjacency relation of the sensor network, which is used to model the spatial dependencies.

Output:
The output of the GraphWaveNet model is the predicted value of the future traffic data. These include:

Future traffic data series:
Representation: $\hat{\mathbf{Y}} \in \mathbb{R}^{Q \times N \times C}$

Where $\hat{\mathbf{Y}}$ is a 3D tensor, $Q$ is the time step to predict, $N$is the number of sensors, and $C$ is the feature dimension (typically traffic flow, speed, etc.).
## Weakness 2:
_The motivation of this work is unclear, what is the main difference between the satellite network traffic data and other spatio-temporal data, e.g., traffic flow/speed data?_

## Response W2:

We are grateful for your feedback. The dynamic shift of nodes, the variety of spatial distance and transmission delay, data sparsity, and incompleteness are the primary factors contributing to the complexity of satellite network traffic data, making its spatial characteristics more complex and challenging to model than those of other data sets (like transportation traffic data). A mathematical comparison of the intricacy of data from satellite networks is provided below:

### 1. Spatial distance and transmission latency
In satellite networks, transmission delay and orbit altitude—both of which are dynamically changing—affect data transmission in addition to the physical distance between nodes. However, a traffic network can be represented by a relatively stable network diagram because the arrangement of the cars and roads is typically fixed or changes gradually.

#### i. For the transportation network:
The path length $d_{ij}$ is usually fixed, and the transmission delay is relatively stable, which is mainly affected by the traffic flow:

$\tau_{ij}=f(d_{ij},v_{ij}(t))$

where $v_{ij}(t)$ is the average speed at time $t$.
​
#### ii. For satellite networks
​
The transmission delay varies with time and is related to the orbital position of the satellite and the relative position between the ground station:

$d_{ij}(t)=\sqrt{(x_i-x_j(t))^2+(y_i-y_j(t))^2+z_j(t)^2}$

$\tau_{ij}(t)=f(\frac{d_{ij}(t)}c)+\delta(t)$

where $(x_i, y_i) $ is the location of the ground station, $(x_j(t), y_j (t), z_j (t))$ $t$ is time the location of the satellite $d_{ij}(t)$ is the distance at time $t$, $c$ is the speed of light, $\delta(t)$ is the other delay factor (such as processing time).

### 2. Data sparsity and incompleteness

Transportation traffic data usually come from fixed sensor networks or GPS devices, and these data sources are relatively stable in time and space. Although sparsity and incompleteness also exist, this sparsity has certain predictability and periodicity because traffic data usually exhibit strong temporal and spatial continuity.

However, for satellite networks, due to the highly dynamic network topology, and the links between satellites and between satellites and ground stations may be intermittent for various reasons (e.g., geographical location, orbital position), resulting in sparse data. Most of the elements in the traffic data matrix $Y\in\mathbb{R}^{I\times J\times T}$ may be zero, since not all ground station-satellite and satellite-satellite pairs have traffic at each time step. 

At the same time, the traffic data in the satellite network changes rapidly and irregularly, and lacks the spatio-temporal continuity in the traffic data. And data missing is often unpredictable, as it can occur at any time and the spatio-temporal pattern of missing is not stable.

Such sparsity and incompleteness make the flow data of satellite networks far more complex than other data sets such as traffic data, which requires higher-order algorithms and models for accurate analysis and estimation.
## Weakness 4:
_There are quite a few approaches to obtain graph embedding (or just a node embedding), why the authors choose this specific GCN for embedding?_
## Response W4:

Thank you for your question. In our cases, satellites have limited computational resources and power, which can restrict the complexity of the traffic estimation models that can be deployed on them. Although there are many graph embedding modules that outperform GCN such as Graph Attention Network (GAT) [1] or deep graph networks such as GraphSAGE [2]. While other graph models such as GAT or GraphSAGE may perform better on some tasks, GCN in most cases provides the necessary balance ---- that is, preserving computational and implementation simplicity while effectively capturing graph structural information. In particular, these properties of GCN are invaluable in applications where resources are constrained, such as limited computing resources in satellite communications, or fast response is required.
​
Let's compare GAT and GraphSAGE to prove our point:

### GAT

GAT optimizes the aggregation process of node features by introducing learnable attention weights for each pair of adjacent nodes in the graph. The mathematical expression is as follows:

$\operatorname{Attention}\left(h_i, h_j\right)=\operatorname{softmax}_j(\vec{a}^T([W h_i \| W h_j])$

$\alpha_{ij}=\frac{\exp(\text{LeakyReLU}(\vec{a}^T[Wh_i\|Wh_j]))}{\sum_{k\in\mathcal{N}(i)}\exp(\text{LeakyReLU}(\vec{a}^T[Wh_i\|Wh_k]))}$

$h_i'=\sigma\left(\sum_{j\in\mathcal{N}(i)}\alpha_{ij}Wh_j\right)$

where $h_i$ is the feature vector of node $i$, $W$ is a learnable weight matrix, $\alpha_{ij}$ is the attention coefficient between nodes $i$ and $j$, $\sigma$ is the nonlinear activation function, and $\vec{a}$ is the parameter of the attention mechanism.

The feature propagation of GCN mainly relies on the product of adjacency matrix and feature matrix, which is usually simpler and more efficient than the attention mechanism calculation of GAT, because GAT needs to calculate complex attention coefficients for each pair of adjacent nodes. This results in relatively high computational complexity, especially in graphs with large node degrees. For large-scale satellite networks, this can lead to excessive consumption of computational resources.
​

At the same time, GAT introduced a weight matrix $W$ and an attention vector $\vec{a}$ as parameters of the model. For each attention head, the total number of parameters of $W$ and $\vec{a}$ is $F^{\prime}\times F+2F^{\prime}$, where $F^{\prime}$ is the dimension of the output feature and $F$ is the dimension of the input feature. Assuming that the model uses multi-head attention, the total number of parameters is $H \times (F^{\prime}\times F+2F^{\prime})$. The combination of multiple heads and high-dimensional features rapidly increases the total number of parameters of the model, which increases the difficulty of training and can lead to overfitting.

### GraphSAGE
GraphSAGE is an inductive graph learning framework that updates the embeddings of nodes by sampling a fixed number of neighbors and leveraging an aggregation function. The basic formula is as follows:

$h_{\mathcal{N}(i)}^{\prime}=\text{AGGREGATE}_k(\{h_j,\forall j\in\mathcal{N}(i)\})$

$h_i^{\prime}=\sigma(W\cdot\text{CONCAT}(h_i,h_{\mathcal{N}(i)}^{\prime}))$

where $\mathcal{N}(i)$ is a neighbor of node $i$, $\text{AGGREGATE}$ are aggregation functions such as mean pooling, etc and $W$ is the learnable weight matrix.

However, GraphSAGE needs to sample the neighbors of each node, which may lead to information loss or computational inefficiency when the graph data is very large or the nodes are very densely connected.

What's more, the performance of GraphSAGE heavily depends on the choice of aggregation function and neighbor sampling strategy, which may lead to unstable model performance in different graph structures or data distributions. For example, if the sampling is not uniform or some types of connections are too sparse in the graph, the aggregation result may not represent the true distribution of neighbors, thus affecting the generalization ability of the model.

### Conclusion

The above analysis and proofs show the challenges that GAT and GraphSAGE may face when they need to deal with large-scale, sparse and dynamic satellite network data. These challenges may be less significant when using GCN. GCN offers a more practical solution in this scenario through its simplified structure and lower computational complexity.

**References**

[1] Veličković, Petar, et al. "Graph attention networks." arXiv preprint arXiv:1710.10903 (2017).

[2] Hamilton, Will, Zhitao Ying, and Jure Leskovec. "Inductive representation learning on large graphs." Advances in neural information processing systems 30 (2017).



We would like to thank the reviewer again for taking the time to review our paper.
## Weakness 3:
_Applying L1 to the attention score matrix cannot make the output to be sparse._

## Response W3:
Thank you for your question. Regularization is usually used in the design of loss functions, but we borrow the idea of regularization here to further process the data. 
What's more, we mathematically and conceptually clarify why L1 regularization is used and how it affects the sparsity of attention scores [1].

L1 regularization works by adding a penalty term equal to the absolute magnitude of the coefficient. For attention scores, consider the attention score matrix $\alpha$, where each element $\alpha_{ij}$represents the attention score of node $i$ to node $j$. The L1 regularization term can be written as follows：

$L 1(\alpha)=\lambda \sum_{i, j}\left|\alpha_{i j}\right|$

where $\lambda$ is a regularization parameter that controls the trade-off between sparsity and fitting the data.
In Satformer, regularization affects attention scores by penalizing non-zero values, effectively pushing less important connections (lower attention scores) toward zero. This is reflected in the update equation for the attention score that includes the L1 term, which can be expressed as follows.

$\alpha_{\text {new }}=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}-\lambda\right)$

where $\lambda$ acts as a threshold that the score must exceed to be considered important, effectively zeroing out smaller scores and thus enforcing sparsity.

Let's take an example to explain in detail:
​
Suppose we have a small satellite network consisting of 3 satellites. Each satellite records communication data with other satellites. We first construct the query (Q) and key (K) matrices, assuming dimension $d_k=3$ (that is, 3 features per satellite):

$Q=\left[\begin{array}{ccc}1 & 0.5 & 0.2 \\\\ 0.5 & 1 & 0.3 \\\\ 0.2 & 0.3 & 1\end{array}\right], Q=K^T$

Calculate $\frac{QK^T}{\sqrt{d_k}}= \left[\begin{array}{ccc}
1.45 & 1.19 & 0.62 \\\\
1.19 & 1.34 & 0.73 \\\\
0.62 & 0.73 & 1.13 \\\\
\end{array}\right] $

Apply the softmax function:
$ \alpha = \text{softmax}\left(\frac{QK^T}{\sqrt{3}}\right) = \left[\begin{array}{ccc}
0.46 & 0.34 & 0.20 \\\\
0.34 & 0.42 & 0.24 \\\\
0.20 & 0.24 & 0.56 \\\\
\end{array}\right] $

Now, we introduce L1 regularization and assume $\lambda = 0.1 $. The updated attention score is calculated as follows:

Subtract $\lambda $ and reapply softmax:
$ \alpha' = \text{softmax}\left(\frac{QK^T}{\sqrt{3}} - \lambda\right) = \left[\begin{array}{ccc}
0.60 & 0.21 & 0.01 \\\\
0.21 & 0.55 & 0.05 \\\\
0.01 & 0.05 & 0.94 \\\\
\end{array}\right] $

It can be seen that the regularization parameter $\lambda $ effectively increases the sparsity of the matrix by decreasing the values of all elements, making some smaller values close to zero.
By comparing $\alpha $ and $\alpha' $, it can be seen that L1 regularization helps the model focus on more important communication paths (higher scores) while suppressing less important paths (paths with lower or near zero scores). This sparsity is particularly useful when dealing with complex networks, as it can reduce unnecessary computations and improve model interpretability.

**References**

[1] McCulloch, Jeremy A., et al. "On sparse regression, Lp‐regularization, and automated model discovery." International Journal for Numerical Methods in Engineering 125.14 (2024): e7481.
## Weakness 2:
_The authors explained that their model outperforms SPIN because “SPIN’s ability to handle sparsity or irregularly sampled data might be limited”. However, SPIN is designed for sparse data, and hence the intuition on why the proposed method is better is not provided._

## Response W2:
Thank you for your question. We compare the performance of SPIN and Satformer in large-scale sparse data scenarios and draw the following conclusions:
### 1. Design limitations of SPIN
SPIN has the problems of blocked information propagation path and poor adaptability when dealing with sparse data. In the extremely sparse data environment, the information flow path is easy to be blocked, and it cannot quickly adapt to different sparse data distribution patterns, which leads to its unstable performance in dealing with dynamic and complex sparse data. These limitations may significantly affect the performance and reliability of the model in practice
#### i. High computational complexity:
Although the spatio-temporal attention mechanism of SPIN is effective, due to its squared Complexity 

$Complexity ∝O((N_{max}+E_{max})T^2)$

in time steps and the number of nodes, where $N_{max}$ and $E_{max}$ are the maximum number of nodes and the maximum number of edges, respectively, and $T$is the time step, resulting in high computational overhead. This is particularly evident when dealing with long time series and large-scale nodes.
​
#### ii. Information transmission path blocked:
Specifically, if node $i$is completely missing at time step $t$, and this node is an important bridge for information transmission between other nodes in the spatio-temporal graph, this missing will lead to the interruption of the information propagation chain and affect the information update and feature extraction of multiple nodes.
​
#### iii. Poor adaptability:
SPIN suffers from dependency on sparsity patterns: SPIN may perform well when dealing with some specific sparsity patterns, but its performance degrades significantly when facing other types of sparsity patterns. For example, for sparse patterns with long continuous misses, SPIN may not be able to efficiently aggregate enough historical information for accurate predictions. The mathematical description is as follows:

$\alpha_{i,j}=\frac{\exp{(e_{i,j})}}{\sum_{k\in N(i)}\exp{(e_{i,k})}}$

where $\alpha_{i,j}$ denotes the attention weight and $e_{i,j}$ is the attention score from node $i$ to node $j$. When the sparsity pattern changes, the recomputation of attention weights may not reflect the new data distribution in time, resulting in poor information aggregation.
​
In contrast, Satformer significantly improves performance in processing highly dynamic complex sparse data in the following ways:
### 2. Advantages of Satformer
#### i. Adaptive Sparse Spatio-Temporal Attention Mechanism (ASSIT) :
ASSIT can deal with a large number of sparse inputs more efficiently through the dynamic adjustment of multi-head self-attention structure and sparse threshold. This mechanism allows the model to focus on the data dense region when dealing with sparse data, thus improving the performance of the model in sparse data scenarios. The mathematical description is as follows:

$Q=Z_{\text {div }} W_q, \quad K=Z_{\text {div }} W_k, \quad V=Z_{\text {div }} W_v$

where $Z_{\text {div}}$represents the local region of the input tensor, and $W_q, W_k$, and $W_v$ are the projection weights for the query, key, and value, respectively. Through the local attention mechanism calculation, the model's attention to the local region of the input sequence is enhanced.
​
#### ii. Graph embedding module:
The Graph Convolutional Network (GCN) was used to process non-Euclidean data, which effectively captured the relationship between nodes and further enhanced the model's ability to process high-dimensional sparse data.

$H^{(l+1)}=\sigma\left(\tilde{D}^{-1 / 2} \tilde{A} \tilde{D}^{-1 / 2} H^{(l)} W^{(l)}\right)$

among them, the $\tilde{A} = A + I, \tilde{D}$ is degree matrix, $\sigma$ is the activation function, $W^{}(l)$ is the weighting matrix layer. This mechanism enables the model to effectively extract local and global information in non-Euclidean space, which performs well especially in sparse data environments.

#### iii. Computational complexity:
Given that the dimension of the input matrix is $M \times K$ and the region size is $D \times D$. For each Query, we can choose a region around it that is $D \times D$. In this case, each Query only considers the $D$ Key vector around it. Therefore, for each Query, the complexity of calculating the dot product is $O(D \times d)$, where $d=\frac{K}{D}$. Since each Query requires such a computation, the total complexity is $O(M \times D \times d)=O(M \times K)$.