# DR-Transformer: A Multi-Features Fusion Framework for Tropical Cyclones Intensity Estimation
## 3 Methods
### Temporal Correlation Learning via Transformer  
It is necessary to utilize temporal information to estimate intensity. As using an image alone is not enough to estimate intensity accurately, there still exists unacceptable errors in some samples. TC intensity at a certain moment relates to the intensities before; learning the temporal correlation of TC is important for remedying results. To achieve this goal, we use the attention mechanism and transformer model.  
Transformer structure: As shown in Figure 2, our transformer has a standard structure and mainly consists of multi-head self-attention modules, feed-forward networks, and cross-attention modules. \(N_{E}\) and \(N_{G}\) are the numbers of encoders and decoders, respectively. Regarding the self-attention of the transformer, we take the input as a query, key, and value. In cross-attention, the outputs from the encoder are regarded as a query and the outputs from decoder self-attention are taken as a query and value. We added fixed positional encodings to the input of each attention layer to record temporal information. To ensure estimation at moment \(t\) can only depend on the known estimation before moment \(t\), we offset the input of the decoder by one position and mask (setting to \(-\infty\)) the attention weights matrix of the decoder. For a sequence of length 4, the decoder weight mask is given by  
\[D_{weights}(i,j)=\begin{cases}1,&\text{if }i<=j\\ \text{-inf},&\text{if }i>j\end{cases} \tag{9}\]  
where blank means we offset the input of the decoder by one position and \"-inf\" means the weights in that position are masked.  
Class vectors: we first utilized the DR-extractor to extract features from the whole training dataset and averaged the features by class (intensity) to construct a global class representation set \(C=\{c_{1},c_{2},...\}\). The class vectors were used to train the transformer and for classification.  
Training transformer: For the training transformer, we needed time series data and class vectors \(C\). We selected a sequence of images and their corresponding intensities from a TC. Then, we utilized the DR-extractor to obtain the discriminative feature embedding of images and, meanwhile, we used the one-hot encoder to encode intensities for its efficiency and orthogonality. Here, \(F=[f(x_{t-T}),f(x_{t-T+1}),\dots,f(x_{t})]\) and \(V=[v_{t-T},v_{t-T+1},\dots,v_{t}]\), respectively, where \(T\) is the length of the sequence. Then, we fed the feature embedding into the encoder and one-hot vectors to the decoder, which can be formulated as:  
\[O=G(\phi,F,V)=[O_{t-T},O_{t-T+1},...,O_{t}] \tag{10}\]  
where \(\phi\) represents the parameters of the transformer and \(O\) is the equal length sequence from the transformer outputs. Finally, we took each element of the transformer outputs \(O\) as an anchor and calculated DC loss between it and class vectors \(C\), which can be formulated as  
\[\ell=\sum_{i=0}^{T}\ell_{DC}(O_{t-i},C) \tag{11}\]  
Equation (11) was used to train the transformer model. The way we trained the transformer increases the constraints on its output and makes it converge better. We still usec DC loss because it can reduce the errors and is harder to overfit than other classification losses.