# Automated Chest X-Ray Diagnosis Report Generation with Cross-Attention Mechanism
## 2 Materials and Methods
### Method
#### 2.2.2 Modal Alignment Module  
Modal alignment is an important part of X-ray report generation. Modal alignment is not the final goal in this task, but an intermediate step in the task, so it is an implicit alignment method. In the field of vision-text, there are two types of modal alignment: coarse-grained alignment and fine-grained alignment. Coarse-grained alignment specifies an entire image and a specific text description. This alignment loses the relevance between the specific area of the image and the text description and cannot make good use of the data. Fine-grained alignment matches the image area with the corresponding description part. In the field of vision-text, a common fine-grained alignment method is to use object detectors such as Fast-Rcnn11 [18] and graph neural networks; this type of target detector can detect each object part in the image and identify the corresponding description part. However, in the medical field, due to the high professional requirements of the medical data field, there is a lack of annotated data, which leads to the lack of ready-made medical X-ray target detectors. In the modality alignment method without target detector, a common method is to use the attention mechanism. The attention method we need is an algorithm that can combine two modalities at the same time. The cross-attention module in Transformer [17] is basically the same as the self-attention module; the difference is that the cross-attention allows two different sequences to be input, so that both modalities can be considered simultaneously when calculating the attention score. The modality alignment module proposed here is based on the cross-attention mechanism. The input of the module is the visual features extracted from the chest X-ray and the text converted into a sequence. The cosine similarity of the image and text sequence is then calculated, the similarity is mapped to a score through the softmax function, and then a dot multiplication operation is performed with the source image sequence. This makes the attention weight of the image take into account the information of the text, which aligns the image and text areas to a certain extent. Let the sequence of images be represented as \(I=(i_{1},i_{2},i_{3},i_{4},\dots,i_{v})\), and the text sequence represented as \(T=(t_{1},t_{2},t_{3},t_{4},\dots,t_{v})\). \(W_{i}\) and \(W_{j}\) are the weight matrix. Its function is to map the two modes into the same dimensional space for easy calculation.  
\[I_{v} =W_{i}\times I \tag{8}\] \[T_{v} =W_{j}\times T \tag{9}\]  
The cosine similarity of the image and text sequences mapped to the same dimension is then solved. Cosine similarity uses the cosine value of the angle between the spatial vectors to measure the similarity between two vectors. The larger the angle between the two vectors, the lower the similarity, and vice versa. In this task, the image and text sequences are in the form of matrices, and the dot product operation of the matrix can be used to solve the cosine similarity. Due to the nature of the dot product operation, when the dimension dim is particularly large, the variance of the matrix will be large. In order to reduce the variance and maintain the integrity of the data, we can add a scaling factor.  
\[Similar=\frac{I_{v}\times T_{v}^{T}}{\sqrt{dim}} \tag{10}\]  
The similarity scores are then compressed into a vector using a normalized exponential function so that each score ranges between 0 and 1 and the sum of all scores is 1. Similar is represented by the abbreviation \(S\).  
\[Score=\frac{e^{\mathcal{S}_{j}}}{\sum_{j=1}^{n}e^{\mathcal{S}_{j}}} \tag{11}\]  
The image features are multiplied by the similarity score, so that the image data can pay attention to the text data at the same time during the training process. In order to prevent the loss of image data features, the residual mechanism is used to add the source image data to the data after the dot product:  
\[I_{v}=I_{v}+I_{v}\times Score \tag{12}\]