## 2 Materials and Methods
### Design and Implementation of swHPFM
#### 2.2.3. Serpentine Register Communication  
When analyzing the structured grid rotor solver algorithm, in many kernel functions, such as the viscous flux, convective flux, and parameter rotation, stencil computing is used. There is a backward or forward dependence on the \(x\)-, \(y\)- and \(z\)-axes of the neighboring data. For instance, in the function viscous flux, the absolute velocity components, namely, \(v_{x}\), \(v_{y}\), and \(v_{z}\), the original conserved quantity, and the derivatives of the flow parameters at the interface are needed in order to update the value of the function. In this paper, a scenario in which both forward and backward data are needed was considered as an example, as illustrated in Figure 9. The green cube stands for the current grid that needs to be updated. The red cubes represent the adjacent grids, on which the green one depends. When updating the green grid data block, it is necessary to simultaneously read the forward data and the backward adjacent red blocks of other data. When executing the data transmission, the \(x\)-axis data are continuous; hence, the \(x\)-axis data must be completely transmitted. When the data on the discontinuous latitude in the \(y\)-axis direction are distributed to the CPE, only one datum value can be updated for every three data blocks, and the data utilization rate is so low that the parallel computing performance is severely degraded. In this case, serpentine register communication (SRC) is proposed to improve the data utilization and reduce the number of invalid communications. In each SW26010 processor, register communication is designed to directly communicate only between the same row or the same column. To briefly explain the serpentine method used in this paper, it is supposed that there are 9 CPE resources, which form a \(3\times 3\) mesh. As in the actual scenario, the registers can only communicate within the same row or the same column. Data blocks 1-27 must be updated, and being limited by the SPM space, the SPM for each CPE can store up to 4 data blocks. Simultaneously, each calculation requires one forward data block and one backward data block. The data layout of the original register communication is illustrated in Figure 10a, in which the first CPE and the last CPE receive four data blocks: 1, 2, 3, and 4, as well as 25, 26, 27, and 28, respectively. The middle CPEs store three data blocks. Before communicating, the intermediate data are updated, such as data 5 in CPE1 and data 8 in CPE2. Then, forward communication is performed, and the data of the non-first column are sent forward in the same row. For example, data 4 are transmitted to the CPE0, and the location of data block 0 is covered to update data 3. However, the data in the first and last column, such as CPE3 and CPE2, must be diagonally communicated. Then, backward communication is executed, and the data of the non-last column are sent backward in the same row. For example, data 15 are transmitted to CPE5 to update data 16. The last-column data, such as CPE5 and CPE6, must be diagonally communicated. However, diagonal communication, such as the communication of CPE2 and CPE3, is limited by its design. It must communicate in the same row, initially to transmit data from CPE2 to CPE0, and subsequently to communicate with CPE3 via column communication. This communication  
Figure 10: Register communication method. (**a**) Original register communication in the Sunway TaihuLight supercomputer. (**b**) Serpentine register communication by changing the logical CPE thread number to a snake-like consecutive number. Figure 9: Data dependency in adjacent grids. There is a backward or forward dependence on the \(x\)-, \(y\)- and \(z\)-axes of the neighboring data. method is complicated and time-consuming. For this scenario, serpentine register communication was proposed. By changing the logical CPE thread number to a snake-like consecutive number, as illustrated in Figure 10b, the number of communications was effectively reduced.