## 2 Methods
### Neural Network Structure  
To achieve the two primary goals, we designed two network structures (ReConNet, ReConNet-Ensemble) for the macular atrophy prediction applications based on multiple patient visits: (1) prediction of future GA and Stargardt atrophy regions using longitudinal FAF images (ReConNet); (2) prediction of future Stargardt atrophy regions using longitudinal SD-OCT images (ReConNet-Ensemble). Additionally, we also investigated the feasibility of the prediction of interval growth of GA and Stargardt atrophy regions using longitudinal FAF images (ReConNet-Interval). All three algorithms were implemented using the open-source deep learning framework Keras. The first neural network, ReConNet, used in this study was for longitudinal prediction of progression of GA and Stargardt atrophy on FAF images based on the network architecture, which has an encoding pathway and a decoding pathway, with a concatenation linking the output of an encoding pathway convolution to the input of a decoding pathway convolution. The term ReConNet was used for both FAF and OCT to be flexible in the future to include additional ensemble network inputs for concurrent learning. Such a deep learning architecture can be used for semantic segmentation using a relatively small training set of images [18]. To incorporate longitudinal data, the neural network in this study had multiple encoding pathways, one for each timestep. At each layer of the encoding pathway, the outputs of the encoding pathway convolutions were combined using a 2D convolutional-LSTM layer. The output of this layer was concatenated with the input of the corresponding decoding pathway convolution. Note that the overall network architectures for ReConNet and ReConNet-Interval were similar, and the difference was that the input for ReConNet was original FAF image (and related ground truth) and for ReConNet-Interval was the FAF image with the region of atrophy set to zero (and related interval ground truth), reflecting the growth regions between any two adjacent patient visits. A schematic of the neural network used in this study is depicted in Figure 1.  
The second neural network structure, ReConNet-Ensemble, used in this study was for longitudinal prediction of progression of Stargardt atrophy on OCT images. The ensemble inputs included multiple enhanced enface feature maps derived from the OCT data surrounding the EZ in the depth (z-direction) of the OCT, the region of the retina most affected by Stargardt atrophy. These feature maps also extended beyond the traditional mean intensity maps, additionally including minimum intensity, maximum intensity, median intensity, standard deviation, skewness, kurtosis, gray-level entropy, and thickness of the EZ similar to that used in our previous work [18]. The network structure of ReConNet-Ensemble was similar to that previously described by our team [18]. The difference was that the previous one was for prediction using baseline alone, and the ReConNet-Ensemble in this project included longitudinal data with three patient visits (month 0, month 6, and month 12), reflecting dynamic changes of atrophy over time. In ReConNet-Ensemble neural network, multiple of the above-described neural networks were used to take in the multiple OCT-derived enface feature maps as input, allowing for the incorporation of the three-dimensional OCT information at EZ. In our previous work, we found significant improvement for the prediction of Stargardt atrophy when incorporating these advanced feature maps compared to using mean intensity alone. The logit layers of the individual neural networks were combined through averaging, with the combined result subsequently being sent through a softmax function to predict probability maps of atrophy. The ReConNet-Ensemble neural network is depicted in Figure 2.  
Figure 1: A schematic of the recurrent neural network architecture. The first T to T + N columns from the top blue squares to the bottom purple squares represent multiple encoding pathways, one for each timestep.