# Computer Vision Techniques for Growth Prediction:
## 3 Results (Extraction & Analysis)
### Rq2: Which Technologies Are Used for Growth Prediction Using Image Processing?
#### 3.4.3 Stn (Spatial Transformer Network)  
Spatial Transformer Networks (STNs) can resolve spatial invariance and can be added as modules to give ordinary neural networks the ability to resolve spatial distortions. Another advantage is that the spatial transformation module can be trained from input data only, without using special training data. STNs are created by combining the Localization Network, the Grid Generator, and the Image Sampler.  
The localization network takes the input image and outputs a set of affine transformation parameters. The grid generator then creates a sampling grid, a set of points sampled from the input image. The accuracy of CNNs can be improved by applying \"distortions\" such as rotation and expansion as affine transformations. In predicting the growth of organisms, the grown image at a future \(t+1\) point is interpreted as a distortion from the past image and transformed by affine operation \(\theta_{t}\). Let a target coordinate of the regular grid in the shape image \(S_{t+1}\) is \((x_{t+1}^{i},y_{t+1}^{i})\), a source coordinate \((x_{t}^{i},y_{t}^{i})\) is evaluated by Equation (4) [36].  
\[\begin{pmatrix}x_{t}^{i}\\ y_{t}^{i}\end{pmatrix}=\theta_{t+1}\begin{pmatrix}x_{t}^{i+1}\\ y_{t}^{i+1}\\ 1\end{pmatrix}=\begin{bmatrix}\theta_{0}&\theta_{1}&\theta_{2}\\ \theta_{3}&\theta_{4}&\theta_{5}\end{pmatrix}\begin{pmatrix}x_{t}^{i+1}\\ y_{t}^{i+1}\\ 1\end{pmatrix} \tag{4}\]  
The plant growth prediction by STN [36] define the loss function with L1 norm (is called the Least Absolute Deviation, LAD) by the following formula:  
\[L=\mu_{1}L_{STN}+\mu_{2}L_{SHAPE}+\mu_{3}L_{COLOR}, \tag{5}\]  
where \(L_{STN}=\sum_{k=0}^{3}L_{1}(S_{t-k},S_{t+1})\), \(L_{SHAPE}=L_{1}(\tilde{S}_{t+1},S_{t+1})\), \(L_{COLOR}=L_{1}(\tilde{I}_{t+1},I_{t+1})\). \(S_{t+1}\) is actual grayscale data at time \(t+1\), \(I_{t+1}\) is actual RGB data at time \(t+1\), \(\tilde{S}_{t+1}\) is grayscale image aligned by STN, \(\tilde{I}_{t+1}\) is RGB image aligned by STN, and \(L_{1}\) is L1 norm metric function. [PERSON] et al. [35] adds a correction term \(L_{FG}\) derived from \(F\) obtained from the image with the background removed to the loss function in Equation (5). \(L_{FG}\) is given by the L1 norm of the Hadamard product of \(S_{t+1}\) and \(I_{t+1}\) and \(\tilde{F}_{t+1}(=\theta_{t+1}(F_{t+1}))\) as \(L_{FG}=L_{1}(\tilde{F}_{t+1},S_{t+1}\otimes I_{t+1})\).  
The STN with the hierarchical auto-encoder method [59] first passes the input grayscale image to the STN, divides it into multiple batches, and passes them through the hierarchical autoencoder layer. Although not explicitly stated in that paper, the diagram shows that the method carried out hierarchization by input (no division, half division, and four divisions) to the STN. The hierarchization by division enables overall change prediction and detailed growth predictions, such as the growth of tiny leaves. The authors define the loss function by minimizing two loss functions, \(L_{shape}=L_{1}(\tilde{S}_{t+1},S_{t+1})\) and \(L_{RGB}=L_{1}(\tilde{I}_{t+1},I_{t+1})\).