# Fuzzy Clustering Approaches Based on Numerical Optimizations of Modified Objective Functions
## 3 Optimization Based on Numerical Techniques
### Adaptive Moment Estimation (Adam)  
The adaptive moment estimation (Adam) technique intertwines ideas from root mean square propagation and momentum. It is a popular optimization algorithm with many applications in the domain of machine learning. The algorithm tracks simultaneously two moments: the first moment (i.e., the mean) and the second moment (i.e., the variance) [26]. The rationale behind this approach is that the first moment will assist in maintaining the direction by reducing the oscillations while moving along the landscape, and the second moment will assist in properly adapting the learning rates based on the previous gradient data. The update of the parameters will be handled employing gradients evaluated using both the first and second moments, according to [29]:  
\[\theta^{(k+1)}=\theta^{(k)}-\frac{\eta\cdot\hat{m}^{(k)}}{\sqrt{\hat{\phi}^{(k )}+\epsilon}} \tag{15}\]  
Here, \(\eta\) represents the learning rate, \(\hat{m}^{(k)}\) represents the first moment at the \(k\)-th iteration after bias correction, \(\hat{\phi}^{(k)}\) represents the second moment at the \(k\)-th iteration after bias correction, and \(\epsilon\) is a small constant utilized to avoid division by zero. The update principles are similar to the root mean squared method but are now applied twofold [30]:  
\[m^{(k+1)}=\beta_{1}\cdot m^{(k)}+(1-\beta_{1})\cdot g^{(k)}v^{(k+1)}=\beta_{2} \cdot v^{(k)}+(1-\beta_{2})\cdot\left(g^{(k)}\right)^{2} \tag{16}\]  
Furthermore, both moments are bias-corrected, yielding  
\[\hat{m}^{(k+1)}=\frac{m^{(k+1)}}{1-\beta_{1}^{\,1+1}}\hat{\phi}^{(k+1)}=\frac{ v^{(k+1)}}{1-\beta_{2}^{\,1+1}} \tag{17}\]When optimizing the modified objective functions based on adaptive moment estimation, the update principles employing two moments will be applied for both updating the membership values and the cluster centers. The primary structure of the algorithm remains the same, consisting of an iterative dual update on membership values and the cluster centers; however, the number of intermediate steps is significantly increased. The complete procedure is described by the following pseudo-code:  
1. Initialize the centers of the clusters, assigning as values random data instances.
2. Initialize the partition matrix (assigning 0 to all its entries).
3. Set \(k\ =\ 1\) (number of iterations).
4. Update the gradients of the memberships according to Equation (12).
5. Update the first and second moments for each membership value according to Equation (16).
6. Apply the bias correction for both moments of the membership values according to Equation (17).
7. Update the membership values, applying Equation (15) for \(\theta=\mu_{ij}\), so \[\mu_{ij}^{(k+1)}=\mu_{ij}^{(k)}-\frac{\eta\cdot\hat{\mu}_{\mu_{ij}^{(k+1)}}^{ (k+1)}}{\sqrt{\sigma_{\mu_{ij}}^{(k+1)}+\epsilon}}\]
8. Normalize the membership values according to Equation (9).
9. Update the gradients of the centers according to Equation (14).
10. Update the first and second moments of the centers according to Equation (16).
11. Apply the bias correction for both moments of the centers according to Equation (17).
12. Update the centers, applying Equation (15) for \(\theta=c_{ij}\), so \[c_{j}^{(k+1)}=c_{j}^{(k)}-\frac{\eta\cdot\hat{\mu}_{c_{j}^{(k+1)}}^{(k+1)}}{ \sqrt{\sigma_{c_{j}}^{(k+1)}+\epsilon}}\]
13. \(k\ =\ k+1\) (increment the number of iterations).
14. If \(\|\mathrm{U_{k}-U_{k-1}}\|>\mathit{Tol}\), jump to step 4.
15. END.