# Fuzzy Clustering Approaches Based on Numerical Optimizations of Modified Objective Functions
## 3 Optimization Based on Numerical Techniques
### Root Mean Square Propagation (RMSprop)  
Although gradient descent is one of the most widely used optimization techniques, especially in the context of machine learning, it may suffer from instabilities due to its sensitivity to the learning rate in complex landscapes of the objective functions and its sensitivity to noise. As gradient descent uses a fixed learning rate, this may affect the process when moving through a steep valley, causing oscillations through the sides of the valley or even divergent updates [26]. Overall, a constant learning rate is not appropriate for complex landscapes, as a small value of the learning rate would cause a slow convergence on a flat surface, while a relatively large learning rate would cause oscillations or divergent updates in a steep terrain [27]. The root mean square propagation method modifies the gradient descent method by scaling the learning rate based on the recent values of the gradients. Thus, the learning rate is dynamically adjusted to properly adapt to the current landscape. The weighted quadratic average of the previous gradients is evaluated and applied to scale the learning rate during the iterations, so for a parameter \(\theta\), the update would be controlled by the equation:  
\[\theta^{(k+1)}=\theta^{(k)}-\frac{\eta}{\sqrt{E[g^{2}]^{(k)}+\epsilon}}g^{(k)} \tag{11}\]  
Here, \(\eta\) represents the learning rate, \(g^{(0)}\) represents the gradient at the \(k\)-th iteration, \(\mathrm{E[g^{2}]^{(k)}}\) represents the weighted quadratic mean of the recent gradients, and \(\epsilon\) is a small constant utilized to avoid division by zero. This modification in the update rule makes the root mean square method more stable and particularly efficient in optimizing complex objective functions. Furthermore, it operates robustly in circumstances with noisy or fluctuating gradients, thus requiring significantly less tuning of the learning rate [28]. The root mean square method can be applied in optimizing the objective functions for fuzzy clustering, especially the more complex modification mentioned in the previous sections. So, the gradient of the membership values would be evaluated as  
\[g^{(k)}_{y_{ij}}=\frac{\partial\,\mathrm{J_{Modified}}}{\partial\,\mu_{ij}}= \mathrm{m}\cdot\mu_{ij}^{m-1}d^{2}\left(x_{i},\;c_{j}\right)+\lambda\cdot \frac{\partial\,\mathrm{R}}{\partial\,\mu_{ij}} \tag{12}\]  
Furthermore, the weighted quadratic average of the gradients would be evaluated (employing a decay factor \(\beta\), typically having a value close to 0.9) as  
\[E\left[g^{2}_{y_{ij}}\right]^{(k+1)}=\beta\cdot E\left[g^{2}_{y_{ij}}\right]^ {(k)}+(1-\beta)\cdot\left(g^{(t)}_{y_{ij}}\right)^{2} \tag{13}\]  
In a very similar way, the principle of the moving average characterizing the root mean squares method is applied for the update of the centers. The gradients of the cluster centers are evaluated based on the partial derivatives of the modified objective functions as  
\[g^{(k)}_{c_{j}}=\frac{\partial\,\mathrm{J_{Modified}}}{\partial\,c_{j}}=2 \left(\sum_{i=1}^{\mathrm{N}}\mu_{ij}^{m}\cdot(x_{i}-c_{j})\right)+\lambda \cdot\frac{\partial\,\mathrm{R}}{\partial\,c_{ij}} \tag{14}\]  
The modus operandi of the fuzzy clustering algorithm based on optimization of the objective functions via root mean square propagation will remain the same as the classical FCM, with the major differences relying on the way the dual updates are handled. The number of steps is naturally increased as there are several intermediate computations necessary for both the calculations of the membership values and the calculations of the new centers. The procedure can be summarized by the following pseudo-code:  
1. Initialize the centers of the clusters, assigning as values random data instances. 2. Initialize the partition matrix (assigning 0 to all its entries). 3. Set \(k\ =\ 1\) (number of iterations). 4. Update the gradients of the memberships according to Equation (12). 5. Update the weighted quadratic average according to Equation (13). 6.