# Fuzzy Clustering Approaches Based on Numerical Optimizations of Modified Objective Functions
## 3 Optimization Based on Numerical Techniques
### Gradient Descent  
Gradient descent is one of the most widely used optimization techniques, especially in the context of machine learning. The fundamental principle of this technique is to iteratively update parameters of interest by tracking the direction of the steepest descent in the objective function that is intended to be minimized. The gradient (i.e., partial derivative) of the objective function with respect to parameters of interest is evaluated, and the parameter values are updated by moving in the opposite direction of the gradient [24].  
At each iteration, the algorithm calculates the gradient (the partial derivative) of the objective function with respect to each parameter of interest and updates the parameter values by moving in the opposite direction of the gradient. The learning rate (i.e., the step size) is a parameter that controls the size of each step. For smaller values of the learning rate, the convergence would be slower but would avoid overshooting the minimum. For larger values of the learning rate, the convergence would be reached faster, but there is an augmented risk of instability [25].  
The gradient descent technique is helpful in designating the iterative procedure for complex modifications of the objective function. The general form of the modified objective functions considered in this paper was  
\[J_{Modified}=J_{FCM}+\lambda R(U,c_{1},c_{2},\ldots,c_{C})=\sum_{i=1}^{n}\sum_ {j=1}^{C}u_{ij}^{m}d^{2}\big{(}x_{i},\ c_{j}\big{)}+\lambda R(U,c_{1},c_{2}, \ldots,c_{C}) \tag{7}\]  
Therefore, based on the gradient descent technique, the memberships of the data instances in the clusters are updated according to the equation:  
\[\mu_{ij}^{(k+1)}=\mu_{ij}^{k}-\eta\frac{\partial\mathrm{J_{Modified}}}{ \partial\,\mu_{ij}}=\mu_{ij}^{k}-\eta\cdot\mathrm{m}\cdot\mu_{ij}^{m-1}d^{2} \big{(}x_{i},\ c_{j}\big{)}-\eta\cdot\lambda\cdot\frac{\partial\,\mathrm{R}}{ \partial\,\mu_{ij}} \tag{8}\]  
Here, \(\mu_{ij}^{k}\) represents the membership value at the \(k\)-th iteration, \(\eta\) represents the learning rate, and \(\partial\) represents the partial derivative. Additionally, in order to guarantee that the constraint about the memberships being values in [0, 1] summing up to 1, an extra normalization step is carried out:  
\[\mu_{ij}^{(k+1)}=\frac{\mu_{ij}^{(k+1)}}{\sum\mu_{ij}^{(k+1)}} \tag{9}\]  
Furthermore, let us consider the distances being evaluated by the Euclidean distance measure, so  
\[d^{2}\big{(}x_{i},\ c_{j}\big{)}=\big{(}x_{i}-c_{j}\big{)}^{2}\]  
Then, the update of centers based on the gradient descent will be  
\[c_{j}^{(k+1)}=c_{j}^{(k)}-\eta\frac{\partial\mathrm{J_{Modified}}}{\partial\, c_{j}}=c_{i}^{(k)}-2\eta\cdot\sum_{i=1}^{N}\mu_{ij}^{m}\cdot\big{(}x_{i}-c_{j} \big{)}-\eta\cdot\lambda\cdot\frac{\partial\mathrm{R}}{\partial\,c_{ij}} \tag{10}\]  
The general modus operandi of the fuzzy clustering algorithm based on numerical optimizations of the modified objective function will be similar to the classical FCM algorithm, the main difference relying on the way the dual updates are handled. The procedure can be summarized by the following pseudo-code:  
1. Initialize the centers of the clusters, assigning as values random data instances.
2. Initialize the partition matrix (assigning 0 to all its entries).
3. Set k = 1 (number of iterations).
4. Update membership values according to Equation (8).
5. Normalize the membership values according to Equation (9).
6. Update the centers of the clusters according to Equation (10).
7. k = k + 1 (increment the number of iterations).
8. If \(\|\mathrm{U_{k}-U_{k-1}}\|>\mathit{Tol},\ \mathrm{jump}\) to step 4.
9. END.  
The given pseudo-code is generic, thus being applicable for every modification form of the objective function being employed. According to the specific type of modification, the details rely on the evaluation of the \(\frac{\partial\mathbf{R}}{\partial y_{ij}}\) and \(\frac{\partial\mathbf{R}}{\partial x_{ij}}\) terms.