When applied to large-scale federated learning problems with millions of parameters, the DZOFL method faces several challenges and limitations. One significant challenge is the computational burden associated with evaluating the loss function for each device. As the number of parameters increases, the time required for forward propagation to compute the scalar loss value also escalates, potentially leading to increased latency in the training process.
Another limitation is the potential for biased gradient estimates due to the zero-order optimization approach. The reliance on function evaluations rather than explicit gradient calculations can result in less accurate updates, particularly in high-dimensional spaces where the landscape of the loss function may be complex and non-convex. This bias can hinder convergence and affect the quality of the final model.
Additionally, the DZOFL method's performance may be adversely impacted by communication constraints, especially in scenarios with a high number of devices. While the method reduces the amount of data transmitted by sending only a scalar, the overhead associated with managing communication protocols and ensuring reliable transmission in a wireless environment can still pose significant challenges.
Lastly, the method's effectiveness may diminish in the presence of severe data heterogeneity or non-IID (Independent and Identically Distributed) data distributions across devices. The assumptions made in the convergence analysis may not hold in such cases, leading to suboptimal performance and convergence rates.