The one-time clipping strategy employed in AClipped-dpSGD presents several potential limitations compared to the per-sample clipping used in DP-SGD.
Bias and Variance Trade-off: While the one-time clipping reduces the number of clipping operations and thus computational overhead, it may introduce a higher bias in the gradient estimation. In per-sample clipping, each gradient is clipped individually, which can lead to a more accurate representation of the true gradient, especially in the presence of outliers. The one-time clipping may not sufficiently mitigate the influence of extreme values, potentially leading to suboptimal convergence behavior.
Sensitivity to Clipping Level: The performance of the one-time clipping strategy is highly sensitive to the choice of the clipping level. If the clipping level is set too low, it may excessively bias the gradient estimates, while a high clipping level may not effectively control the influence of outliers. In contrast, per-sample clipping allows for more granular control over each individual gradient, potentially leading to better performance in diverse scenarios.
Convergence Guarantees: The convergence guarantees for the one-time clipping strategy may not be as robust as those for per-sample clipping. The theoretical analysis of AClipped-dpSGD must account for the accumulated bias introduced by the one-time clipping, which could complicate the establishment of high-probability bounds on excess population risk.
Implementation Complexity: Although the one-time clipping strategy simplifies the implementation by reducing the number of clipping operations, it may require more careful tuning of hyperparameters, such as the batch size and clipping level, to achieve optimal performance. This could lead to increased complexity in practical applications.