Distributed Learning for Large-Scale Models at Edge With Privacy Protection

Big data and strong computing power have promoted artificial intelligence to the era of big models. In particular, ChatGPT's debut heralded the vigorous development of large models. It is an urgent problem to train large models with trillion-level parameters efficiently. Traditional single-machine training stores all data and model parameters in memory. However, due to the limitation of memory and communication resources, when the amount of data or model parameters increases, the problem of memory shortage and communication blocking often occurs. Therefore, distributed training is the most effective ways to solve the above problems and improve training efficiency. In this paper, we propose the algorithm <bold>DL-DP</bold>, which can achieve an asymptotically optimal convergence rate <inline-formula><tex-math notation="LaTeX">$O(1/{\sqrt{TK\Gamma^{*}}})$</tex-math><alternatives><mml:math><mml:mi>O</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mn>1</mml:mn><mml:mrow><mml:mo>/</mml:mo></mml:mrow><mml:mrow><mml:msqrt><mml:mi>T</mml:mi><mml:mi>K</mml:mi><mml:msup><mml:mi mathvariant="normal">Γ</mml:mi><mml:mrow><mml:mo>∗</mml:mo></mml:mrow></mml:msup></mml:msqrt></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:math><inline-graphic xlink:href="yuan-ieq1-3352814.gif"/></alternatives></inline-formula> while satisfying <inline-formula><tex-math notation="LaTeX">$\varepsilon$</tex-math><alternatives><mml:math><mml:mi>ε</mml:mi></mml:math><inline-graphic xlink:href="yuan-ieq2-3352814.gif"/></alternatives></inline-formula>-differential privacy, where <inline-formula><tex-math notation="LaTeX">$T$</tex-math><alternatives><mml:math><mml:mi>T</mml:mi></mml:math><inline-graphic xlink:href="yuan-ieq3-3352814.gif"/></alternatives></inline-formula> is the local epoch number, <inline-formula><tex-math notation="LaTeX">$K$</tex-math><alternatives><mml:math><mml:mi>K</mml:mi></mml:math><inline-graphic xlink:href="yuan-ieq4-3352814.gif"/></alternatives></inline-formula> is the global maximum iteration number and <inline-formula><tex-math notation="LaTeX">$\Gamma^{*}$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="normal">Γ</mml:mi><mml:mrow><mml:mo>∗</mml:mo></mml:mrow></mml:msup></mml:math><inline-graphic xlink:href="yuan-ieq5-3352814.gif"/></alternatives></inline-formula> is the minimum covering index. In particular, when <inline-formula><tex-math notation="LaTeX">$\Gamma^{*}=N$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="normal">Γ</mml:mi><mml:mrow><mml:mo>∗</mml:mo></mml:mrow></mml:msup><mml:mo>=</mml:mo><mml:mi>N</mml:mi></mml:math><inline-graphic xlink:href="yuan-ieq6-3352814.gif"/></alternatives></inline-formula>, DL-DP achieves a convergence rate of <inline-formula><tex-math notation="LaTeX">$O(1/\sqrt{TKN})$</tex-math><alternatives><mml:math><mml:mi>O</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mn>1</mml:mn><mml:mrow><mml:mo>/</mml:mo></mml:mrow><mml:msqrt><mml:mi>T</mml:mi><mml:mi>K</mml:mi><mml:mi>N</mml:mi></mml:msqrt><mml:mo stretchy="false">)</mml:mo></mml:math><inline-graphic xlink:href="yuan-ieq7-3352814.gif"/></alternatives></inline-formula>, which is equivalent to the best-known FedAvg approach implemented by training the full model at each client. When <inline-formula><tex-math notation="LaTeX">$\Gamma^{*}=1$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="normal">Γ</mml:mi><mml:mrow><mml:mo>∗</mml:mo></mml:mrow></mml:msup><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:math><inline-graphic xlink:href="yuan-ieq8-3352814.gif"/></alternatives></inline-formula>, DL-DP achieves a convergence rate of <inline-formula><tex-math notation="LaTeX">$O(1/\sqrt{TK})$</tex-math><alternatives><mml:math><mml:mi>O</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mn>1</mml:mn><mml:mrow><mml:mo>/</mml:mo></mml:mrow><mml:msqrt><mml:mi>T</mml:mi><mml:mi>K</mml:mi></mml:msqrt><mml:mo stretchy="false">)</mml:mo></mml:math><inline-graphic xlink:href="yuan-ieq9-3352814.gif"/></alternatives></inline-formula>, which is comparable to OAP that assumes all parameters need to be trained at least once in each iteration. Finally, our algorithm has been demonstrated to converge through extensive experiments.