A method to train deep learning models, the method comprising: labelling, by a reference generator, by executing an instruction with at least one processor at a local device, input data received at the local device to generate training data; training, by executing an instruction with the at least one processor, a local deep learning model; characterized in transmitting the local deep learning model to a server, the server to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model by averaging corresponding weights received from the plurality of local devices, wherein the set of weights are aggregated weights based on the plurality of local deep learning models from the plurality of local devices; and updating, by an updater, by executing an instruction with the at least one processor at the local device, the local deep learning model by determining a weighted average of the averaged set of weights received from the server and weights of the local deep learning model; wherein the local device comprises a copy of the global deep learning model trained by the server; wherein training the local deep learning model includes: determining a first output of the copy of the global deep learning model for the input data; determining a second output of the local deep learning model for the input data; determining a first difference between the first output and the second output; determining a second difference between the second output and the label determined by the labelling for the input data; and training, by a trainer, the local deep learning model based on a third difference between the first difference and the second difference. labelling, by a reference generator, by executing an instruction with at least one processor at a local device, input data received at the local device to generate training data; training, by executing an instruction with the at least one processor, a local deep learning model; characterized in transmitting the local deep learning model to a server, the server to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model by averaging corresponding weights received from the plurality of local devices, wherein the set of weights are aggregated weights based on the plurality of local deep learning models from the plurality of local devices; and updating, by an updater, by executing an instruction with the at least one processor at the local device, the local deep learning model by determining a weighted average of the averaged set of weights received from the server and weights of the local deep learning model; wherein the local device comprises a copy of the global deep learning model trained by the server; determining a first output of the copy of the global deep learning model for the input data; determining a second output of the local deep learning model for the input data; determining a first difference between the first output and the second output; determining a second difference between the second output and the label determined by the labelling for the input data; and training, by a trainer, the local deep learning model based on a third difference between the first difference and the second difference.