Office Action 1: Notice of Allowance (2025-05-07)

CLAIMS:
CLAIMS

1\. A method comprising:receiving input data for processing by a transformer model;
applying a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises:
a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N, a transformation operation in the lower dimension, and
an up projection operation to reconstruct the data back to the higher dimension of size N;
processing the reconstructed data through the execution layer of the transformer model;
generating an encryption key for encrypting weights of the input privacy layer;
splitting the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1;
distributing the K shares among K distinct blockchain addresses;
encrypting the weights of the input privacy layer using the encryption key;
receiving a set of T signatures corresponding to T distinct blockchain addresses from the K addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold;
reconstructing the encryption key using the received T signatures and their corresponding shares through polynomial interpolation;
verifying the authenticity of each signature using the corresponding blockchain address's public key;
decrypting the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and
updating the transformer model by replacing the original weights of the input privacy layer with the decrypted weights.

2\. The method of claim 1, wherein positioning the input privacy layer horizontally between the input layer and the execution layer comprises:processing all input data through the input privacy layer before passing the reconstructed data to any subsequent layers of the transformer model.

3\. The method of claim 1, wherein the input privacy layer positioned horizontally between the input layer and the execution layer maintains the same input and output dimensionality as the input layer.

4\. The method of claim 1, wherein the input privacy layer positioned horizontally between the input layer and the execution layer is configured for:applying the privacy-preserving transformations uniformly across all input features;
processing batches of input data through the input privacy layer in parallel; and
separating the privacy-preserving computations from the main execution logic of the transformer model.

5\. The method of claim 1, wherein the input privacy layer further comprises a noise addition operation applied to the compressed data in the lower dimension, the noise addition operation configured to:generate Gaussian noise scaled based on a current privacy budget;
add the scaled noise to the compressed data; and
dynamically adjust the noise scale as the privacy budget is consumed during model training or inference.

6\. The method of claim 1, wherein the input privacy layer is configured to adaptively adjust the size M of the lower dimension based on:a current privacy budget;
a target accuracy threshold for the transformer model; and
a computational resource constraint,

wherein the size M is decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.

7\. The method of claim 1, further comprising:monitoring privacy metrics for the input privacy layer during training and inference, the privacy metrics including at least one of:a compression ratio between N and M;
a reconstruction error between the input data and the reconstructed data;
a privacy budget consumption rate; and
a noise magnitude added in the lower dimension;
adjusting hyperparameters of the input privacy layer based on the monitored privacy metrics to maintain a desired privacy-utility trade-off; and
storing the privacy metrics and corresponding model performance data to inform future input privacy layer configurations.

8\. The method of claim 1, wherein generating the encryption key comprises:generating a random seed using a cryptographically secure random number generator;
deriving the encryption key from the random seed using a key derivation function; and
storing metadata about the key generation process in a secure enclave.

9\. The method of claim 1, wherein splitting the encryption key into K shares uses Shamir's Secret Sharing scheme, and further comprises:selecting a polynomial of degree T-1, where T is the predefined threshold;
setting the constant term of the polynomial to be the encryption key;
generating K points on the polynomial to create the K shares; and
associating each share with a unique identifier.

10\. The method of claim 1, wherein distributing the K shares among K distinct blockchain addresses comprises:encrypting each share using the public key of its corresponding blockchain address;
transmitting the encrypted shares through a secure communication channel;
storing a mapping between share identifiers and blockchain addresses in a distributed ledger; and
implementing a time-lock mechanism that prevents share retrieval before a specified time.

11\. The method of claim 1, wherein reconstructing the encryption key comprises:verifying that the number of received signatures meets or exceeds the predefined threshold T;
validating each signature using the public key of the corresponding blockchain address;
retrieving the encrypted shares associated with the verified signatures;
decrypting the shares using the private keys of the participating blockchain addresses;
and applying Lagrange interpolation to the decrypted shares to reconstruct the original encryption key.

12\. The method of claim 1, further comprising:implementing a key rotation protocol for the encryption key, wherein:
a new encryption key is generated and split into shares at predefined intervals;
the new shares are distributed to the blockchain addresses;
a transition period is established where both the old and new encryption keys are valid;
the weights of the input privacy layer are re-encrypted using the new encryption key; and
the old encryption key and its shares are securely destroyed after the transition period.

13\. Th :method of claim 1, further comprising distributing operations across at least three distinct regions to comply with sovereign data requirements, the method further comprising:performing, in an on-premises environment:
storing and processing raw input data;
generating initial embeddings from the raw input data; and
applying preliminary privacy-preserving transformations to the initial embeddings;
xecuting, in a sovereign-compliant regional environment:
receiving the privacy-preserved embeddings from the on-premises environment;
applying the dimensionality reduction operation using the input privacy layer;
processing the reconstructed data through at least a portion of the execution layer;
generating the encryption key for the input privacy layer weights;
splitting the encryption key into K shares;
encrypting the weights of the input privacy layer; and
storing the encrypted weights and key shares;
nd conducting, in a separate cloud environment not controlled by the sovereign entity:
receiving the processed data from the sovereign-compliant regional environment;
completing any remaining execution layer processing;
distributing the K shares among K distinct blockchain addresses;
collecting the set of T signatures from the blockchain addresses;
reconstructing the encryption key using the received signatures and shares;
verifying the authenticity of each signature;
decrypting the weights of the input privacy layer; and
updating the transformer model with the decrypted weights.

14\. A ystem comprising:a processor; and

a memory storing instructions that, when executed by the processor, cause the system to:receive input data for processing by a transformer model;
apply a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises:
a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N,
a transformation operation in the lower dimension, and
an up projection operation to reconstruct the data back to the higher dimension of size N;
process the reconstructed data through the execution layer of the transformer model;
generate an encryption key for encrypting weights of the input privacy layer;
split the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1;
distribute the K shares among K distinct blockchain addresses;
encrypt the weights of the input privacy layer using the encryption key;
receive a set of T signatures corresponding to T distinct blockchain addresses from the K addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold;
reconstruct the encryption key using the received T signatures and their corresponding shares through polynomial interpolation;
verify the authenticity of each signature using the corresponding blockchain address's public key;
decrypt the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and
update the transformer model by replacing the original weights of the input privacy layer with the decrypted weights.

15\. The system of claim 14, wherein positioning the input privacy layer horizontally between the input layer and the execution layer comprises:

processing all input data through the input privacy layer before passing the reconstructed data to any subsequent layers of the transformer model.

16\. The system of claim 14, wherein the input privacy layer positioned horizontally between the input layer and the execution layer maintains the same input and output dimensionality as the input layer.

17\. The system of claim 14, wherein the input privacy layer positioned horizontally between the input layer and the execution layer is configured for:applying the privacy-preserving transformations uniformly across all input features;
processing batches of input data through the input privacy layer in parallel; and
separating the privacy-preserving computations from the main execution logic of the transformer model.

18\. The system of claim 14, wherein the input privacy layer further comprises a noise addition operation applied to the compressed data in the lower dimension, the noise addition operation configured to:generate Gaussian noise scaled based on a current privacy budget;
add the scaled noise to the compressed data; and
dynamically adjust the noise scale as the privacy budget is consumed during model training or inference.

19\. The system of claim 14, wherein the input privacy layer is configured to adaptively adjust the size M of the lower dimension based on:a current privacy budget;
a target accuracy threshold for the transformer model; and
a computational resource constraint,
wherein the size M is decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.

20\. The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to:monitor privacy metrics for the input privacy layer during training and inference, the privacy metrics including at least one of:
a compression ratio between N and M;
a reconstruction error between the input data and the reconstructed data;
a privacy budget consumption rate; and
a noise magnitude added in the lower dimension;
adjust hyperparameters of the input privacy layer based on the monitored privacy metrics to maintain a desired privacy-utility trade-off; and
store the privacy metrics and corresponding model performance data to inform future input privacy layer configurations.

21\. The system of claim 14, wherein generating the encryption key comprises:generating a random seed using a cryptographically secure random number generator;
deriving the encryption key from the random seed using a key derivation function; and
storing metadata about the key generation process in a secure enclave.

22\. The system of claim 14, wherein operations are distributed across at least three distinct regions to comply with sovereign data requirements, the system further comprising:performing, in an on-premises environment:
storing and processing raw input data;
generating initial embeddings from the raw input data; and
applying preliminary privacy-preserving transformations to the initial embeddings;
executing, in a sovereign-compliant regional environment:
receiving the privacy-preserved embeddings from the on-premises environment;
applying the dimensionality reduction operation using the input privacy layer;

processing the reconstructed data through at least a portion of the execution layer;
generating the encryption key for the input privacy layer weights;
splitting the encryption key into K shares;
encrypting the weights of the input privacy layer; and
storing the encrypted weights and key shares;
and conducting, in a separate cloud environment not controlled by the sovereign entity:receiving the processed data from the sovereign-compliant regional environment;
completing any remaining execution layer processing;
distributing the K shares among K distinct blockchain addresses;
collecting the set of T signatures from the blockchain addresses;
reconstructing the encryption key using the received signatures and shares;
verifying the authenticity of each signature;
decrypting the weights of the input privacy layer; and
updating the transformer model with the decrypted weights.

23\. The system of claim 14, wherein splitting the encryption key into K shares uses Shamir's Secret Sharing scheme, and further comprises:selecting a polynomial of degree T-1, where T is the predefined threshold;
setting the constant term of the polynomial to be the encryption key;
generating K points on the polynomial to create the K shares; and
associating each share with a unique identifier.

24\. The system of claim 14, wherein distributing the K shares among K distinct blockchain addresses comprises:encrypting each share using the public key of its corresponding blockchain address;
transmitting the encrypted shares through a secure communication channel;
storing a mapping between share identifiers and blockchain addresses in a distributed ledger; and

implementing a time-lock mechanism that prevents share retrieval before a specified time.

25\. The system of claim 14, wherein reconstructing the encryption key comprises:verifying that the number of received signatures meets or exceeds the predefined threshold T;
validating each signature using the public key of the corresponding blockchain address;
retrieving the encrypted shares associated with the verified signatures;
decrypting the shares using the private keys of the participating blockchain addresses; and
applying Lagrange interpolation to the decrypted shares to reconstruct the original encryption key.

26\. The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to:implement a key rotation protocol for the encryption key, wherein:
a new encryption key is generated and split into shares at predefined intervals;
the new shares are distributed to the blockchain addresses;
a transition period is established where both the old and new encryption keys are valid;
the weights of the input privacy layer are re-encrypted using the new encryption key; and
the old encryption key and its shares are securely destroyed after the transition period.

27\. The system of claim 14, further comprising:a network interface configured to communicate with the blockchain addresses; and
a secure hardware element configured to store cryptographic keys and perform cryptographic operations.

28\. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:receiving input data for processing by a transformer model;
applying a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises:
a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N,
a transformation operation in the lower dimension, and
an up projection operation to reconstruct the data back to the higher dimension of size N;
processing the reconstructed data through the execution layer of the transformer model;
generating an encryption key for encrypting weights of the input privacy layer;
splitting the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1;
distributing the K shares among K distinct blockchain addresses;
encrypting the weights of the input privacy layer using the encryption key;
receiving a set of T signatures corresponding to T distinct blockchain addresses from the K addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold;
reconstructing the encryption key using the received T signatures and their corresponding shares through polynomial interpolation;
verifying the authenticity of each signature using the corresponding blockchain address's public key;
decrypting the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and
updating the transformer model by replacing the original weights of the input privacy layer with the decrypted weights.

29\. The non-transitory computer-readable medium of claim 28, wherein the operations further comprise:monitoring privacy metrics for the input privacy layer during training and inference, the privacy metrics including at least one of:
a compression ratio between N and M;
a reconstruction error between the input data and the reconstructed data;
a privacy budget consumption rate; and
a noise magnitude added in the lower dimension;
adjusting hyperparameters of the input privacy layer based on the monitored privacy metrics to maintain a desired privacy-utility trade-off; and
storing the privacy metrics and corresponding model performance data to inform future input privacy layer configurations.

30\. The non-transitory computer-readable medium of claim 28, wherein the input privacy layer is configured to adaptively adjust the size M of the lower dimension based on:a current privacy budget;
a target accuracy threshold for the transformer model; and
a computational resource constraint,
wherein the size M may be decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.

OFFICE ACTION:
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2016, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office Action is in response to the application 19/028,234 filed on 01/17/2025. Claims 1-17, 20-25, 28 and 30 have been amended.  Claims 1-30 have been examined and are pending.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 01/25/2025 is compliance with the provisions of 37 CFR 1.97.  Accordingly, the information disclosure statements have been considered by the examiner.
Examiner’s Amendments
Authorization for this Examiner’s Amendment was made via phone with the Applicant’s representative, James Denaro (Reg. No.: 54,063). James Denaro has agreed and authorized the Examiner to amend claims 1-17, 20-25, 28 and 30.
Claims
Replacing claims 1-30 as follows:
1. 	(Currently Amended) A computerized method for implementing a privacy-preserving transformer model with secure key management comprising:
receiving input data for processing by a transformer model from a computerized storage device;
using a computerized processor, applying a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises:
a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N,
a transformation operation in the lower dimension, and
an up projection operation to reconstruct the input data back to the higher dimension of size N;
wherein the input privacy layer positioned horizontally between the input layer and the execution layer is configured for:           applying privacy-preserving transformations across a plurality of input features;           processing one or more batches of input data through the input privacy layer in parallel, and            separating the privacy-preserving computations from main execution logic of the transformer model,
processing the reconstructed input data through at least a portion of the execution layer of the transformer model;
generating an encryption key for encrypting weights of the input privacy layer using a cryptographically secure random number generator;
splitting the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1;
distributing the K shares among K distinct blockchain addresses;
encrypting individual key shares of the K shares using public keys of their corresponding blockchain addresses before transmission;
encrypting the weights of the input privacy layer using the encryption key;
receiving a set of T signatures corresponding to T distinct blockchain addresses from the K distinct blockchain addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold;
reconstructing the encryption key using the received T signatures and their corresponding shares through polynomial interpolation;
verifying the authenticity of each signature of the T signatures using a corresponding blockchain address's public key;
decrypting the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and
updating the transformer model by replacing original weights of the input privacy layer with the decrypted weights; and
protecting against tampering of the transformer model by utilizing the secret sharing scheme and the blockchain addresses.
2. (Currently Amended) The computerized method of claim 1, wherein positioning the input privacy layer horizontally between the input layer and the execution layer comprises:
processing all input data through the input privacy layer before passing the reconstructed data to any subsequent layers of the transformer model.
3. (Currently Amended) The computerized method of claim 1, wherein the input privacy layer positioned horizontally between the input layer and the execution layer maintains the same input and output dimensionality as the input layer.
4. (Currently Amended) The computerized method of claim 1, wherein the input privacy layer to apply the privacy-preserving transformations uniformly across all input features .
5. (Currently Amended) The computerized method of claim 1, wherein the input privacy layer further comprises a noise addition operation applied to the compressed data in the lower dimension, the noise addition operation configured to:
generate Gaussian noise scaled based on a current privacy budget;
add the scaled noise to the compressed data; and
dynamically adjust the noise scale as the privacy budget is consumed during model training or inference.
6. (Currently Amended) The computerized method of claim 1, wherein the input privacy layer is configured to adaptively adjust the size M of the lower dimension based on:
a current privacy budget;
a target accuracy threshold for the transformer model; and
a computational resource constraint,
wherein the size M is decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.
7. (Currently Amended) The computerized method of claim 1, further comprising:
monitoring privacy metrics for the input privacy layer during training and inference, the privacy metrics including at least one of:
a compression ratio between N and M;
a reconstruction error between the input data and the reconstructed data;
a privacy budget consumption rate; and
a noise magnitude added in the lower dimension;
adjusting hyperparameters of the input privacy layer based on the monitored privacy metrics to maintain a desired privacy-utility trade-off; and
storing the privacy metrics and corresponding model performance data to inform future input privacy layer configurations.
8. (Currently Amended) The computerized method of claim 1, wherein generating the encryption key further comprises:
deriving the encryption key from a random seed using a key derivation function; and
storing metadata about the key generation process in a secure enclave.
9. (Currently Amended) The computerized method of claim 1, wherein splitting the encryption key into K shares uses Shamir's Secret Sharing scheme, and further comprises:
selecting a polynomial of degree T-1, where T is the predefined threshold;
setting the constant term of the polynomial to be the encryption key;
generating K points on the polynomial to create the K shares; and
associating each share with a unique identifier.
10. (Currently Amended) The computerized method of claim 1, wherein distributing the K shares among K distinct blockchain addresses further comprises:
transmitting the encrypted shares through a secure communication channel;
storing a mapping between share identifiers and blockchain addresses in a distributed ledger; and
implementing a time-lock mechanism that prevents share retrieval before a specified time.
11. (Currently Amended) The computerized method of claim 1, wherein reconstructing the encryption key comprises:
verifying that the number of received signatures meets or exceeds the predefined threshold T;
validating each signature using the public key of the corresponding blockchain address;
retrieving the encrypted shares associated with the verified signatures;
decrypting the shares using the private keys of the participating blockchain addresses; and
applying Lagrange interpolation to the decrypted shares to reconstruct the original encryption key.
​
12. (Currently Amended) The computerized method of claim 1, further comprising:
implementing a key rotation protocol for the encryption key, wherein:
a new encryption key is generated and split into shares at predefined intervals;
the new shares are distributed to the blockchain addresses;
a transition period is established where both the old and new encryption keys are valid;
the weights of the input privacy layer are re-encrypted using the new encryption key; and
the old encryption key and its shares are securely destroyed after the transition period.
​
13. (Currently Amended) The computerized method of claim 1, further comprising distributing operations across at least three distinct regions to comply with sovereign data requirements, the method further comprising:
performing, in an on-premises environment:
storing and processing raw input data;
generating initial embeddings from the raw input data; and
applying preliminary privacy-preserving transformations to the initial embeddings;
executing, in a sovereign-compliant regional environment:
receiving the privacy-preserved embeddings from the on-premises environment;
applying the dimensionality reduction operation using the input privacy layer;
processing the reconstructed data through at least a portion of the execution layer;
generating the encryption key for the input privacy layer weights;
splitting the encryption key into K shares;
encrypting the weights of the input privacy layer; and
storing the encrypted weights and key shares;
and conducting, in a separate cloud environment not controlled by the sovereign entity:
receiving the processed data from the sovereign-compliant regional environment;
completing any remaining execution layer processing;
distributing the K shares among K distinct blockchain addresses;
collecting the set of T signatures from the blockchain addresses;
reconstructing the encryption key using the received signatures and shares;
verifying the authenticity of each signature;
decrypting the weights of the input privacy layer; and
updating the transformer model with the decrypted weights.
14. (Currently Amended) A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, cause the system to:
receive input data for processing by a transformer model from a computerized storage device;
apply a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises:
a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N,
a transformation operation in the lower dimension, and
an up projection operation to reconstruct the input data back to the higher dimension of size N;
wherein the input privacy layer positioned horizontally between the input layer and the execution layer is configured for:
applying privacy-preserving transformations across a plurality of input features;
processing one or more batches of input data through the input privacy layer in parallel, and
separating the privacy-preserving computations from main execution logic of the transformer model,
process the reconstructed data through the execution layer of the transformer model;
generate an encryption key for encrypting weights of the input privacy layer using a cryptographically secure random number generator;
split the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1;
distribute the K shares among K distinct blockchain addresses;
encrypt individual key shares of the K shares using public keys of their corresponding blockchain addresses before transmission;
encrypt the weights of the input privacy layer using the encryption key;
receive a set of T signatures corresponding to T distinct blockchain addresses from the K distinct blockchain addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold;
reconstruct the encryption key using the received T signatures and their corresponding shares through polynomial interpolation;
verify the authenticity of each signature using the corresponding blockchain address's public key;
decrypt the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and
update the transformer model by replacing ; and
protect against tampering of the transformer model by utilizing the secret sharing scheme and the blockchain addresses.
15. (Currently Amended) The system of claim 14, wherein
the memory stores further instructions that, when executed by the processor, cause the system to process all input data through the input privacy layer before passing the reconstructed data to any subsequent layers of the transformer model.
16. (Original) The system of claim 14, wherein the input privacy layer positioned horizontally between the input layer and the execution layer maintains the same input and output dimensionality as the input layer.
17. (Currently Amended) The system of claim 14, wherein the
memory stores further instructions that, when executed by the processor, cause the system to apply the privacy-preserving transformations uniformly across all input features .
18. (Original) The system of claim 14, wherein the input privacy layer further comprises a noise addition operation applied to the compressed data in the lower dimension, the noise addition operation configured to:
generate Gaussian noise scaled based on a current privacy budget;
add the scaled noise to the compressed data; and
dynamically adjust the noise scale as the privacy budget is consumed during model training or inference.
19. (Original) The system of claim 14, wherein the input privacy layer is configured to adaptively adjust the size M of the lower dimension based on:
a current privacy budget;
a target accuracy threshold for the transformer model; and
a computational resource constraint,
wherein the size M is decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.
20. (Original) The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to:
monitor privacy metrics for the input privacy layer during training and inference, the privacy metrics including at least one of:
a compression ratio between N and M;
a reconstruction error between the input data and the reconstructed data;
a privacy budget consumption rate; and
a noise magnitude added in the lower dimension;
adjust hyperparameters of the input privacy layer based on the monitored privacy metrics to maintain a desired privacy-utility trade-off; and
store the privacy metrics and corresponding model performance data to inform future input privacy layer configurations.
21. (Currently Amended) The system of claim 14, wherein memory stores further instructions that, when executed by the processor, cause the system to:
generate a random seed using the cryptographically secure random number generator;
derive the encryption key from the random seed using a key derivation function; and
store metadata about the key generation process in a secure enclave.
22. (Currently Amended) The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to:
perform, in an on-premises environment:
storing and processing raw input data;
generating initial embeddings from the raw input data; and
applying preliminary privacy-preserving transformations to the initial embeddings;
execute, in a sovereign-compliant regional environment:
receiving the privacy-preserved embeddings from the on-premises environment;
applying the dimensionality reduction operation using the input privacy layer;
processing the reconstructed data through at least a portion of the execution layer;
generating the encryption key for the input privacy layer weights;
splitting the encryption key into K shares;
encrypting the weights of the input privacy layer; and
storing the encrypted weights and key shares;
and conduct, in a separate cloud environment not controlled by the sovereign entity:
receiving the processed data from the sovereign-compliant regional environment;
completing any remaining execution layer processing;
distributing the K shares among K distinct blockchain addresses;
collecting the set of T signatures from the blockchain addresses;
reconstructing the encryption key using the received signatures and shares;
verifying the authenticity of each signature;
decrypting the weights of the input privacy layer; and
updating the transformer model with the decrypted weights.
23. (Currently Amended) The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to split the encryption key using Shamir's Secret Sharing schemeby:
selecting a polynomial of degree T-1, where T is the predefined threshold;
setting the constant term of the polynomial to be the encryption key;
generating K points on the polynomial to create the K shares; and
associating each share with a unique identifier.
24. (Currently Amended) The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to distribute the K shares among K distinct blockchain addresses by:
transmitting the encrypted shares through a secure communication channel;
storing a mapping between share identifiers and blockchain addresses in a distributed ledger; and
implementing a time-lock mechanism that prevents share retrieval before a specified time.
25. (Currently Amended) The system of claim 14, wherein memory stores further instructions that, when executed by the processor, cause the system to reconstruct the encryption key by:
verifying that the number of received signatures meets or exceeds the predefined threshold T;
validating each signature using the public key of the corresponding blockchain address;
retrieving the encrypted shares associated with the verified signatures;
decrypting the shares using the private keys of the participating blockchain addresses; and
applying Lagrange interpolation to the decrypted shares to reconstruct the original encryption key.
26. (Original) The system of claim 14, wherein the memory stores further instructions that, when executed by the processor, cause the system to:
implement a key rotation protocol for the encryption key, wherein:
a new encryption key is generated and split into shares at predefined intervals;
the new shares are distributed to the blockchain addresses;
a transition period is established where both the old and new encryption keys are valid;
the weights of the input privacy layer are re-encrypted using the new encryption key; and
the old encryption key and its shares are securely destroyed after the transition period.
27. (Original) The system of claim 14, further comprising:
a network interface configured to communicate with the blockchain addresses; and
a secure hardware element configured to store cryptographic keys and perform cryptographic operations.
28. (Currently Amended) A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving input data for processing by a transformer model from a computerized storage device;
applying a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises:
a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N,
a transformation operation in the lower dimension, and
an up projection operation to reconstruct the input data back to the higher dimension of size N;
wherein the input privacy layer positioned horizontally between the input layer and the execution layer is configured for:
applying privacy-preserving transformations across a plurality of input features;
processing one or more batches of input data through the input privacy layer in parallel, and
separating the privacy-preserving computations from main execution logic of the transformer model,
processing the reconstructed data through the execution layer of the transformer model;
generating an encryption key for encrypting weights of the input privacy layer using a cryptographically secure random number generator;
splitting the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1;
distributing the K shares among K distinct blockchain addresses;
encrypting individual key shares of the K shares using public keys of their corresponding blockchain addresses before transmission;
encrypting the weights of the input privacy layer using the encryption key;
receiving a set of T signatures corresponding to T distinct blockchain addresses from the K distinct blockchain addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold;
reconstructing the encryption key using the received T signatures and their corresponding shares through polynomial interpolation;
verifying the authenticity of each signature using the corresponding blockchain address's public key;
decrypting the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and
updating the transformer model by replacing original weights of the input privacy layer with the decrypted weights ; and
protecting against tampering of the transformer model by utilizing the secret sharing scheme and the blockchain addresses.
29. (Original) The non-transitory computer-readable medium of claim 28, wherein the operations further comprise:
monitoring privacy metrics for the input privacy layer during training and inference, the privacy metrics including at least one of:
a compression ratio between N and M;
a reconstruction error between the input data and the reconstructed data;
a privacy budget consumption rate; and
a noise magnitude added in the lower dimension;
adjusting hyperparameters of the input privacy layer based on the monitored privacy metrics to maintain a desired privacy-utility trade-off; and
storing the privacy metrics and corresponding model performance data to inform future input privacy layer configurations.
30. (Currently Amended)  The non-transitory computer-readable medium of claim 28, wherein the input privacy layer is configured to adaptively adjust the size M of the lower dimension based on:
a current privacy budget;
a target accuracy threshold for the transformer model; and
a computational resource constraint,
wherein the size M is decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.
Examiner’s Statement of Reasons for Allowance
Claims 1-30 are allowed.
The following is an examiner’s statement of reasons for allowance.
The invention is directed to a method that includes receiving input data for processing by a transformer model. The method applies a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model. The input privacy layer comprises a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N, a transformation operation in the lower dimension, and an up projection operation to reconstruct the data back to the higher dimension of size N. The method processes the reconstructed data through the execution layer of the transformer model. The method generates an encryption key for encrypting weights of the input privacy layer. The method splits the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1. The method distributes the K shares among K distinct blockchain addresses. The method encrypts the weights of the input privacy layer using the encryption key. The method receives a set of T signatures corresponding to T distinct blockchain addresses from the K addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold. The method reconstructs the encryption key using the received T signatures and their corresponding shares through polynomial interpolation. The method verifies the authenticity of each signature using the corresponding blockchain address's public key. The method decrypts the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified. The method updates the transformer model by replacing the original weights of the input privacy layer with the decrypted weights.
According to other aspects of the present disclosure, the method may include one or more of the following features. Positioning the input privacy layer horizontally between the input layer and the execution layer may comprise processing all input data through the input privacy layer before passing the reconstructed data to any subsequent layers of the transformer model. The input privacy layer positioned horizontally between the input layer and the execution layer may maintain the same input and output dimensionality as the input layer. The input privacy layer positioned horizontally between the input layer and the execution layer may be configured for applying the privacy-preserving transformations uniformly across all input features, processing batches of input data through the input privacy layer in parallel, and separating the privacy- preserving computations from the main execution logic of the transformer model. The input privacy layer may further comprise a noise addition operation applied to the compressed data in the lower dimension, the noise addition operation configured to generate Gaussian noise scaled based on a current privacy budget, add the scaled noise to the compressed data, and dynamically adjust the noise scale as the privacy budget is consumed during model training or inference. The input privacy layer may be configured to adaptively adjust the size M of the lower dimension based on a current privacy budget, a target accuracy threshold for the transformer model, and a computational resource constraint, wherein the size M is decreased to enhance privacy when the privacy budget is low or the accuracy is above the target threshold, and increased to preserve more information when the privacy budget is high or the accuracy falls below the target threshold.
Distributing the K shares among K distinct blockchain addresses may comprise
encrypting each share using the public key of its corresponding blockchain address, transmitting the encrypted shares through a secure communication channel, storing a mapping between share identifiers and blockchain addresses in a distributed ledger, and implementing a time-lock mechanism that prevents share retrieval before a specified time. Reconstructing the encryption key may comprise verifying that the number of received signatures meets or exceeds the predefined threshold T, validating each signature using the public key of the corresponding blockchain address, retrieving the encrypted shares associated with the verified signatures, decrypting the shares using the private keys of the participating blockchain addresses, and applying Lagrange interpolation to the decrypted shares to reconstruct the original encryption key. The method may further comprise implementing a key rotation protocol for the encryption key, wherein a new encryption key is generated and split into shares at predefined intervals, the new shares are distributed to the blockchain addresses, a transition period is established where both the old and new encryption keys are valid, the weights of the input privacy layer are re-encrypted using the new encryption key, and the old encryption key and its shares are securely destroyed after the transition period.
The closest prior art are Beaufays et al. (“Beaufays,” US 20220270590, published Aug. 25, 2022), Nolan et al. (“Nolan,” US 20190034919, published Jan. 31, 2019) and Zeng et al. (“Zeng,” 2024, Secure GPT, A Framework for Multi-Party Privacy-Perserving Transformer Inference in GPT, IEEE Transactions on Information Forensics and Security, vol. 19, pp 9482-9484, 9487), which are also generally directed to various aspects of privacy-preserving transformer model training using secure encryption and key split with blockchain verification.
However, none of Beaufays, Nolan and Zeng, alone or in combination, the particular combination of steps or elements as recited in the independent claim 1. For example, these references fails to teach all limitations recited in claim 1 as a whole, especially “A computerized method for implementing a privacy-preserving transformer model with secure key management comprising: receiving input data for processing by a transformer model from a computerized storage device; using a computerized processor, applying a dimensionality reduction operation to the input data using an input privacy layer positioned horizontally between an input layer and an execution layer of the transformer model, wherein the input privacy layer comprises: a down projection operation to compress the input data from a higher dimension of size N to a lower dimension of size M, where M < N, a transformation operation in the lower dimension, and an up projection operation to reconstruct the input data back to the higher dimension of size N; wherein the input privacy layer positioned horizontally between the input layer and the execution layer is configured for: applying privacy-preserving transformations across a plurality of input features; processing one or more batches of input data through the input privacy layer in parallel, and separating the privacy-preserving computations from main execution logic of the transformer model, processing the reconstructed input data through at least a portion of the execution layer of the transformer model; generating an encryption key for encrypting weights of the input privacy layer using a cryptographically secure random number generator; splitting the encryption key into K shares using a secret sharing scheme, where K is an integer greater than 1; distributing the K shares among K distinct blockchain addresses; encrypting individual key shares of the K shares using public keys of their corresponding blockchain addresses before transmission; encrypting the weights of the input privacy layer using the encryption key; receiving a set of T signatures corresponding to T distinct blockchain addresses from the K distinct blockchain addresses, where T is an integer less than or equal to K and greater than or equal to a predefined threshold; reconstructing the encryption key using the received T signatures and their corresponding shares through polynomial interpolation; verifying the authenticity of each signature of the T signatures using the a corresponding blockchain address's public key; decrypting the weights of the input privacy layer using the reconstructed encryption key only if all T signatures are successfully verified; and updating the transformer model by replacing original weights of the input privacy layer with the decrypted weights; and protecting against tampering of the transformer model by utilizing the secret sharing scheme and the blockchain addresses.”
These features in light of other features described in the independent claim 1 is allowable over the prior art of record.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to void processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion



CITED REFERENCE SUMMARIES: Ref US20220270590A1: [content unavailable]
Ref US20190034919A1: Claims:
An apparatus for securing electronic wallet transactions, comprising: an e-wallet application configured to obtain parameters for an e-wallet transaction from a user; a transaction generator configured to: sign a first transaction using a first private key and verify the first transaction using a corresponding first public key; encrypt the first transaction to form a second transaction; and sign the second transaction using a second private key and verify the second transaction using a corresponding second public key.
an e-wallet application configured to obtain parameters for an e-wallet transaction from a user;
a transaction generator configured to: sign a first transaction using a first private key and verify the first transaction using a corresponding first public key; encrypt the first transaction to form a second transaction; and sign the second transaction using a second private key and verify the second transaction using a corresponding second public key.
sign a first transaction using a first private key and verify the first transaction using a corresponding first public key;
encrypt the first transaction to form a second transaction; and
sign the second transaction using a second private key and verify the second transaction using a corresponding second public key.
The apparatus of claim 1 , comprising a trusted platform module (TPM) to create a trusted execute environment (TEE) in which the e-wallet application operates.
The apparatus of claim 1 , comprising a blockchain to record transactions from the transaction generator.
The apparatus of claim 1 , comprising an insurance sidechain to record contracts for compensating transactions.
The apparatus of claim 1 , wherein the e-wallet application comprises an application programming interface (API) configured to send a transaction to a web farm comprising a network of temporal distributed wallets.
The apparatus of claim 1 , comprising a touchscreen display, wherein the e-wallet application is configured to display operations of the e-wallet application, and obtain input from a user of the e-wallet application.
A machine implemented method for securing e-wallet transactions, comprising: creating a transaction in a device hosting a first e-wallet; signing the transaction with a first wallet key; and signing the transaction with a subsequent wallet key to create a subsequent transaction.
creating a transaction in a device hosting a first e-wallet;
signing the transaction with a first wallet key; and
signing the transaction with a subsequent wallet key to create a subsequent transaction.
The machine implemented method of claim 7 , comprising committing the subsequent transaction to a blockchain after signing with the subsequent wallet key.
The machine implemented method of claim 7 , comprising sequentially signing the subsequent transaction with a plurality of wallet keys to create a final transaction.
The machine implemented method of claim 7 , comprising sending the subsequent transaction to a web farm hosting a plurality of distributed wallets to be encrypted using keys for the plurality of distributed wallets.
The machine implemented method of claim 10 , comprising determining whether the subsequent transaction is to be submitted to the web farm, based, at least in part, on a policy that determines if the subsequent transaction is of sufficient value.
The machine implemented method of claim 10 , comprising translating a message syntax from a first syntax to a second syntax.
The machine implemented method of claim 7 , comprising requesting a compensating transaction for the subsequent transaction by making a copy of the subsequent transaction on a blockchain, and posting the copy to a sidechain.
The machine implemented method of claim 13 , comprising: detecting an insurer's offer on the sidechain, to provide insurance for the subsequent transaction; counter signing the insurer's offer to form an insurance contract transaction; and posting the insurance contract transaction to the blockchain.
detecting an insurer's offer on the sidechain, to provide insurance for the subsequent transaction;
counter signing the insurer's offer to form an insurance contract transaction; and
posting the insurance contract transaction to the blockchain.
The machine implemented method of claim 14 , comprising: creating a claim against the insurance; and posting the claim to the blockchain.
creating a claim against the insurance; and
posting the claim to the blockchain.
The machine implemented method of claim 15 , comprising receiving a compensating transaction for the subsequent transaction from the insurance.
A non-transitory, machine readable medium, comprising code that, when executed, directs a processor to: create a transaction; sign the transaction with a first key; encrypt the transaction to form an encrypted transaction; and sign the encrypted transaction with a second key.
create a transaction;
sign the transaction with a first key;
encrypt the transaction to form an encrypted transaction; and
sign the encrypted transaction with a second key.
The non-transitory, machine readable medium of claim 17 , comprising code that, when executed, directs the processor to: send the encrypted transaction to a web farm hosting multiple-distributed e-wallets; and request the web farm further encrypt the encrypted transaction using keys for the multiple-distributed e-wallets.
send the encrypted transaction to a web farm hosting multiple-distributed e-wallets; and
request the web farm further encrypt the encrypted transaction using keys for the multiple-distributed e-wallets.
The non-transitory, machine readable medium of claim 17 , comprising code that, when executed, directs the processor to request an offer for a compensating transaction from an insurer for the encrypted transaction.
The non-transitory, machine readable medium of claim 19 , comprising code that, when executed, directs the processor to accept an offer for the compensating transaction from the insurer.
The non-transitory, machine readable medium of claim 20 , comprising code that, when executed, directs the processor to make a claim for the compensating transaction from the insurer.
An apparatus for securing blockchain transactions, comprising: an e-wallet application configured to obtain parameters for an e-wallet transaction from a user; and a means for signing the e-wallet transaction by a plurality of e-wallet shares to create a multi-signed transaction.
an e-wallet application configured to obtain parameters for an e-wallet transaction from a user; and
a means for signing the e-wallet transaction by a plurality of e-wallet shares to create a multi-signed transaction.
The apparatus of claim 22 , comprising a means to record transactions.
The apparatus of claim 22 , comprising a means to record compensating transactions.
The apparatus of claim 22 , wherein the e-wallet application comprises a means to use a network of temporal distributed wallets to encrypt a transaction.
Ref US20240388442A1: Claims:
A computerized method of performing secure exchanges of data over a computer network, the method comprising, using one or more computer processors: generating, by a computerized blockchain entity, a plurality of data points of the blockchain entity, wherein the generating of data points of the blockchain entity is performed based on a data point from a first computerized signing entity of one or more computerized signing entities; reconstructing, by the blockchain entity, one or more data points of one or more of the signing entities, the data points of the signing entities generated based on one or more of the data points of the blockchain entity; and performing an exchange of data between one or more of: the blockchain entity, and one or more of the signing entities based on one or more of the reconstructed data points.
generating, by a computerized blockchain entity, a plurality of data points of the blockchain entity, wherein the generating of data points of the blockchain entity is performed based on a data point from a first computerized signing entity of one or more computerized signing entities;
reconstructing, by the blockchain entity, one or more data points of one or more of the signing entities, the data points of the signing entities generated based on one or more of the data points of the blockchain entity; and
performing an exchange of data between one or more of: the blockchain entity, and one or more of the signing entities based on one or more of the reconstructed data points.
The computerized method of claim 1 , comprising: broadcasting one or more of the reconstructed data points, the broadcasting using a public component of the blockchain entity.
The computerized method of claim 1 , comprising receiving the data point from the first computerized signing entity using a private component of the blockchain entity, wherein the generating of data points of the blockchain entity is performed using the private component.
The computerized method of claim 1 , comprising: calculating, by one or more of the signing entities, a plurality of public key shares based on a secret key share of the first signing entity and a secret key share of the blockchain entity; and resolving, by the blockchain entity, one or more of the public key shares, wherein at least one of: the generating of data points of the blockchain entity, the reconstructing of data points of the signing entities, and the performing of an exchange of data are performed based on the resolved public key shares.
calculating, by one or more of the signing entities, a plurality of public key shares based on a secret key share of the first signing entity and a secret key share of the blockchain entity; and
resolving, by the blockchain entity, one or more of the public key shares, wherein at least one of: the generating of data points of the blockchain entity, the reconstructing of data points of the signing entities, and the performing of an exchange of data are performed based on the resolved public key shares.
The computerized method of claim 1 , comprising simulating, by the blockchain entity, one or more computerized operations, the simulated operations associated with one or more of the data points of the signing entities.
The computerized method of claim 1 , wherein the blockchain entity is implemented on a blockchain network.
The computerized method of claim 6 , wherein the exchange of data comprises excluding the first entity from the blockchain network.
The computerized method of claim 5 , wherein the simulated operations comprise one or more pseudorandom values, and wherein the simulated operations describe an adversarial computerized behavior by one or more of the signing entities.
The computerized method of claim 4 , wherein the public key shares are generated using an elliptic curve algorithm.
The computerized method of claim 6 , wherein the blockchain network comprises an Ethereum blockchain.
A computerized system for performing secure exchanges of data over a computer network, the system comprising: a memory; and one or more processors configured to: generate, by a computerized blockchain entity, a plurality of data points of the blockchain entity, wherein the generating of data points of the blockchain entity is performed based on a data point from a first computerized signing entity of one or more computerized signing entities; reconstruct, by the blockchain entity, one or more data points of one or more of the signing entities, the data points of the signing entities generated based on one or more of the data points of the blockchain entity; and perform of an exchange of data between one or more of: the blockchain entity, and one or more of the signing entities based on one or more of the reconstructed data points.
one or more processors configured to: generate, by a computerized blockchain entity, a plurality of data points of the blockchain entity, wherein the generating of data points of the blockchain entity is performed based on a data point from a first computerized signing entity of one or more computerized signing entities; reconstruct, by the blockchain entity, one or more data points of one or more of the signing entities, the data points of the signing entities generated based on one or more of the data points of the blockchain entity; and perform of an exchange of data between one or more of: the blockchain entity, and one or more of the signing entities based on one or more of the reconstructed data points.
generate, by a computerized blockchain entity, a plurality of data points of the blockchain entity, wherein the generating of data points of the blockchain entity is performed based on a data point from a first computerized signing entity of one or more computerized signing entities;
reconstruct, by the blockchain entity, one or more data points of one or more of the signing entities, the data points of the signing entities generated based on one or more of the data points of the blockchain entity; and
perform of an exchange of data between one or more of: the blockchain entity, and one or more of the signing entities based on one or more of the reconstructed data points.
The computerized system of claim 1 , wherein one or more of the processors are to broadcast one or more of the reconstructed data points, the broadcasting using a public component of the blockchain entity.
The computerized system of claim 1 , wherein one or more of the processors are to receive the data point from the first computerized signing entity using a private component of the blockchain entity, and wherein the generating of data points of the blockchain entity is performed using the private component.
The computerized system of claim 1 , wherein one or more of the processors are to: calculate, by one or more of the signing entities, a plurality of public key shares based on a secret key share of the first signing entity and a secret key share of the blockchain entity; and resolve, by the blockchain entity, one or more of the public key shares, wherein at least one of: the generating of data points of the blockchain entity, the reconstructing of data points of the signing entities, and the performing of an exchange of data are performed based on the resolved public key shares.
calculate, by one or more of the signing entities, a plurality of public key shares based on a secret key share of the first signing entity and a secret key share of the blockchain entity; and
resolve, by the blockchain entity, one or more of the public key shares, wherein at least one of: the generating of data points of the blockchain entity, the reconstructing of data points of the signing entities, and the performing of an exchange of data are performed based on the resolved public key shares.
The computerized system of claim 1 , wherein one or more of the processors are to simulate, by the blockchain entity, one or more computerized operations, the simulated operations associated with one or more of the data points of the signing entities.
The computerized system of claim 1 , wherein the blockchain entity is implemented on a blockchain network.
The computerized system of claim 16 , wherein the exchange of data comprises excluding the first entity from the blockchain network.
The computerized system of claim 15 , wherein the simulated operations comprise one or more pseudorandom values, and wherein the simulated operations describe an adversarial computerized behavior by one or more of the signing entities.
The computerized system of claim 14 , wherein the public key shares are generated using an elliptic curve algorithm.
The computerized system of claim 16 , wherein the blockchain network comprises an Ethereum blockchain.
Ref US20220270590: Claims:
1. A method performed by one or more processors of a client device, the method comprising:
detecting, via one or more microphones of the client device, audio data that captures at least part of a spoken utterance of a user of the client device; processing, using a local machine learning model stored locally on the client device, the audio data to generate predicted output, wherein the local machine learning model includes a portion used in generating an encoding of the audio data; generating, using unsupervised learning, a gradient based on the predicted output; and transmitting, to a remote system and from the client device, the generated gradient to cause the remote system to utilize the generated gradient to update weights of global machine learning model layers that are stored remotely at the remote system and that correspond to the portion, of the local machine learning model, used in generating the encoding of the audio data, and subsequent to the remote system updating the weights of the global machine learning model layers utilizing the generated gradient and the remote system further updating the weights based on additional gradients from additional client devices:
receiving, at the client device and from the remote system, a combined machine learning model that includes the updated global machine learning model layers and one or more additional layers; and
using the combined machine learning model to make at least one prediction based on further audio data, detected via one or more of the microphones of the client device, that captures at least part of a further spoken utterance of the user of the client device.
2. The method of claim 1, wherein the portion of the local machine learning model stored locally on the client device is used to generate the encoding, wherein the encoding is processed using an additional portion of the local machine learning model to generate the predicted output, and wherein the predicted output is predicted audio data.
3. The method of claim 2, wherein generating the gradient based on the predicted output comprises:
comparing the audio data that captures at least part of the spoken utterance of the user of the client device to the predicted audio data; and generating the gradient based on comparing the audio data and the predicted audio data.
4. The method of claim 2, wherein the audio data that captures at least part of the spoken utterance of the user of the client device captures a first portion of the spoken utterance that is followed by a second portion of the spoken utterance, and wherein generating the gradient is based on comparing the predicted audio data to additional audio data corresponding to the second portion of the spoken utterance.
5. The method of claim 4, wherein comparing the predicted audio data to the additional audio data corresponding to the second portion of the spoken utterance comprises:
comparing an audio waveform corresponding to the additional audio data that corresponds to the second portion of the spoken utterance to a predicted audio waveform corresponding to the predicted audio data; and generating the gradient based on comparing the audio waveform and the predicted audio waveform.
6. The method of claim 1, wherein the combined machine learning model is an automatic speech recognition (ASR) model, and wherein the at least one prediction comprises a plurality of predicted phonemes, or a plurality of predicted tokens, that correspond to the further spoken utterance.
7. The method of claim 1, wherein receiving the combined machine learning model and using the combined machine learning model are further subsequent to the remote system updating the weights of the global machine learning model layers utilizing publicly available audio data to generate further gradients, wherein each of the further gradients are generated remotely at the remote server based on:
unsupervised learning, at the remote system, that is based on processing, using a global machine learning model stored remotely at the remote server, the publicly available audio data that captures a publicly available spoken utteran
[Truncated to input limit]