Patent ID: 11928601
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer-implemented method for neural network compression, the method comprising:
receiving a neural network;
identifying a set of weights of the neural network;
determining initial values of a set of anchor points based on initial values of the set of weights of the neural network;
training the neural network by, at each of multiple training iterations, performing operations comprising:
adjusting current values of the set of weights of the neural network at the training iteration by backpropagating gradients of a loss function, wherein the loss function comprises:
a first loss function term based on a prediction accuracy of the neural network; and
a second loss function term based on a similarity of the current values of the set of weights of the neural network at the training iteration to current values of the set of anchor points at the training iteration; and

adjusting the current values of the set of anchor points at the training iteration based on the current values of the set of weights of the neural network at the training iteration, the adjusting comprising:
determining a set of quantiles of the current values of the set of weights of the neural network at the training iteration; and
setting, for each of one or more anchor points in the set of anchor points, a current value of the anchor point equal to a statistic of current values of a plurality of weights of the neural network that are included in an interval defined by a respective pair of quantiles from the set of quantiles;

quantizing the current values of the set of weights of the neural network, comprising, for each weight of the set of weights of the neural network:
determining an anchor point in the set of anchor points corresponding to the weight; and
setting a current value of the weight to a current value of the corresponding anchor point.