Patent ID: 11875262
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A system implemented by one or more computers for training a neural network, the system comprising:
a neural network shrinking engine that is configured to:
train the neural network based on a shrinking engine loss function that includes shrinkage terms penalizing active neurons of the neural network;
remove inactive neurons from the neural network;

a neural network expansion engine that is configured to:
add new neurons to the neural network;
train the neural network based on an expanding engine loss function;

a training subsystem that is configured to, at each of a plurality of iterations:
shrink the neural network, comprising:
determining that the neural network does not satisfy a constraint and, in response, modifying the shrinking engine loss function to encourage the neural network to satisfy the constraint, wherein:
the shrinking engine loss function comprises a set of one or more shrinkage factors that parameterize the shrinking engine loss function by modulating respective shrinkage terms in the shrinking engine loss function; and
modifying the shrinking engine loss function comprises adjusting a respective value of each of one or more of the shrinkage factors of the shrinking engine loss function; and

providing the neural network to the neural network shrinking engine; and

expand the neural network by providing the neural network to the neural network expansion engine.