Patent ID: 11875262
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 18:
19. A method performed by one or more computers for training a neural network, the method comprising, at each of a plurality of iterations:
shrinking the neural network, comprising:
determining that the neural network does not satisfy a constraint and, in response, modifying a shrinking engine loss function to encourage the neural network to satisfy the constraint, wherein:
the shrinking engine loss function comprises a set of one or more shrinkage factors that parameterize the shrinking engine loss function by modulating respective shrinkage terms in the shrinking engine loss function, wherein each shrinkage term of the shrinking engine loss function penalizes active neurons of the neural network; and
modifying the shrinking engine loss function comprises adjusting a respective value of each of one or more of the shrinkage factors of the shrinking engine loss function; and

providing the neural network to a neural network shrinking engine that is configured to:
train the neural network based on the shrinking engine loss function; and
remove inactive neurons from the neural network; and

expand the neural network by providing the neural network to a neural network expansion engine that is configured to:
add new neurons to the neural network; and
train the neural network based on an expanding engine loss function.