neulab/docprompting-conala huggingface dataset:
tf.keras.optimizers.Adagrad     View source on GitHub    Optimizer that implements the Adagrad algorithm. Inherits From: Optimizer  View aliases  Main aliases 
tf.optimizers.Adagrad Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.Adagrad  
tf.keras.optimizers.Adagrad(
    learning_rate=0.001, initial_accumulator_value=0.1, epsilon=1e-07,
    name='Adagrad', **kwargs
)
 Adagrad is an optimizer with parameter-specific learning rates, which are adapted relative to how frequently a parameter gets updated during training. The more updates a parameter receives, the smaller the updates.
 


 Args
  learning_rate   A Tensor, floating point value, or a schedule that is a tf.keras.optimizers.schedules.LearningRateSchedule. The learning rate.  
  initial_accumulator_value   A floating point value. Starting value for the accumulators, must be non-negative.  
  epsilon   A small floating point value to avoid zero denominator.  
  name   Optional name prefix for the operations created when applying gradients. Defaults to "Adagrad".  
  **kwargs   Keyword arguments. Allowed to be one of "clipnorm" or "clipvalue". "clipnorm" (float) clips gradients by norm; "clipvalue" (float) clips gradients by value.    Reference:  
Duchi et al., 2011. 
 


 Raises
  ValueError   in case of any invalid argument.
tf.keras.optimizers.Adamax     View source on GitHub    Optimizer that implements the Adamax algorithm. Inherits From: Optimizer  View aliases  Main aliases 
tf.optimizers.Adamax Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.Adamax  
tf.keras.optimizers.Adamax(
    learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07,
    name='Adamax', **kwargs
)
 It is a variant of Adam based on the infinity norm. Default parameters follow those provided in the paper. Adamax is sometimes superior to adam, specially in models with embeddings. Initialization: m = 0  # Initialize initial 1st moment vector
v = 0  # Initialize the exponentially weighted infinity norm
t = 0  # Initialize timestep
 The update rule for parameter w with gradient g is described at the end of section 7.1 of the paper: t += 1
m = beta1 * m + (1 - beta) * g
v = max(beta2 * v, abs(g))
current_lr = learning_rate / (1 - beta1 ** t)
w = w - current_lr * m / (v + epsilon)
 Similarly to Adam, the epsilon is added for numerical stability (especially to get rid of division by zero when v_t == 0). In contrast to Adam, the sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of tf.gather or an embedding lookup in the forward pass) only updates variable slices and corresponding m_t, v_t terms when that part of the variable was used in the forward pass. This means that the sparse behavior is contrast to the dense behavior (similar to some momentum implementations which ignore momentum unless a variable slice was actually used).
 


 Args
  learning_rate   A Tensor, floating point value, or a schedule that is a tf.keras.optimizers.schedules.LearningRateSchedule. The learning rate.  
  beta_1   A float value or a constant float tensor. The exponential decay rate for the 1st moment estimates.  
  beta_2   A float value or a constant float tensor. The exponential decay rate for the exponentially weighted infinity norm.  
  epsilon   A small constant for numerical stability.  
  name   Optional name for the operations created when applying gradients. Defaults to "Adamax".  
  **kwargs   Keyword arguments. Allowed to be one of "clipnorm" or "clipvalue". "clipnorm" (float) clips gradients by norm; "clipvalue" (float) clips gradients by value.    Reference:  Kingma et al., 2014 
 


 Raises
  ValueError   in case of any invalid argument.
tf.keras.optimizers.deserialize     View source on GitHub    Inverse of the serialize function.  View aliases  Main aliases 
tf.optimizers.deserialize Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.deserialize  
tf.keras.optimizers.deserialize(
    config, custom_objects=None
)

 


 Arguments
  config   Optimizer configuration dictionary.  
  custom_objects   Optional dictionary mapping names (strings) to custom objects (classes and functions) to be considered during deserialization.   
 


 Returns   A Keras Optimizer instance.
tf.keras.optimizers.Ftrl       View source on GitHub    Optimizer that implements the FTRL algorithm. Inherits From: Optimizer  View aliases  Main aliases 
tf.optimizers.Ftrl Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.Ftrl  
tf.keras.optimizers.Ftrl(
    learning_rate=0.001, learning_rate_power=-0.5, initial_accumulator_value=0.1,
    l1_regularization_strength=0.0, l2_regularization_strength=0.0,
    name='Ftrl', l2_shrinkage_regularization_strength=0.0, beta=0.0,
    **kwargs
)
 See Algorithm 1 of this paper. This version has support for both online L2 (the L2 penalty given in the paper above) and shrinkage-type L2 (which is the addition of an L2 penalty to the loss function). Initialization:  $$t = 0$$   $$n_{0} = 0$$   $$\sigma_{0} = 0$$   $$z_{0} = 0$$  Update (  $$i$$  is variable index,   $$\alpha$$  is the learning rate):  $$t = t + 1$$   $$n_{t,i} = n_{t-1,i} + g_{t,i}^{2}$$   $$\sigma_{t,i} = (\sqrt{n_{t,i} } - \sqrt{n_{t-1,i} }) / \alpha$$   $$z_{t,i} = z_{t-1,i} + g_{t,i} - \sigma_{t,i} * w_{t,i}$$   $$w_{t,i} = - ((\beta+\sqrt{n_{t,i} }) / \alpha + 2 * \lambda_{2})^{-1} * (z_{i} - sgn(z_{i}) * \lambda_{1}) if \abs{z_{i} } > \lambda_{i} else 0$$  Check the documentation for the l2_shrinkage_regularization_strength parameter for more details when shrinkage is enabled, in which case gradient is replaced with gradient_with_shrinkage.
 


 Args
  learning_rate   A Tensor, floating point value, or a schedule that is a tf.keras.optimizers.schedules.LearningRateSchedule. The learning rate.  
  learning_rate_power   A float value, must be less or equal to zero. Controls how the learning rate decreases during training. Use zero for a fixed learning rate.  
  initial_accumulator_value   The starting value for accumulators. Only zero or positive values are allowed.  
  l1_regularization_strength   A float value, must be greater than or equal to zero.  
  l2_regularization_strength   A float value, must be greater than or equal to zero.  
  name   Optional name prefix for the operations created when applying gradients. Defaults to "Ftrl".  
  l2_shrinkage_regularization_strength   A float value, must be greater than or equal to zero. This differs from L2 above in that the L2 above is a stabilization penalty, whereas this L2 shrinkage is a magnitude penalty. When input is sparse shrinkage will only happen on the active weights.  
  beta   A float value, representing the beta value from the paper.  
  **kwargs   Keyword arguments. Allowed to be one of "clipnorm" or "clipvalue". "clipnorm" (float) clips gradients by norm; "clipvalue" (float) clips gradients by value.    Reference:  paper 
 


 Raises
  ValueError   in case of any invalid argument.
tf.keras.optimizers.get     View source on GitHub    Retrieves a Keras Optimizer instance.  View aliases  Main aliases 
tf.optimizers.get Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.get  
tf.keras.optimizers.get(
    identifier
)

 


 Arguments
  identifier   Optimizer identifier, one of  String: name of an optimizer Dictionary: configuration dictionary. - Keras Optimizer instance (it will be returned unchanged). - TensorFlow Optimizer instance (it will be wrapped as a Keras Optimizer). 

  
 


 Returns   A Keras Optimizer instance.  

 


 Raises
  ValueError   If identifier cannot be interpreted.
tf.keras.optimizers.Nadam     View source on GitHub    Optimizer that implements the NAdam algorithm. Inherits From: Optimizer  View aliases  Main aliases 
tf.optimizers.Nadam Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.Nadam  
tf.keras.optimizers.Nadam(
    learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07,
    name='Nadam', **kwargs
)
 Much like Adam is essentially RMSprop with momentum, Nadam is Adam with Nesterov momentum.
 


 Args
  learning_rate   A Tensor or a floating point value. The learning rate.  
  beta_1   A float value or a constant float tensor. The exponential decay rate for the 1st moment estimates.  
  beta_2   A float value or a constant float tensor. The exponential decay rate for the exponentially weighted infinity norm.  
  epsilon   A small constant for numerical stability.  
  name   Optional name for the operations created when applying gradients. Defaults to "Nadam".  
  **kwargs   Keyword arguments. Allowed to be one of "clipnorm" or "clipvalue". "clipnorm" (float) clips gradients by norm; "clipvalue" (float) clips gradients by value.    Reference:  
Dozat, 2015. 
 


 Raises
  ValueError   in case of any invalid argument.
tf.keras.optimizers.RMSprop     View source on GitHub    Optimizer that implements the RMSprop algorithm. Inherits From: Optimizer  View aliases  Main aliases 
tf.optimizers.RMSprop Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.RMSprop  
tf.keras.optimizers.RMSprop(
    learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False,
    name='RMSprop', **kwargs
)
 The gist of RMSprop is to:  Maintain a moving (discounted) average of the square of gradients Divide the gradient by the root of this average  This implementation of RMSprop uses plain momentum, not Nesterov momentum. The centered version additionally maintains a moving average of the gradients, and uses that average to estimate the variance.
 


 Args
  learning_rate   A Tensor, floating point value, or a schedule that is a tf.keras.optimizers.schedules.LearningRateSchedule, or a callable that takes no arguments and returns the actual value to use. The learning rate. Defaults to 0.001.  
  rho   Discounting factor for the history/coming gradient. Defaults to 0.9.  
  momentum   A scalar or a scalar Tensor. Defaults to 0.0.  
  epsilon   A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to 1e-7.  
  centered   Boolean. If True, gradients are normalized by the estimated variance of the gradient; if False, by the uncentered second moment. Setting this to True may help with training, but is slightly more expensive in terms of computation and memory. Defaults to False.  
  name   Optional name prefix for the operations created when applying gradients. Defaults to "RMSprop".  
  **kwargs   Keyword arguments. Allowed to be one of "clipnorm" or "clipvalue". "clipnorm" (float) clips gradients by norm; "clipvalue" (float) clips gradients by value.    Note that in the dense implementation of this algorithm, variables and their corresponding accumulators (momentum, gradient moving average, square gradient moving average) will be updated even if the gradient is zero (i.e. accumulators will decay, momentum will be applied). The sparse implementation (used when the gradient is an IndexedSlices object, typically because of tf.gather or an embedding lookup in the forward pass) will not update variable slices or their accumulators unless those slices were used in the forward pass (nor is there an "eventual" correction to account for these omitted updates). This leads to more efficient updates for large embedding lookup tables (where most of the slices are not accessed in a particular graph execution), but differs from the published algorithm. Usage: 
opt = tf.keras.optimizers.RMSprop(learning_rate=0.1)
var1 = tf.Variable(10.0)
loss = lambda: (var1 ** 2) / 2.0    # d(loss) / d(var1) = var1
step_count = opt.minimize(loss, [var1]).numpy()
var1.numpy()
9.683772
 Reference:  Hinton, 2012
Module: tf.keras.optimizers.schedules Public API for tf.keras.optimizers.schedules namespace.  View aliases  Main aliases 
tf.optimizers.schedules  Classes class ExponentialDecay: A LearningRateSchedule that uses an exponential decay schedule. class InverseTimeDecay: A LearningRateSchedule that uses an inverse time decay schedule. class LearningRateSchedule: A serializable learning rate decay schedule. class PiecewiseConstantDecay: A LearningRateSchedule that uses a piecewise constant decay schedule. class PolynomialDecay: A LearningRateSchedule that uses a polynomial decay schedule. Functions deserialize(...) serialize(...)
tf.keras.optimizers.schedules.deserialize     View source on GitHub     View aliases  Main aliases 
tf.optimizers.schedules.deserialize Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.schedules.deserialize  
tf.keras.optimizers.schedules.deserialize(
    config, custom_objects=None
)
tf.keras.optimizers.schedules.ExponentialDecay     View source on GitHub    A LearningRateSchedule that uses an exponential decay schedule. Inherits From: LearningRateSchedule  View aliases  Main aliases 
tf.optimizers.schedules.ExponentialDecay Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.schedules.ExponentialDecay  
tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate, decay_steps, decay_rate, staircase=False, name=None
)
 When training a model, it is often recommended to lower the learning rate as the training progresses. This schedule applies an exponential decay function to an optimizer step, given a provided initial learning rate. The schedule a 1-arg callable that produces a decayed learning rate when passed the current optimizer step. This can be useful for changing the learning rate value across different invocations of optimizer functions. It is computed as: def decayed_learning_rate(step):
  return initial_learning_rate * decay_rate ^ (step / decay_steps)
 If the argument staircase is True, then step / decay_steps is an integer division and the decayed learning rate follows a staircase function. You can pass this schedule directly into a tf.keras.optimizers.Optimizer as the learning rate. Example: When fitting a Keras model, decay every 100000 steps with a base of 0.96: initial_learning_rate = 0.1
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate,
    decay_steps=100000,
    decay_rate=0.96,
    staircase=True)

model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=lr_schedule),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(data, labels, epochs=5)
 The learning rate schedule is also serializable and deserializable using tf.keras.optimizers.schedules.serialize and tf.keras.optimizers.schedules.deserialize.
 


 Returns   A 1-arg callable learning rate schedule that takes the current optimizer step and outputs the decayed learning rate, a scalar Tensor of the same type as initial_learning_rate.  

 


 Args
  initial_learning_rate   A scalar float32 or float64 Tensor or a Python number. The initial learning rate.  
  decay_steps   A scalar int32 or int64 Tensor or a Python number. Must be positive. See the decay computation above.  
  decay_rate   A scalar float32 or float64 Tensor or a Python number. The decay rate.  
  staircase   Boolean. If True decay the learning rate at discrete intervals  
  name   String. Optional name of the operation. Defaults to 'ExponentialDecay'.    Methods from_config View source 
@classmethod
from_config(
    config
)
 Instantiates a LearningRateSchedule from its config.
 


 Args
  config   Output of get_config().   
 


 Returns   A LearningRateSchedule instance.  
 get_config View source 
get_config()
 __call__ View source 
__call__(
    step
)
 Call self as a function.
tf.keras.optimizers.schedules.InverseTimeDecay     View source on GitHub    A LearningRateSchedule that uses an inverse time decay schedule. Inherits From: LearningRateSchedule  View aliases  Main aliases 
tf.optimizers.schedules.InverseTimeDecay Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.schedules.InverseTimeDecay  
tf.keras.optimizers.schedules.InverseTimeDecay(
    initial_learning_rate, decay_steps, decay_rate, staircase=False, name=None
)
 When training a model, it is often recommended to lower the learning rate as the training progresses. This schedule applies the inverse decay function to an optimizer step, given a provided initial learning rate. It requires a step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step. The schedule a 1-arg callable that produces a decayed learning rate when passed the current optimizer step. This can be useful for changing the learning rate value across different invocations of optimizer functions. It is computed as: def decayed_learning_rate(step):
  return initial_learning_rate / (1 + decay_rate * step / decay_step)
 or, if staircase is True, as: def decayed_learning_rate(step):
  return initial_learning_rate / (1 + decay_rate * floor(step / decay_step))
 You can pass this schedule directly into a tf.keras.optimizers.Optimizer as the learning rate. Example: Fit a Keras model when decaying 1/t with a rate of 0.5: ...
initial_learning_rate = 0.1
decay_steps = 1.0
decay_rate = 0.5
learning_rate_fn = keras.optimizers.schedules.InverseTimeDecay(
  initial_learning_rate, decay_steps, decay_rate)

model.compile(optimizer=tf.keras.optimizers.SGD(
                  learning_rate=learning_rate_fn),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(data, labels, epochs=5)

 


 Returns   A 1-arg callable learning rate schedule that takes the current optimizer step and outputs the decayed learning rate, a scalar Tensor of the same type as initial_learning_rate.  

 


 Args
  initial_learning_rate   A scalar float32 or float64 Tensor or a Python number. The initial learning rate.  
  decay_steps   How often to apply decay.  
  decay_rate   A Python number. The decay rate.  
  staircase   Whether to apply decay in a discrete staircase, as opposed to continuous, fashion.  
  name   String. Optional name of the operation. Defaults to 'InverseTimeDecay'.    Methods from_config View source 
@classmethod
from_config(
    config
)
 Instantiates a LearningRateSchedule from its config.
 


 Args
  config   Output of get_config().   
 


 Returns   A LearningRateSchedule instance.  
 get_config View source 
get_config()
 __call__ View source 
__call__(
    step
)
 Call self as a function.
tf.keras.optimizers.schedules.LearningRateSchedule     View source on GitHub    A serializable learning rate decay schedule.  View aliases  Main aliases 
tf.optimizers.schedules.LearningRateSchedule Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.schedules.LearningRateSchedule  LearningRateSchedules can be passed in as the learning rate of optimizers in tf.keras.optimizers. They can be serialized and deserialized using tf.keras.optimizers.schedules.serialize and tf.keras.optimizers.schedules.deserialize. Methods from_config View source 
@classmethod
from_config(
    config
)
 Instantiates a LearningRateSchedule from its config.
 


 Args
  config   Output of get_config().   
 


 Returns   A LearningRateSchedule instance.  
 get_config View source 
@abc.abstractmethod
get_config()
 __call__ View source 
@abc.abstractmethod
__call__(
    step
)
 Call self as a function.
tf.keras.optimizers.schedules.PiecewiseConstantDecay     View source on GitHub    A LearningRateSchedule that uses a piecewise constant decay schedule. Inherits From: LearningRateSchedule  View aliases  Main aliases 
tf.optimizers.schedules.PiecewiseConstantDecay Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.schedules.PiecewiseConstantDecay  
tf.keras.optimizers.schedules.PiecewiseConstantDecay(
    boundaries, values, name=None
)
 The function returns a 1-arg callable to compute the piecewise constant when passed the current optimizer step. This can be useful for changing the learning rate value across different invocations of optimizer functions. Example: use a learning rate that's 1.0 for the first 100001 steps, 0.5 for the next 10000 steps, and 0.1 for any additional steps. step = tf.Variable(0, trainable=False)
boundaries = [100000, 110000]
values = [1.0, 0.5, 0.1]
learning_rate_fn = keras.optimizers.schedules.PiecewiseConstantDecay(
    boundaries, values)

# Later, whenever we perform an optimization step, we pass in the step.
learning_rate = learning_rate_fn(step)
 You can pass this schedule directly into a tf.keras.optimizers.Optimizer as the learning rate. The learning rate schedule is also serializable and deserializable using tf.keras.optimizers.schedules.serialize and tf.keras.optimizers.schedules.deserialize.
 


 Returns   A 1-arg callable learning rate schedule that takes the current optimizer step and outputs the decayed learning rate, a scalar Tensor of the same type as the boundary tensors. The output of the 1-arg function that takes the step is values[0] when step <= boundaries[0], values[1] when step > boundaries[0] and step <= boundaries[1], ..., and values[-1] when step > boundaries[-1]. 
 

 


 Args
  boundaries   A list of Tensors or ints or floats with strictly increasing entries, and with all elements having the same type as the optimizer step.  
  values   A list of Tensors or floats or ints that specifies the values for the intervals defined by boundaries. It should have one more element than boundaries, and all elements should have the same type.  
  name   A string. Optional name of the operation. Defaults to 'PiecewiseConstant'.   
 


 Raises
  ValueError   if the number of elements in the lists do not match.    Methods from_config View source 
@classmethod
from_config(
    config
)
 Instantiates a LearningRateSchedule from its config.
 


 Args
  config   Output of get_config().   
 


 Returns   A LearningRateSchedule instance.  
 get_config View source 
get_config()
 __call__ View source 
__call__(
    step
)
 Call self as a function.
tf.keras.optimizers.schedules.serialize     View source on GitHub     View aliases  Main aliases 
tf.optimizers.schedules.serialize Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.schedules.serialize  
tf.keras.optimizers.schedules.serialize(
    learning_rate_schedule
)
tf.keras.optimizers.serialize     View source on GitHub     View aliases  Main aliases 
tf.optimizers.serialize Compat aliases for migration See Migration guide for more details. tf.compat.v1.keras.optimizers.serialize  
tf.keras.optimizers.serialize(
    optimizer
)