neulab/docprompting-conala huggingface dataset:
tf.estimator.add_metrics     View source on GitHub    Creates a new tf.estimator.Estimator which has given metrics.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.add_metrics  
tf.estimator.add_metrics(
    estimator, metric_fn
)
 Example: def my_auc(labels, predictions):
  auc_metric = tf.keras.metrics.AUC(name="my_auc")
  auc_metric.update_state(y_true=labels, y_pred=predictions['logistic'])
  return {'auc': auc_metric}

estimator = tf.estimator.DNNClassifier(...)
estimator = tf.estimator.add_metrics(estimator, my_auc)
estimator.train(...)
estimator.evaluate(...)
 Example usage of custom metric which uses features: def my_auc(labels, predictions, features):
  auc_metric = tf.keras.metrics.AUC(name="my_auc")
  auc_metric.update_state(y_true=labels, y_pred=predictions['logistic'],
                          sample_weight=features['weight'])
  return {'auc': auc_metric}

estimator = tf.estimator.DNNClassifier(...)
estimator = tf.estimator.add_metrics(estimator, my_auc)
estimator.train(...)
estimator.evaluate(...)

 


 Args
  estimator   A tf.estimator.Estimator object.  
  metric_fn   A function which should obey the following signature:  Args: can only have following four arguments in any order: predictions: Predictions Tensor or dict of Tensor created by given estimator. features: Input dict of Tensor objects created by input_fn which is given to estimator.evaluate as an argument. labels: Labels Tensor or dict of Tensor created by input_fn which is given to estimator.evaluate as an argument. config: config attribute of the estimator. Returns: Dict of metric results keyed by name. Final metrics are a union of this and estimator's existing metrics. If there is a name conflict between this and estimators existing metrics, this will override the existing one. The values of the dict are the results of calling a metric function, namely a (metric_tensor, update_op) tuple. 

  
 


 Returns   A new tf.estimator.Estimator which has a union of original metrics with given ones.
tf.estimator.BestExporter     View source on GitHub    This class exports the serving graph and checkpoints of the best models. Inherits From: Exporter  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.BestExporter  
tf.estimator.BestExporter(
    name='best_exporter', serving_input_receiver_fn=None,
    event_file_pattern='eval/*.tfevents.*', compare_fn=_loss_smaller,
    assets_extra=None, as_text=False, exports_to_keep=5
)
 This class performs a model export everytime the new model is better than any existing model.
 


 Args
  name   unique name of this Exporter that is going to be used in the export path.  
  serving_input_receiver_fn   a function that takes no arguments and returns a ServingInputReceiver.  
  event_file_pattern   event file name pattern relative to model_dir. If None, however, the exporter would not be preemption-safe. To be preemption-safe, event_file_pattern must be specified.  
  compare_fn   a function that compares two evaluation results and returns true if current evaluation result is better. Follows the signature:  Args: 
best_eval_result: This is the evaluation result of the best model. 
current_eval_result: This is the evaluation result of current candidate model. Returns: True if current evaluation result is better; otherwise, False. 

 
  assets_extra   An optional dict specifying how to populate the assets.extra directory within the exported SavedModel. Each key should give the destination path (including the filename) relative to the assets.extra directory. The corresponding value gives the full path of the source file to be copied. For example, the simple case of copying a single file without renaming it is specified as {'my_asset_file.txt': '/path/to/my_asset_file.txt'}.  
  as_text   whether to write the SavedModel proto in text format. Defaults to False.  
  exports_to_keep   Number of exports to keep. Older exports will be garbage-collected. Defaults to 5. Set to None to disable garbage collection.   
 


 Raises
  ValueError   if any argument is invalid.   
 


 Attributes
  name   Directory name. A directory name under the export base directory where exports of this type are written. Should not be None nor empty. 
   Methods export View source 
export(
    estimator, export_path, checkpoint_path, eval_result, is_the_final_export
)
 Exports the given Estimator to a specific format.
 


 Args
  estimator   the Estimator to export.  
  export_path   A string containing a directory where to write the export.  
  checkpoint_path   The checkpoint path to export.  
  eval_result   The output of Estimator.evaluate on this checkpoint.  
  is_the_final_export   This boolean is True when this is an export in the end of training. It is False for the intermediate exports during the training. When passing Exporter to tf.estimator.train_and_evaluate is_the_final_export is always False if TrainSpec.max_steps is None.   
 


 Returns   The string path to the exported directory or None if export is skipped.
tf.estimator.CheckpointSaverListener Interface for listeners that take action before or after checkpoint save.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.CheckpointSaverListener, tf.compat.v1.train.CheckpointSaverListener  CheckpointSaverListener triggers only in steps when CheckpointSaverHook is triggered, and provides callbacks at the following points:  before using the session before each call to Saver.save()
 after each call to Saver.save()
 at the end of session  To use a listener, implement a class and pass the listener to a CheckpointSaverHook, as in this example: class ExampleCheckpointSaverListener(CheckpointSaverListener):
  def begin(self):
    # You can add ops to the graph here.
    print('Starting the session.')
    self.your_tensor = ...

  def before_save(self, session, global_step_value):
    print('About to write a checkpoint')

  def after_save(self, session, global_step_value):
    print('Done writing checkpoint.')
    if decided_to_stop_training():
      return True

  def end(self, session, global_step_value):
    print('Done with the session.')

...
listener = ExampleCheckpointSaverListener()
saver_hook = tf.estimator.CheckpointSaverHook(
    checkpoint_dir, listeners=[listener])
with
tf.compat.v1.train.MonitoredTrainingSession(chief_only_hooks=[saver_hook]):
  ...
 A CheckpointSaverListener may simply take some action after every checkpoint save. It is also possible for the listener to use its own schedule to act less frequently, e.g. based on global_step_value. In this case, implementors should implement the end() method to handle actions related to the last checkpoint save. But the listener should not act twice if after_save() already handled this last checkpoint save. A CheckpointSaverListener can request training to be stopped, by returning True in after_save. Please note that, in replicated distributed training setting, only chief should use this behavior. Otherwise each worker will do their own evaluation, which may be wasteful of resources. Methods after_save View source 
after_save(
    session, global_step_value
)
 before_save View source 
before_save(
    session, global_step_value
)
 begin View source 
begin()
 end View source 
end(
    session, global_step_value
)
tf.estimator.EstimatorSpec     View source on GitHub    Ops and objects returned from a model_fn and passed to an Estimator.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.EstimatorSpec  
tf.estimator.EstimatorSpec(
    mode, predictions=None, loss=None, train_op=None, eval_metric_ops=None,
    export_outputs=None, training_chief_hooks=None, training_hooks=None,
    scaffold=None, evaluation_hooks=None, prediction_hooks=None
)
 EstimatorSpec fully defines the model to be run by an Estimator.
 


 Args
  mode   A ModeKeys. Specifies if this is training, evaluation or prediction.  
  predictions   Predictions Tensor or dict of Tensor.  
  loss   Training loss Tensor. Must be either scalar, or with shape [1].  
  train_op   Op for the training step.  
  eval_metric_ops   Dict of metric results keyed by name. The values of the dict can be one of the following: (1) instance of Metric class. (2) Results of calling a metric function, namely a (metric_tensor, update_op) tuple. metric_tensor should be evaluated without any impact on state (typically is a pure computation results based on variables.). For example, it should not trigger the update_op or requires any input fetching.  
  export_outputs   Describes the output signatures to be exported to SavedModel and used during serving. A dict {name: output} where:  name: An arbitrary name for this output. output: an ExportOutput object such as ClassificationOutput, RegressionOutput, or PredictOutput. Single-headed models only need to specify one entry in this dictionary. Multi-headed models should specify one entry for each head, one of which must be named using tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY. If no entry is provided, a default PredictOutput mapping to predictions will be created. 

 
  training_chief_hooks   Iterable of tf.train.SessionRunHook objects to run on the chief worker during training.  
  training_hooks   Iterable of tf.train.SessionRunHook objects to run on all workers during training.  
  scaffold   A tf.train.Scaffold object that can be used to set initialization, saver, and more to be used in training.  
  evaluation_hooks   Iterable of tf.train.SessionRunHook objects to run during evaluation.  
  prediction_hooks   Iterable of tf.train.SessionRunHook objects to run during predictions.   
 


 Raises
  ValueError   If validation fails.  
  TypeError   If any of the arguments is not the expected type.   
 


 Attributes
  mode  
 
  predictions  
 
  loss  
 
  train_op  
 
  eval_metric_ops  
 
  export_outputs  
 
  training_chief_hooks  
 
  training_hooks  
 
  scaffold  
 
  evaluation_hooks  
 
  prediction_hooks
tf.estimator.EvalSpec     View source on GitHub    Configuration for the "eval" part for the train_and_evaluate call.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.EvalSpec  
tf.estimator.EvalSpec(
    input_fn, steps=100, name=None, hooks=None, exporters=None,
    start_delay_secs=120, throttle_secs=600
)
 EvalSpec combines details of evaluation of the trained model as well as its export. Evaluation consists of computing metrics to judge the performance of the trained model. Export writes out the trained model on to external storage.
 


 Args
  input_fn   A function that constructs the input data for evaluation. See Premade Estimators for more information. The function should construct and return one of the following:  A 'tf.data.Dataset' object: Outputs of Dataset object must be a tuple (features, labels) with same constraints as below. A tuple (features, labels): Where features is a Tensor or a dictionary of string feature name to Tensor and labels is a Tensor or a dictionary of string label name to Tensor. 

 
  steps   Int. Positive number of steps for which to evaluate model. If None, evaluates until input_fn raises an end-of-input exception. See Estimator.evaluate for details.  
  name   String. Name of the evaluation if user needs to run multiple evaluations on different data sets. Metrics for different evaluations are saved in separate folders, and appear separately in tensorboard.  
  hooks   Iterable of tf.train.SessionRunHook objects to run during evaluation.  
  exporters   Iterable of Exporters, or a single one, or None. exporters will be invoked after each evaluation.  
  start_delay_secs   Int. Start evaluating after waiting for this many seconds.  
  throttle_secs   Int. Do not re-evaluate unless the last evaluation was started at least this many seconds ago. Of course, evaluation does not occur if no new checkpoints are available, hence, this is the minimum.   
 


 Raises
  ValueError   If any of the input arguments is invalid.  
  TypeError   If any of the arguments is not of the expected type.   
 


 Attributes
  input_fn  
 
  steps  
 
  name  
 
  hooks  
 
  exporters  
 
  start_delay_secs  
 
  throttle_secs
Module: tf.estimator.experimental Public API for tf.estimator.experimental namespace. Classes class InMemoryEvaluatorHook: Hook to run evaluation in training without a checkpoint. class LinearSDCA: Stochastic Dual Coordinate Ascent helper for linear estimators. class RNNClassifier: A classifier for TensorFlow RNN models. class RNNEstimator: An Estimator for TensorFlow RNN models with user-specified head. Functions build_raw_supervised_input_receiver_fn(...): Build a supervised_input_receiver_fn for raw features and labels. call_logit_fn(...): Calls logit_fn (experimental). make_early_stopping_hook(...): Creates early-stopping hook. make_stop_at_checkpoint_step_hook(...): Creates a proper StopAtCheckpointStepHook based on chief status. stop_if_higher_hook(...): Creates hook to stop if the given metric is higher than the threshold. stop_if_lower_hook(...): Creates hook to stop if the given metric is lower than the threshold. stop_if_no_decrease_hook(...): Creates hook to stop if metric does not decrease within given max steps. stop_if_no_increase_hook(...): Creates hook to stop if metric does not increase within given max steps.
tf.estimator.experimental.build_raw_supervised_input_receiver_fn Build a supervised_input_receiver_fn for raw features and labels.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.build_raw_supervised_input_receiver_fn  
tf.estimator.experimental.build_raw_supervised_input_receiver_fn(
    features, labels, default_batch_size=None
)
 This function wraps tensor placeholders in a supervised_receiver_fn with the expectation that the features and labels appear precisely as the model_fn expects them. Features and labels can therefore be dicts of tensors, or raw tensors.
 


 Args
  features   a dict of string to Tensor or Tensor.  
  labels   a dict of string to Tensor or Tensor.  
  default_batch_size   the number of query examples expected per batch. Leave unset for variable batch size (recommended).   
 


 Returns   A supervised_input_receiver_fn.  

 


 Raises
  ValueError   if features and labels have overlapping keys.
tf.estimator.experimental.call_logit_fn     View source on GitHub    Calls logit_fn (experimental).  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.call_logit_fn  
tf.estimator.experimental.call_logit_fn(
    logit_fn, features, mode, params, config
)
 THIS FUNCTION IS EXPERIMENTAL. Keras layers/models are the recommended APIs for logit and model composition. A utility function that calls the provided logit_fn with the relevant subset of provided arguments. Similar to tf.estimator._call_model_fn().
 


 Args
  logit_fn   A logit_fn as defined above.  
  features   The features dict.  
  mode   TRAIN / EVAL / PREDICT ModeKeys.  
  params   The hyperparameter dict.  
  config   The configuration object.   
 


 Returns   A logit Tensor, the output of logit_fn.  

 


 Raises
  ValueError   if logit_fn does not return a Tensor or a dictionary mapping strings to Tensors.
tf.estimator.experimental.make_early_stopping_hook     View source on GitHub    Creates early-stopping hook.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.make_early_stopping_hook  
tf.estimator.experimental.make_early_stopping_hook(
    estimator, should_stop_fn, run_every_secs=60, run_every_steps=None
)
 Returns a SessionRunHook that stops training when should_stop_fn returns True. Usage example: estimator = ...
hook = early_stopping.make_early_stopping_hook(
    estimator, should_stop_fn=make_stop_fn(...))
train_spec = tf.estimator.TrainSpec(..., hooks=[hook])
tf.estimator.train_and_evaluate(estimator, train_spec, ...)
 Caveat: Current implementation supports early-stopping both training and evaluation in local mode. In distributed mode, training can be stopped but evaluation (where it's a separate job) will indefinitely wait for new model checkpoints to evaluate, so you will need other means to detect and stop it. Early-stopping evaluation in distributed mode requires changes in train_and_evaluate API and will be addressed in a future revision.
 


 Args
  estimator   A tf.estimator.Estimator instance.  
  should_stop_fn   callable, function that takes no arguments and returns a bool. If the function returns True, stopping will be initiated by the chief.  
  run_every_secs   If specified, calls should_stop_fn at an interval of run_every_secs seconds. Defaults to 60 seconds. Either this or run_every_steps must be set.  
  run_every_steps   If specified, calls should_stop_fn every run_every_steps steps. Either this or run_every_secs must be set.   
 


 Returns   A SessionRunHook that periodically executes should_stop_fn and initiates early stopping if the function returns True.  

 


 Raises
  TypeError   If estimator is not of type tf.estimator.Estimator.  
  ValueError   If both run_every_secs and run_every_steps are set.
tf.estimator.experimental.make_stop_at_checkpoint_step_hook     View source on GitHub    Creates a proper StopAtCheckpointStepHook based on chief status.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.make_stop_at_checkpoint_step_hook  
tf.estimator.experimental.make_stop_at_checkpoint_step_hook(
    estimator, last_step, wait_after_file_check_secs=30
)
tf.estimator.experimental.stop_if_higher_hook     View source on GitHub    Creates hook to stop if the given metric is higher than the threshold.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.stop_if_higher_hook  
tf.estimator.experimental.stop_if_higher_hook(
    estimator, metric_name, threshold, eval_dir=None, min_steps=0,
    run_every_secs=60, run_every_steps=None
)
 Usage example: estimator = ...
# Hook to stop training if accuracy becomes higher than 0.9.
hook = early_stopping.stop_if_higher_hook(estimator, "accuracy", 0.9)
train_spec = tf.estimator.TrainSpec(..., hooks=[hook])
tf.estimator.train_and_evaluate(estimator, train_spec, ...)
 Caveat: Current implementation supports early-stopping both training and evaluation in local mode. In distributed mode, training can be stopped but evaluation (where it's a separate job) will indefinitely wait for new model checkpoints to evaluate, so you will need other means to detect and stop it. Early-stopping evaluation in distributed mode requires changes in train_and_evaluate API and will be addressed in a future revision.
 


 Args
  estimator   A tf.estimator.Estimator instance.  
  metric_name   str, metric to track. "loss", "accuracy", etc.  
  threshold   Numeric threshold for the given metric.  
  eval_dir   If set, directory containing summary files with eval metrics. By default, estimator.eval_dir() will be used.  
  min_steps   int, stop is never requested if global step is less than this value. Defaults to 0.  
  run_every_secs   If specified, calls should_stop_fn at an interval of run_every_secs seconds. Defaults to 60 seconds. Either this or run_every_steps must be set.  
  run_every_steps   If specified, calls should_stop_fn every run_every_steps steps. Either this or run_every_secs must be set.   
 


 Returns   An early-stopping hook of type SessionRunHook that periodically checks if the given metric is higher than specified threshold and initiates early stopping if true.
tf.estimator.experimental.stop_if_lower_hook     View source on GitHub    Creates hook to stop if the given metric is lower than the threshold.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.stop_if_lower_hook  
tf.estimator.experimental.stop_if_lower_hook(
    estimator, metric_name, threshold, eval_dir=None, min_steps=0,
    run_every_secs=60, run_every_steps=None
)
 Usage example: estimator = ...
# Hook to stop training if loss becomes lower than 100.
hook = early_stopping.stop_if_lower_hook(estimator, "loss", 100)
train_spec = tf.estimator.TrainSpec(..., hooks=[hook])
tf.estimator.train_and_evaluate(estimator, train_spec, ...)
 Caveat: Current implementation supports early-stopping both training and evaluation in local mode. In distributed mode, training can be stopped but evaluation (where it's a separate job) will indefinitely wait for new model checkpoints to evaluate, so you will need other means to detect and stop it. Early-stopping evaluation in distributed mode requires changes in train_and_evaluate API and will be addressed in a future revision.
 


 Args
  estimator   A tf.estimator.Estimator instance.  
  metric_name   str, metric to track. "loss", "accuracy", etc.  
  threshold   Numeric threshold for the given metric.  
  eval_dir   If set, directory containing summary files with eval metrics. By default, estimator.eval_dir() will be used.  
  min_steps   int, stop is never requested if global step is less than this value. Defaults to 0.  
  run_every_secs   If specified, calls should_stop_fn at an interval of run_every_secs seconds. Defaults to 60 seconds. Either this or run_every_steps must be set.  
  run_every_steps   If specified, calls should_stop_fn every run_every_steps steps. Either this or run_every_secs must be set.   
 


 Returns   An early-stopping hook of type SessionRunHook that periodically checks if the given metric is lower than specified threshold and initiates early stopping if true.
tf.estimator.experimental.stop_if_no_decrease_hook     View source on GitHub    Creates hook to stop if metric does not decrease within given max steps.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.estimator.experimental.stop_if_no_decrease_hook  
tf.estimator.experimental.stop_if_no_decrease_hook(
    estimator, metric_name, max_steps_without_decrease, eval_dir=None, min_steps=0,
    run_every_secs=60, run_every_steps=None
)
 Usage example: estimator = ...
# Hook to stop training if loss does not decrease in over 100000 steps.
hook = early_stopping.stop_if_no_decrease_hook(estimator, "loss", 100000)
train_spec = tf.estimator.TrainSpec(..., hooks=[hook])
tf.estimator.train_and_evaluate(estimator, train_spec, ...)
 Caveat: Current implementation supports early-stopping both training and evaluation in local mode. In distributed mode, training can be stopped but evaluation (where it's a separate job) will indefinitely wait for new model checkpoints to evaluate, so you will need other means to detect and stop it. Early-stopping evaluation in distributed mode requires changes in train_and_evaluate API and will be addressed in a future revision.
 


 Args
  estimator   A tf.estimator.Estimator instance.  
  metric_name   str, metric to track. "loss", "accuracy", etc.  
  max_steps_without_decrease   int, maximum number of training steps with no decrease in the given metric.  
  eval_dir   If set, directory containing summary files with eval metrics. By default, estimator.eval_dir() will be used.  
  min_steps   int, stop is never requested if global step is less than this value. Defaults to 0.  
  run_every_secs   If specified, calls should_stop_fn at an interval of run_every_secs seconds. Defaults to 60 seconds. Either this or run_every_steps must be set.  
  run_every_steps   If specified, calls should_stop_fn every run_every_steps steps. Either this or run_every_secs must be set.   
 


 Returns   An early-stopping hook of type SessionRunHook that periodically checks if the given metric shows no decrease over given maximum number of training steps, and initiates early stopping if true.