self.states, self.unique_state = sanity_check_states(states)<EOL>self.actions, self.unique_action = sanity_check_actions(actions)<EOL>self.batched_observe = batched_observe<EOL>self.batching_capacity = batching_capacity<EOL>self.current_states = None<EOL>self.current_actions = None<EOL>self.current_internals = None<EOL>self.next_internals = None<EOL>self.current_terminal = None<EOL>self.current_reward = None<EOL>self.timestep = None<EOL>self.episode = None<EOL>self.model = self.initialize_model()<EOL>if self.batched_observe:<EOL><INDENT>assert self.batching_capacity is not None<EOL>self.observe_terminal = [list() for _ in range(self.model.num_parallel)]<EOL>self.observe_reward = [list() for _ in range(self.model.num_parallel)]<EOL><DEDENT>self.reset()<EOL>

Initializes the agent.

Args:
    states (spec, or dict of specs): States specification, with the following attributes
        (required):
        - type: one of 'bool', 'int', 'float' (default: 'float').
        - shape: integer, or list/tuple of integers (required).
    actions (spec, or dict of specs): Actions specification, with the following attributes
        (required):
        - type: one of 'bool', 'int', 'float' (required).
        - shape: integer, or list/tuple of integers (default: []).
        - num_actions: integer (required if type == 'int').
        - min_value and max_value: float (optional if type == 'float', default: none).
    batched_observe (bool): Specifies whether calls to model.observe() are batched, for
        improved performance (default: true).
    batching_capacity (int): Batching capacity of agent and model (default: 1000).


raise NotImplementedError<EOL>

Creates and returns the model (including a local replica in case of distributed learning) for this agent
based on specifications given by user. This method needs to be implemented by the different agent subclasses.


self.episode, self.timestep, self.next_internals = self.model.reset()<EOL>self.current_internals = self.next_internals<EOL>

Resets the agent to its initial state (e.g. on experiment start). Updates the Model's internal episode and
time step counter, internal states, and resets preprocessors.


def act(self, states, deterministic=False, independent=False, fetch_tensors=None, buffered=True, index=<NUM_LIT:0>):

self.current_internals = self.next_internals<EOL>if self.unique_state:<EOL><INDENT>self.current_states = dict(state=np.asarray(states))<EOL><DEDENT>else:<EOL><INDENT>self.current_states = {name: np.asarray(states[name]) for name in sorted(states)}<EOL><DEDENT>if fetch_tensors is not None:<EOL><INDENT>self.current_actions, self.next_internals, self.timestep, self.fetched_tensors = self.model.act(<EOL>states=self.current_states,<EOL>internals=self.current_internals,<EOL>deterministic=deterministic,<EOL>independent=independent,<EOL>fetch_tensors=fetch_tensors,<EOL>index=index<EOL>)<EOL>if self.unique_action:<EOL><INDENT>return self.current_actions['<STR_LIT:action>'], self.fetched_tensors<EOL><DEDENT>else:<EOL><INDENT>return self.current_actions, self.fetched_tensors<EOL><DEDENT><DEDENT>self.current_actions, self.next_internals, self.timestep = self.model.act(<EOL>states=self.current_states,<EOL>internals=self.current_internals,<EOL>deterministic=deterministic,<EOL>independent=independent,<EOL>index=index<EOL>)<EOL>if buffered:<EOL><INDENT>if self.unique_action:<EOL><INDENT>return self.current_actions['<STR_LIT:action>']<EOL><DEDENT>else:<EOL><INDENT>return self.current_actions<EOL><DEDENT><DEDENT>else:<EOL><INDENT>if self.unique_action:<EOL><INDENT>return self.current_actions['<STR_LIT:action>'], self.current_states, self.current_internals<EOL><DEDENT>else:<EOL><INDENT>return self.current_actions, self.current_states, self.current_internals<EOL><DEDENT><DEDENT>

Return action(s) for given state(s). States preprocessing and exploration are applied if
configured accordingly.

Args:
    states (any): One state (usually a value tuple) or dict of states if multiple states are expected.
    deterministic (bool): If true, no exploration and sampling is applied.
    independent (bool): If true, action is not followed by observe (and hence not included
        in updates).
    fetch_tensors (list): Optional String of named tensors to fetch
    buffered (bool): If true (default), states and internals are not returned but buffered
        with observes. Must be false for multi-threaded mode as we need atomic inserts.
Returns:
    Scalar value of the action or dict of multiple actions the agent wants to execute.
    (fetched_tensors) Optional dict() with named tensors fetched


def observe(self, terminal, reward, index=<NUM_LIT:0>):

self.current_terminal = terminal<EOL>self.current_reward = reward<EOL>if self.batched_observe:<EOL><INDENT>self.observe_terminal[index].append(self.current_terminal)<EOL>self.observe_reward[index].append(self.current_reward)<EOL>if self.current_terminal or len(self.observe_terminal[index]) >= self.batching_capacity:<EOL><INDENT>self.episode = self.model.observe(<EOL>terminal=self.observe_terminal[index],<EOL>reward=self.observe_reward[index],<EOL>index=index<EOL>)<EOL>self.observe_terminal[index] = list()<EOL>self.observe_reward[index] = list()<EOL><DEDENT><DEDENT>else:<EOL><INDENT>self.episode = self.model.observe(<EOL>terminal=self.current_terminal,<EOL>reward=self.current_reward<EOL>)<EOL><DEDENT>

Observe experience from the environment to learn from. Optionally pre-processes rewards
Child classes should call super to get the processed reward
EX: terminal, reward = super()...

Args:
    terminal (bool): boolean indicating if the episode terminated after the observation.
    reward (float): scalar reward that resulted from executing the action.


def atomic_observe(self, states, actions, internals, reward, terminal):

<EOL>self.current_terminal = terminal<EOL>self.current_reward = reward<EOL>if self.unique_state:<EOL><INDENT>states = dict(state=states)<EOL><DEDENT>if self.unique_action:<EOL><INDENT>actions = dict(action=actions)<EOL><DEDENT>self.episode = self.model.atomic_observe(<EOL>states=states,<EOL>actions=actions,<EOL>internals=internals,<EOL>terminal=self.current_terminal,<EOL>reward=self.current_reward<EOL>)<EOL>

Utility method for unbuffered observing where each tuple is inserted into TensorFlow via
a single session call, thus avoiding race conditions in multi-threaded mode.

Observe full experience  tuplefrom the environment to learn from. Optionally pre-processes rewards
Child classes should call super to get the processed reward
EX: terminal, reward = super()...

Args:
    states (any): One state (usually a value tuple) or dict of states if multiple states are expected.
    actions (any): One action (usually a value tuple) or dict of states if multiple actions are expected.
    internals (any): Internal list.
    terminal (bool): boolean indicating if the episode terminated after the observation.
    reward (float): scalar reward that resulted from executing the action.


def save_model(self, directory=None, append_timestep=True):

return self.model.save(directory=directory, append_timestep=append_timestep)<EOL>

Save TensorFlow model. If no checkpoint directory is given, the model's default saver
directory is used. Optionally appends current timestep to prevent overwriting previous
checkpoint files. Turn off to be able to load model from the same given path argument as
given here.

Args:
    directory (str): Optional checkpoint directory.
    append_timestep (bool):  Appends the current timestep to the checkpoint file if true.
        If this is set to True, the load path must include the checkpoint timestep suffix.
        For example, if stored to models/ and set to true, the exported file will be of the
        form models/model.ckpt-X where X is the last timestep saved. The load path must
        precisely match this file name. If this option is turned off, the checkpoint will
        always overwrite the file specified in path and the model can always be loaded under
        this path.

Returns:
    Checkpoint path were the model was saved.


def restore_model(self, directory=None, file=None):

self.model.restore(directory=directory, file=file)<EOL>

Restore TensorFlow model. If no checkpoint file is given, the latest checkpoint is
restored. If no checkpoint directory is given, the model's default saver directory is
used (unless file specifies the entire path).

Args:
    directory: Optional checkpoint directory.
    file: Optional checkpoint file, or path if directory not given.


agent = util.get_object(<EOL>obj=spec,<EOL>predefined_objects=tensorforce.agents.agents,<EOL>kwargs=kwargs<EOL>)<EOL>assert isinstance(agent, Agent)<EOL>return agent<EOL>

Creates an agent from a specification dict.


<EOL>if update_mode is None:<EOL><INDENT>update_mode = dict(<EOL>unit='<STR_LIT>',<EOL>batch_size=<NUM_LIT:10><EOL>)<EOL><DEDENT>elif '<STR_LIT>' in update_mode:<EOL><INDENT>pass<EOL><DEDENT>else:<EOL><INDENT>update_mode['<STR_LIT>'] = '<STR_LIT>'<EOL><DEDENT>if memory is None:<EOL><INDENT>memory = dict(<EOL>type='<STR_LIT>',<EOL>include_next_states=False,<EOL>capacity=(<NUM_LIT:1000> * update_mode['<STR_LIT>'])<EOL>)<EOL><DEDENT>else:<EOL><INDENT>assert not memory['<STR_LIT>']<EOL><DEDENT>if optimizer is None:<EOL><INDENT>optimizer = dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=<NUM_LIT><EOL>)<EOL><DEDENT>self.baseline_mode = baseline_mode<EOL>self.baseline = baseline<EOL>self.baseline_optimizer = baseline_optimizer<EOL>self.gae_lambda = gae_lambda<EOL>super(VPGAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity,<EOL>scope=scope,<EOL>device=device,<EOL>saver=saver,<EOL>summarizer=summarizer,<EOL>execution=execution,<EOL>variable_noise=variable_noise,<EOL>states_preprocessing=states_preprocessing,<EOL>actions_exploration=actions_exploration,<EOL>reward_preprocessing=reward_preprocessing,<EOL>update_mode=update_mode,<EOL>memory=memory,<EOL>optimizer=optimizer,<EOL>discount=discount,<EOL>network=network,<EOL>distributions=distributions,<EOL>entropy_regularization=entropy_regularization<EOL>)<EOL>

Initializes the VPG agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes:
        - unit: 'episodes' if given (default: 'episodes').
        - batch_size: integer (default: 10).
        - frequency: integer (default: batch_size).
    memory (spec): Memory specification, see core.memories module for more information
        (default: {type='latest', include_next_states=false, capacity=1000*batch_size}).
    optimizer (spec): Optimizer specification, see core.optimizers module for more
        information (default: {type='adam', learning_rate=1e-3}).
    baseline_mode (str): One of 'states', 'network' (default: none).
    baseline (spec): Baseline specification, see core.baselines module for more information
        (default: none).
    baseline_optimizer (spec): Baseline optimizer specification, see core.optimizers module
        for more information (default: none).
    gae_lambda (float): Lambda factor for generalized advantage estimation (default: none).


<EOL>if update_mode is None:<EOL><INDENT>update_mode = dict(<EOL>unit='<STR_LIT>',<EOL>batch_size=<NUM_LIT:10><EOL>)<EOL><DEDENT>elif '<STR_LIT>' in update_mode:<EOL><INDENT>pass<EOL><DEDENT>else:<EOL><INDENT>update_mode['<STR_LIT>'] = '<STR_LIT>'<EOL><DEDENT>if memory is None:<EOL><INDENT>memory = dict(<EOL>type='<STR_LIT>',<EOL>include_next_states=False,<EOL>capacity=(<NUM_LIT:1000> * update_mode['<STR_LIT>'])<EOL>)<EOL><DEDENT>else:<EOL><INDENT>assert not memory['<STR_LIT>']<EOL><DEDENT>assert (update_mode['<STR_LIT>'] != '<STR_LIT>' or memory['<STR_LIT:type>'] == '<STR_LIT>')<EOL>if step_optimizer is None:<EOL><INDENT>step_optimizer = dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=<NUM_LIT><EOL>)<EOL><DEDENT>optimizer = dict(<EOL>type='<STR_LIT>',<EOL>optimizer=dict(<EOL>type='<STR_LIT>',<EOL>optimizer=step_optimizer,<EOL>fraction=subsampling_fraction<EOL>),<EOL>num_steps=optimization_steps<EOL>)<EOL>self.baseline_mode = baseline_mode<EOL>self.baseline = baseline<EOL>self.baseline_optimizer = baseline_optimizer<EOL>self.gae_lambda = gae_lambda<EOL>self.likelihood_ratio_clipping = likelihood_ratio_clipping<EOL>super(PPOAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity,<EOL>scope=scope,<EOL>device=device,<EOL>saver=saver,<EOL>summarizer=summarizer,<EOL>execution=execution,<EOL>variable_noise=variable_noise,<EOL>states_preprocessing=states_preprocessing,<EOL>actions_exploration=actions_exploration,<EOL>reward_preprocessing=reward_preprocessing,<EOL>update_mode=update_mode,<EOL>memory=memory,<EOL>optimizer=optimizer,<EOL>discount=discount,<EOL>network=network,<EOL>distributions=distributions,<EOL>entropy_regularization=entropy_regularization<EOL>)<EOL>

Initializes the PPO agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes:
        - unit: 'episodes' if given (default: 'episodes').
        - batch_size: integer (default: 10).
        - frequency: integer (default: batch_size).
    memory (spec): Memory specification, see core.memories module for more information
        (default: {type='latest', include_next_states=false, capacity=1000*batch_size}).
    optimizer (spec): PPO agent implicitly defines a multi-step subsampling optimizer.
    baseline_mode (str): One of 'states', 'network' (default: none).
    baseline (spec): Baseline specification, see core.baselines module for more information
        (default: none).
    baseline_optimizer (spec): Baseline optimizer specification, see core.optimizers module
        for more information (default: none).
    gae_lambda (float): Lambda factor for generalized advantage estimation (default: none).
    likelihood_ratio_clipping (float): Likelihood ratio clipping for policy gradient
        (default: 0.2).
    step_optimizer (spec): Step optimizer specification of implicit multi-step subsampling
        optimizer, see core.optimizers module for more information (default: {type='adam',
        learning_rate=1e-3}).
    subsampling_fraction (float): Subsampling fraction of implicit subsampling optimizer
        (default: 0.1).
    optimization_steps (int): Number of optimization steps for implicit multi-step
        optimizer (default: 50).


self.scope = scope<EOL>self.device = device<EOL>self.saver = saver<EOL>self.summarizer = summarizer<EOL>self.execution = sanity_check_execution_spec(execution)<EOL>self.variable_noise = variable_noise<EOL>self.states_preprocessing = states_preprocessing<EOL>self.actions_exploration = actions_exploration<EOL>self.reward_preprocessing = reward_preprocessing<EOL>self.update_mode = update_mode<EOL>self.memory = memory<EOL>self.optimizer = optimizer<EOL>self.discount = discount<EOL>self.network = network<EOL>self.distributions = distributions<EOL>self.entropy_regularization = entropy_regularization<EOL>if self.summarizer is None:<EOL><INDENT>summary_labels = set()<EOL><DEDENT>else:<EOL><INDENT>summary_labels = set(self.summarizer.get('<STR_LIT>', ()))<EOL><DEDENT>self.meta_param_recorder = None<EOL>if any(k in summary_labels for k in ['<STR_LIT>', '<STR_LIT>']):<EOL><INDENT>self.meta_param_recorder = MetaParameterRecorder(inspect.currentframe())<EOL>if '<STR_LIT>' in self.summarizer:<EOL><INDENT>self.meta_param_recorder.merge_custom(self.summarizer['<STR_LIT>'])<EOL><DEDENT>if '<STR_LIT>' in summary_labels:<EOL><INDENT>self.summarizer['<STR_LIT>'] = self.meta_param_recorder<EOL><DEDENT>if '<STR_LIT>' in summary_labels:<EOL><INDENT>self.meta_param_recorder.text_output(format_type=<NUM_LIT:1>)<EOL><DEDENT><DEDENT>super(LearningAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity<EOL>)<EOL>

Initializes the learning agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes
        (required):
        - unit: one of 'timesteps', 'episodes', 'sequences' (required).
        - batch_size: integer (required).
        - frequency: integer (default: batch_size).
        - length: integer (optional if unit == 'sequences', default: 8).
    memory (spec): Memory specification, see core.memories module for more information
        (required).
    optimizer (spec): Optimizer specification, see core.optimizers module for more
        information (required).
    network (spec): Network specification, usually a list of layer specifications, see
        core.networks module for more information (required).
    scope (str): TensorFlow scope (default: name of agent).
    device: TensorFlow device (default: none)
    saver (spec): Saver specification, with the following attributes (default: none):
        - directory: model directory.
        - file: model filename (optional).
        - seconds or steps: save frequency (default: 600 seconds).
        - load: specifies whether model is loaded, if existent (default: true).
        - basename: optional file basename (default: 'model.ckpt').
    summarizer (spec): Summarizer specification, with the following attributes (default:
        none):
        - directory: summaries directory.
        - seconds or steps: summarize frequency (default: 120 seconds).
        - labels: list of summary labels to record (default: []).
        - meta_param_recorder_class: ???.
    execution (spec): Execution specification (see sanity_check_execution_spec for details).
    variable_noise (float): Standard deviation of variable noise (default: none).
    states_preprocessing (spec, or dict of specs): States preprocessing specification, see
        core.preprocessors module for more information (default: none)
    actions_exploration (spec, or dict of specs): Actions exploration specification, see
        core.explorations module for more information (default: none).
    reward_preprocessing (spec): Reward preprocessing specification, see core.preprocessors
        module for more information (default: none).
    discount (float): Discount factor for future rewards (default: 0.99).
    distributions (spec / dict of specs): Distributions specifications, see
        core.distributions module for more information (default: none).
    entropy_regularization (float): Entropy regularization weight (default: none).


if isinstance(experiences, dict):<EOL><INDENT>if self.unique_state:<EOL><INDENT>experiences['<STR_LIT>'] = dict(state=experiences['<STR_LIT>'])<EOL><DEDENT>if self.unique_action:<EOL><INDENT>experiences['<STR_LIT>'] = dict(action=experiences['<STR_LIT>'])<EOL><DEDENT>self.model.import_experience(**experiences)<EOL><DEDENT>else:<EOL><INDENT>if self.unique_state:<EOL><INDENT>states = dict(state=list())<EOL><DEDENT>else:<EOL><INDENT>states = {name: list() for name in experiences[<NUM_LIT:0>]['<STR_LIT>']}<EOL><DEDENT>internals = [list() for _ in experiences[<NUM_LIT:0>]['<STR_LIT>']]<EOL>if self.unique_action:<EOL><INDENT>actions = dict(action=list())<EOL><DEDENT>else:<EOL><INDENT>actions = {name: list() for name in experiences[<NUM_LIT:0>]['<STR_LIT>']}<EOL><DEDENT>terminal = list()<EOL>reward = list()<EOL>for experience in experiences:<EOL><INDENT>if self.unique_state:<EOL><INDENT>states['<STR_LIT:state>'].append(experience['<STR_LIT>'])<EOL><DEDENT>else:<EOL><INDENT>for name in sorted(states):<EOL><INDENT>states[name].append(experience['<STR_LIT>'][name])<EOL><DEDENT><DEDENT>for n, internal in enumerate(internals):<EOL><INDENT>internal.append(experience['<STR_LIT>'][n])<EOL><DEDENT>if self.unique_action:<EOL><INDENT>actions['<STR_LIT:action>'].append(experience['<STR_LIT>'])<EOL><DEDENT>else:<EOL><INDENT>for name in sorted(actions):<EOL><INDENT>actions[name].append(experience['<STR_LIT>'][name])<EOL><DEDENT><DEDENT>terminal.append(experience['<STR_LIT>'])<EOL>reward.append(experience['<STR_LIT>'])<EOL><DEDENT>self.model.import_experience(<EOL>states=states,<EOL>internals=internals,<EOL>actions=actions,<EOL>terminal=terminal,<EOL>reward=reward<EOL>)<EOL><DEDENT>

Imports experiences.

Args:
    experiences:


<EOL>if update_mode is None:<EOL><INDENT>update_mode = dict(<EOL>unit='<STR_LIT>',<EOL>batch_size=<NUM_LIT:10><EOL>)<EOL><DEDENT>elif '<STR_LIT>' in update_mode:<EOL><INDENT>assert update_mode['<STR_LIT>'] == '<STR_LIT>'<EOL><DEDENT>else:<EOL><INDENT>update_mode['<STR_LIT>'] = '<STR_LIT>'<EOL><DEDENT>if memory is None:<EOL><INDENT>memory = dict(<EOL>type='<STR_LIT>',<EOL>include_next_states=True,<EOL>capacity=(<NUM_LIT:1000> * update_mode['<STR_LIT>'])<EOL>)<EOL><DEDENT>else:<EOL><INDENT>assert memory['<STR_LIT>']<EOL><DEDENT>if optimizer is None:<EOL><INDENT>optimizer = dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=<NUM_LIT><EOL>)<EOL><DEDENT>if critic_optimizer is None:<EOL><INDENT>critic_optimizer = dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=<NUM_LIT><EOL>)<EOL><DEDENT>self.critic_network = critic_network<EOL>self.critic_optimizer = critic_optimizer<EOL>self.target_sync_frequency = target_sync_frequency<EOL>self.target_update_weight = target_update_weight<EOL>super(DDPGAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity,<EOL>scope=scope,<EOL>device=device,<EOL>saver=saver,<EOL>summarizer=summarizer,<EOL>execution=execution,<EOL>variable_noise=variable_noise,<EOL>states_preprocessing=states_preprocessing,<EOL>actions_exploration=actions_exploration,<EOL>reward_preprocessing=reward_preprocessing,<EOL>update_mode=update_mode,<EOL>memory=memory,<EOL>optimizer=optimizer,<EOL>discount=discount,<EOL>network=network,<EOL>distributions=distributions,<EOL>entropy_regularization=entropy_regularization<EOL>)<EOL>

Initializes the DDPG agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes:
        - unit: 'timesteps' if given (default: 'timesteps').
        - batch_size: integer (default: 10).
        - frequency: integer (default: batch_size).
    memory (spec): Memory specification, see core.memories module for more information
        (default: {type='replay', include_next_states=true, capacity=1000*batch_size}).
    optimizer (spec): Optimizer specification, see core.optimizers module for more
        information (default: {type='adam', learning_rate=1e-3}).
    critic_network (spec): Critic network specification, size_t0 and size_t1.
    critic_optimizer (spec): Critic optimizer specification, see core.optimizers module for
        more information (default: {type='adam', learning_rate=1e-3}).
    target_sync_frequency (int): Target network sync frequency (default: 10000).
    target_update_weight (float): Target network update weight (default: 1.0).


<EOL>if update_mode is None:<EOL><INDENT>update_mode = dict(<EOL>unit='<STR_LIT>',<EOL>batch_size=<NUM_LIT:10><EOL>)<EOL><DEDENT>elif '<STR_LIT>' in update_mode:<EOL><INDENT>pass<EOL><DEDENT>else:<EOL><INDENT>update_mode['<STR_LIT>'] = '<STR_LIT>'<EOL><DEDENT>if memory is None:<EOL><INDENT>memory = dict(<EOL>type='<STR_LIT>',<EOL>include_next_states=False,<EOL>capacity=(<NUM_LIT:1000> * update_mode['<STR_LIT>'])<EOL>)<EOL><DEDENT>else:<EOL><INDENT>assert not memory['<STR_LIT>']<EOL><DEDENT>assert (update_mode['<STR_LIT>'] != '<STR_LIT>' or memory['<STR_LIT:type>'] == '<STR_LIT>')<EOL>optimizer = dict(<EOL>type='<STR_LIT>',<EOL>optimizer=dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=learning_rate,<EOL>cg_max_iterations=cg_max_iterations,<EOL>cg_damping=cg_damping,<EOL>cg_unroll_loop=cg_unroll_loop,<EOL>),<EOL>ls_max_iterations=ls_max_iterations,<EOL>ls_accept_ratio=ls_accept_ratio,<EOL>ls_mode='<STR_LIT>',  <EOL>ls_parameter=<NUM_LIT:0.5>,  <EOL>ls_unroll_loop=ls_unroll_loop<EOL>)<EOL>self.baseline_mode = baseline_mode<EOL>self.baseline = baseline<EOL>self.baseline_optimizer = baseline_optimizer<EOL>self.gae_lambda = gae_lambda<EOL>self.likelihood_ratio_clipping = likelihood_ratio_clipping<EOL>super(TRPOAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity,<EOL>scope=scope,<EOL>device=device,<EOL>saver=saver,<EOL>summarizer=summarizer,<EOL>execution=execution,<EOL>variable_noise=variable_noise,<EOL>states_preprocessing=states_preprocessing,<EOL>actions_exploration=actions_exploration,<EOL>reward_preprocessing=reward_preprocessing,<EOL>update_mode=update_mode,<EOL>memory=memory,<EOL>optimizer=optimizer,<EOL>discount=discount,<EOL>network=network,<EOL>distributions=distributions,<EOL>entropy_regularization=entropy_regularization<EOL>)<EOL>

Initializes the TRPO agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes:
        - unit: 'episodes' if given (default: 'episodes').
        - batch_size: integer (default: 10).
        - frequency: integer (default: batch_size).
    memory (spec): Memory specification, see core.memories module for more information
        (default: {type='latest', include_next_states=false, capacity=1000*batch_size}).
    optimizer (spec): TRPO agent implicitly defines a optimized-step natural-gradient
        optimizer.
    baseline_mode (str): One of 'states', 'network' (default: none).
    baseline (spec): Baseline specification, see core.baselines module for more information
        (default: none).
    baseline_optimizer (spec): Baseline optimizer specification, see core.optimizers module
        for more information (default: none).
    gae_lambda (float): Lambda factor for generalized advantage estimation (default: none).
    likelihood_ratio_clipping (float): Likelihood ratio clipping for policy gradient
        (default: none).
    learning_rate (float): Learning rate of natural-gradient optimizer (default: 1e-3).
    cg_max_iterations (int): Conjugate-gradient max iterations (default: 20).
    cg_damping (float): Conjugate-gradient damping (default: 1e-3).
    cg_unroll_loop (bool): Conjugate-gradient unroll loop (default: false).
    ls_max_iterations (int): Line-search max iterations (default: 10).
    ls_accept_ratio (float): Line-search accept ratio (default: 0.9).
    ls_unroll_loop (bool): Line-search unroll loop (default: false).


<EOL>if update_mode is None:<EOL><INDENT>update_mode = dict(<EOL>unit='<STR_LIT>',<EOL>batch_size=<NUM_LIT:32>,<EOL>frequency=<NUM_LIT:4><EOL>)<EOL><DEDENT>elif '<STR_LIT>' in update_mode:<EOL><INDENT>assert update_mode['<STR_LIT>'] == '<STR_LIT>'<EOL><DEDENT>else:<EOL><INDENT>update_mode['<STR_LIT>'] = '<STR_LIT>'<EOL><DEDENT>if memory is None:<EOL><INDENT>memory = dict(<EOL>type='<STR_LIT>',<EOL>include_next_states=True,<EOL>capacity=(<NUM_LIT:1000> * update_mode['<STR_LIT>'])<EOL>)<EOL><DEDENT>else:<EOL><INDENT>assert memory['<STR_LIT>']<EOL><DEDENT>if optimizer is None:<EOL><INDENT>optimizer = dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=<NUM_LIT><EOL>)<EOL><DEDENT>self.target_sync_frequency = target_sync_frequency<EOL>self.target_update_weight = target_update_weight<EOL>self.double_q_model = double_q_model<EOL>self.huber_loss = huber_loss<EOL>super(NAFAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity,<EOL>scope=scope,<EOL>device=device,<EOL>saver=saver,<EOL>summarizer=summarizer,<EOL>execution=execution,<EOL>variable_noise=variable_noise,<EOL>states_preprocessing=states_preprocessing,<EOL>actions_exploration=actions_exploration,<EOL>reward_preprocessing=reward_preprocessing,<EOL>update_mode=update_mode,<EOL>memory=memory,<EOL>optimizer=optimizer,<EOL>discount=discount,<EOL>network=network,<EOL>distributions=distributions,<EOL>entropy_regularization=entropy_regularization<EOL>)<EOL>

Initializes the NAF agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes:
        - unit: 'timesteps' if given (default: 'timesteps').
        - batch_size: integer (default: 32).
        - frequency: integer (default: 4).
    memory (spec): Memory specification, see core.memories module for more information
        (default: {type='replay', include_next_states=true, capacity=1000*batch_size}).
    optimizer (spec): Optimizer specification, see core.optimizers module for more
        information (default: {type='adam', learning_rate=1e-3}).
    target_sync_frequency (int): Target network sync frequency (default: 10000).
    target_update_weight (float): Target network update weight (default: 1.0).
    double_q_model (bool): Specifies whether double DQN mode is used (default: false).
    huber_loss (float): Huber loss clipping (default: none).


<EOL>if update_mode is None:<EOL><INDENT>update_mode = dict(<EOL>unit='<STR_LIT>',<EOL>batch_size=<NUM_LIT:32>,<EOL>frequency=<NUM_LIT:4><EOL>)<EOL><DEDENT>elif '<STR_LIT>' in update_mode:<EOL><INDENT>assert update_mode['<STR_LIT>'] == '<STR_LIT>'<EOL><DEDENT>else:<EOL><INDENT>update_mode['<STR_LIT>'] = '<STR_LIT>'<EOL><DEDENT>if memory is None:<EOL><INDENT>memory = dict(<EOL>type='<STR_LIT>',<EOL>include_next_states=True,<EOL>capacity=(<NUM_LIT:1000> * update_mode['<STR_LIT>'])<EOL>)<EOL><DEDENT>else:<EOL><INDENT>assert memory['<STR_LIT>']<EOL><DEDENT>if optimizer is None:<EOL><INDENT>optimizer = dict(<EOL>type='<STR_LIT>',<EOL>learning_rate=<NUM_LIT><EOL>)<EOL><DEDENT>self.target_sync_frequency = target_sync_frequency<EOL>self.target_update_weight = target_update_weight<EOL>self.double_q_model = True<EOL>self.huber_loss = huber_loss<EOL>self.expert_margin = expert_margin<EOL>self.supervised_weight = supervised_weight<EOL>self.demo_memory_capacity = demo_memory_capacity<EOL>self.demo_batch_size = int(demo_sampling_ratio * update_mode['<STR_LIT>'] / (<NUM_LIT:1.0> - demo_sampling_ratio))<EOL>assert self.demo_batch_size > <NUM_LIT:0>, '<STR_LIT>''<STR_LIT>''<STR_LIT>'.format(self.demo_batch_size)<EOL>super(DQFDAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity,<EOL>scope=scope,<EOL>device=device,<EOL>saver=saver,<EOL>summarizer=summarizer,<EOL>execution=execution,<EOL>variable_noise=variable_noise,<EOL>states_preprocessing=states_preprocessing,<EOL>actions_exploration=actions_exploration,<EOL>reward_preprocessing=reward_preprocessing,<EOL>update_mode=update_mode,<EOL>memory=memory,<EOL>optimizer=optimizer,<EOL>discount=discount,<EOL>network=network,<EOL>distributions=distributions,<EOL>entropy_regularization=entropy_regularization<EOL>)<EOL>

Initializes the DQFD agent.

Args:
    update_mode (spec): Update mode specification, with the following attributes:
        - unit: 'timesteps' if given (default: 'timesteps').
        - batch_size: integer (default: 32).
        - frequency: integer (default: 4).
    memory (spec): Memory specification, see core.memories module for more information
        (default: {type='replay', include_next_states=true, capacity=1000*batch_size}).
    optimizer (spec): Optimizer specification, see core.optimizers module for more
        information (default: {type='adam', learning_rate=1e-3}).
    target_sync_frequency (int): Target network sync frequency (default: 10000).
    target_update_weight (float): Target network update weight (default: 1.0).
    huber_loss (float): Huber loss clipping (default: none).
    expert_margin (float): Enforced supervised margin between expert action Q-value and
        other Q-values (default: 0.5).
    supervised_weight (float): Weight of supervised loss term (default: 0.1).
    demo_memory_capacity (int): Capacity of expert demonstration memory (default: 10000).
    demo_sampling_ratio (float): Runtime sampling ratio of expert data (default: 0.2).


if isinstance(demonstrations, dict):<EOL><INDENT>if self.unique_state:<EOL><INDENT>demonstrations['<STR_LIT>'] = dict(state=demonstrations['<STR_LIT>'])<EOL><DEDENT>if self.unique_action:<EOL><INDENT>demonstrations['<STR_LIT>'] = dict(action=demonstrations['<STR_LIT>'])<EOL><DEDENT>self.model.import_demo_experience(**demonstrations)<EOL><DEDENT>else:<EOL><INDENT>if self.unique_state:<EOL><INDENT>states = dict(state=list())<EOL><DEDENT>else:<EOL><INDENT>states = {name: list() for name in demonstrations[<NUM_LIT:0>]['<STR_LIT>']}<EOL><DEDENT>internals = {name: list() for name in demonstrations[<NUM_LIT:0>]['<STR_LIT>']}<EOL>if self.unique_action:<EOL><INDENT>actions = dict(action=list())<EOL><DEDENT>else:<EOL><INDENT>actions = {name: list() for name in demonstrations[<NUM_LIT:0>]['<STR_LIT>']}<EOL><DEDENT>terminal = list()<EOL>reward = list()<EOL>for demonstration in demonstrations:<EOL><INDENT>if self.unique_state:<EOL><INDENT>states['<STR_LIT:state>'].append(demonstration['<STR_LIT>'])<EOL><DEDENT>else:<EOL><INDENT>for name, state in states.items():<EOL><INDENT>state.append(demonstration['<STR_LIT>'][name])<EOL><DEDENT><DEDENT>for name, internal in internals.items():<EOL><INDENT>internal.append(demonstration['<STR_LIT>'][name])<EOL><DEDENT>if self.unique_action:<EOL><INDENT>actions['<STR_LIT:action>'].append(demonstration['<STR_LIT>'])<EOL><DEDENT>else:<EOL><INDENT>for name, action in actions.items():<EOL><INDENT>action.append(demonstration['<STR_LIT>'][name])<EOL><DEDENT><DEDENT>terminal.append(demonstration['<STR_LIT>'])<EOL>reward.append(demonstration['<STR_LIT>'])<EOL><DEDENT>self.model.import_demo_experience(<EOL>states=states,<EOL>internals=internals,<EOL>actions=actions,<EOL>terminal=terminal,<EOL>reward=reward<EOL>)<EOL><DEDENT>

Imports demonstrations, i.e. expert observations. Note that for large numbers of observations,
set_demonstrations is more appropriate, which directly sets memory contents to an array an expects
a different layout.

Args:
    demonstrations: List of observation dicts


for _ in xrange(steps):<EOL><INDENT>self.model.demo_update()<EOL><DEDENT>

Computes pre-train updates.

Args:
    steps: Number of updates to execute.


self.scope = scope<EOL>self.device = device<EOL>self.saver = saver<EOL>self.summarizer = summarizer<EOL>self.execution = sanity_check_execution_spec(execution)<EOL>self.batching_capacity = batching_capacity<EOL>self.action_values = action_values<EOL>super(ConstantAgent, self).__init__(<EOL>states=states,<EOL>actions=actions,<EOL>batched_observe=batched_observe,<EOL>batching_capacity=batching_capacity<EOL>)<EOL>

Initializes the constant agent.

Args:
    action_values (value, or dict of values): Action values returned by the agent
        (required).
    scope (str): TensorFlow scope (default: name of agent).
    device: TensorFlow device (default: none)
    saver (spec): Saver specification, with the following attributes (default: none):
        - directory: model directory.
        - file: model filename (optional).
        - seconds or steps: save frequency (default: 600 seconds).
        - load: specifies whether model is loaded, if existent (default: true).
        - basename: optional file basename (default: 'model.ckpt').
    summarizer (spec): Summarizer specification, with the following attributes (default:
        none):
        - directory: summaries directory.
        - seconds or steps: summarize frequency (default: 120 seconds).
        - labels: list of summary labels to record (default: []).
        - meta_param_recorder_class: ???.
    execution (spec): Execution specification (see sanity_check_execution_spec for details).


self.sigma = sigma<EOL>self.mu = float(mu)  <EOL>super(GaussianNoise, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Initializes distribution values for gaussian noise


self.sigma = sigma<EOL>self.mu = float(mu)  <EOL>self.theta = theta<EOL>super(OrnsteinUhlenbeckProcess, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Initializes an Ornstein-Uhlenbeck process which is a mean reverting stochastic process
introducing time-correlated noise.


def tf_explore(self, episode, timestep, shape):

raise NotImplementedError<EOL>

Creates exploration value, e.g. compute an epsilon for epsilon-greedy or sample normal
noise.


return [self.variables[key] for key in sorted(self.variables)]<EOL>

Returns exploration variables.

Returns:
    List of variables.


exploration = util.get_object(<EOL>obj=spec,<EOL>predefined_objects=tensorforce.core.explorations.explorations<EOL>)<EOL>assert isinstance(exploration, Exploration)<EOL>return exploration<EOL>

Creates an exploration object from a specification dict.


def __init__(self, named_tensors=None, scope='<STR_LIT>', summary_labels=None):

self.scope = scope<EOL>self.summary_labels = set(summary_labels or ())<EOL>self.named_tensors = named_tensors<EOL>self.variables = dict()<EOL>self.all_variables = dict()<EOL>def custom_getter(getter, name, registered=False, **kwargs):<EOL><INDENT>variable = getter(name=name, registered=True, **kwargs)<EOL>if registered:<EOL><INDENT>pass<EOL><DEDENT>elif name in self.all_variables:<EOL><INDENT>assert variable is self.all_variables[name]<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>assert variable is self.variables[name]<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>self.all_variables[name] = variable<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>self.variables[name] = variable<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>return variable<EOL><DEDENT>self.apply = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_apply,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.regularization_loss = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_regularization_loss,<EOL>custom_getter_=custom_getter<EOL>)<EOL>


raise NotImplementedError<EOL>

Creates the TensorFlow operations for applying the layer to the given input.

Args:
    x: Layer input tensor.
    update: Boolean tensor indicating whether this call happens during an update.

Returns:
    Layer output tensor.


return None<EOL>

Creates the TensorFlow operations for the layer regularization loss.

Returns:
    Regularization loss tensor.


return dict()<EOL>

Returns the internal states specification.

Returns:
    Internal states specification


if include_nontrainable:<EOL><INDENT>return [self.all_variables[key] for key in sorted(self.all_variables)]<EOL><DEDENT>else:<EOL><INDENT>return [self.variables[key] for key in sorted(self.variables)]<EOL><DEDENT>

Returns the TensorFlow variables used by the layer.

Returns:
    List of variables.


layer = util.get_object(<EOL>obj=spec,<EOL>predefined_objects=tensorforce.core.networks.layers,<EOL>kwargs=kwargs<EOL>)<EOL>assert isinstance(layer, Layer)<EOL>return layer<EOL>

Creates a layer from a specification dict.


self.names = names<EOL>self.aggregation_type = aggregation_type<EOL>self.axis = axis<EOL>super(Input, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Input layer.

Args:
    names: A list of strings that name the inputs to merge
    axis: Axis to merge the inputs


self.name = name<EOL>super(Output, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Output layer.

Args:
    output: A string that names the tensor, will be added to available inputs


def __init__(self, layer, named_tensors=None, scope='<STR_LIT>', summary_labels=(), **kwargs):

self.layer_spec = layer<EOL>self.layer = util.get_object(obj=layer, predefined_objects=TFLayer.tf_layers, kwargs=kwargs)<EOL>self.first_scope = None<EOL>super(TFLayer, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Creates a new layer instance of a TensorFlow layer.

Args:
    name: The name of the layer, one of 'dense'.
    **kwargs: Additional arguments passed on to the TensorFlow layer constructor.


self.name = name<EOL>self.alpha = None<EOL>self.max = None<EOL>self.min = None<EOL>self.beta_learn = False<EOL>super(Nonlinearity, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>if max is not None:<EOL><INDENT>self.max = float(max)<EOL><DEDENT>if min is not None:<EOL><INDENT>self.min = float(min)<EOL><DEDENT>if alpha is not None:<EOL><INDENT>self.alpha = float(alpha)<EOL><DEDENT>if beta == '<STR_LIT>':<EOL><INDENT>self.beta_learn = True<EOL>self.beta = None<EOL><DEDENT>else:<EOL><INDENT>self.beta = tf.constant(float(beta), dtype=util.tf_dtype('<STR_LIT:float>'))<EOL><DEDENT>

Non-linearity activation layer.

Args:
    name: Non-linearity name, one of 'elu', 'relu', 'selu', 'sigmoid', 'swish',
        'leaky_relu' (or 'lrelu'), 'crelu', 'softmax', 'softplus', 'softsign', 'tanh' or 'none'.
    alpha: (float|int) Alpha value for leaky Relu
    beta: (float|int|'learn') Beta value or 'learn' to train value (default 1.0)
    max: (float|int) maximum (beta * input) value passed to non-linearity function
    min: (float|int) minimum (beta * input) value passed to non-linearity function
    summary_labels: Requested summary labels for tensorboard export, add 'beta' to watch beta learning


self.pooling_type = pooling_type<EOL>if isinstance(window, int):<EOL><INDENT>self.window = (<NUM_LIT:1>, window, window, <NUM_LIT:1>)<EOL><DEDENT>elif len(window) == <NUM_LIT:2>:<EOL><INDENT>self.window = (<NUM_LIT:1>, window[<NUM_LIT:0>], window[<NUM_LIT:1>], <NUM_LIT:1>)<EOL><DEDENT>else:<EOL><INDENT>raise TensorForceError('<STR_LIT>'.format(window))<EOL><DEDENT>if isinstance(stride, int):<EOL><INDENT>self.stride = (<NUM_LIT:1>, stride, stride, <NUM_LIT:1>)<EOL><DEDENT>elif len(window) == <NUM_LIT:2>:<EOL><INDENT>self.stride = (<NUM_LIT:1>, stride[<NUM_LIT:0>], stride[<NUM_LIT:1>], <NUM_LIT:1>)<EOL><DEDENT>else:<EOL><INDENT>raise TensorForceError('<STR_LIT>'.format(stride))<EOL><DEDENT>self.padding = padding<EOL>super(Pool2d, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

2-dimensional pooling layer.

Args:
    pooling_type: Either 'max' or 'average'.
    window: Pooling window size, either an integer or pair of integers.
    stride: Pooling stride, either an integer or pair of integers.
    padding: Pooling padding, one of 'VALID' or 'SAME'.


self.indices = indices<EOL>self.size = size<EOL>self.l2_regularization = l2_regularization<EOL>self.l1_regularization = l1_regularization<EOL>super(Embedding, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Embedding layer.

Args:
    indices: Number of embedding indices.
    size: Embedding size.
    l2_regularization: L2 regularization weight.
    l1_regularization: L1 regularization weight.


self.size = size<EOL>self.weights_init = weights<EOL>self.bias_init = bias<EOL>self.l2_regularization = l2_regularization<EOL>self.l1_regularization = l1_regularization<EOL>self.trainable = trainable<EOL>super(Linear, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Linear layer.

Args:
    size: Layer size.
    weights: Weight initialization, random if None.
    bias: Bias initialization, random if True, no bias added if False.
    l2_regularization: L2 regularization weight.
    l1_regularization: L1 regularization weight.


self.skip = skip<EOL>if self.skip and size is not None:<EOL><INDENT>raise TensorForceError(<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>)<EOL><DEDENT>self.linear = Linear(<EOL>size=size,<EOL>weights=weights,<EOL>bias=bias,<EOL>l2_regularization=l2_regularization,<EOL>l1_regularization=l1_regularization,<EOL>summary_labels=summary_labels,<EOL>trainable=trainable<EOL>)<EOL>if self.skip:<EOL><INDENT>self.linear_skip = Linear(<EOL>size=size,<EOL>bias=bias,<EOL>l2_regularization=l2_regularization,<EOL>l1_regularization=l1_regularization,<EOL>summary_labels=summary_labels,<EOL>trainable=trainable<EOL>)<EOL><DEDENT>self.nonlinearity = Nonlinearity(summary_labels=summary_labels, **util.prepare_kwargs(activation))<EOL>super(Dense, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Dense layer.

Args:
    size: Layer size, if None than input size matches the output size of the layer
    weights: Weight initialization, random if None.
    bias: If true, bias is added.
    activation: Type of nonlinearity, or dict with name & arguments
    l2_regularization: L2 regularization weight.
    l1_regularization: L1 regularization weight.
    skip: Add skip connection like ResNet (https://arxiv.org/pdf/1512.03385.pdf),
          doubles layers and ShortCut from Input to output


<EOL>self.expectation_layer = Linear(<EOL>size=<NUM_LIT:1>, bias=bias,<EOL>l2_regularization=l2_regularization,<EOL>l1_regularization=l1_regularization,<EOL>summary_labels=summary_labels,<EOL>)<EOL>self.advantage_layer = Linear(<EOL>size=size,<EOL>bias=bias,<EOL>l2_regularization=l2_regularization,<EOL>l1_regularization=l1_regularization,<EOL>summary_labels=summary_labels,<EOL>)<EOL>self.output = output<EOL>self.nonlinearity = Nonlinearity(summary_labels=summary_labels, **util.prepare_kwargs(activation))<EOL>super(Dueling, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

Dueling layer.

[Dueling Networks] (https://arxiv.org/pdf/1511.06581.pdf)
Implement Y = Expectation[x] + (Advantage[x] - Mean(Advantage[x]))

Args:
    size: Layer size.
    bias: If true, bias is added.
    activation: Type of nonlinearity, or dict with name & arguments
    l2_regularization: L2 regularization weight.
    l1_regularization: L1 regularization weight.
    output: None or tuple of output names for ('expectation','advantage','mean_advantage')


self.size = size<EOL>self.window = window<EOL>self.stride = stride<EOL>self.padding = padding<EOL>self.bias = bias<EOL>self.l2_regularization = l2_regularization<EOL>self.l1_regularization = l1_regularization<EOL>self.nonlinearity = Nonlinearity(summary_labels=summary_labels, **util.prepare_kwargs(activation))<EOL>super(Conv1d, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

1D convolutional layer.

Args:
    size: Number of filters
    window: Convolution window size
    stride: Convolution stride
    padding: Convolution padding, one of 'VALID' or 'SAME'
    bias: If true, a bias is added
    activation: Type of nonlinearity, or dict with name & arguments
    l2_regularization: L2 regularization weight
    l1_regularization: L1 regularization weight


self.size = size<EOL>if isinstance(window, int):<EOL><INDENT>self.window = (window, window)<EOL><DEDENT>elif len(window) == <NUM_LIT:2>:<EOL><INDENT>self.window = tuple(window)<EOL><DEDENT>else:<EOL><INDENT>raise TensorForceError('<STR_LIT>'.format(window))<EOL><DEDENT>self.stride = stride<EOL>self.padding = padding<EOL>self.bias = bias<EOL>self.l2_regularization = l2_regularization<EOL>self.l1_regularization = l1_regularization<EOL>self.nonlinearity = Nonlinearity(summary_labels=summary_labels, **util.prepare_kwargs(activation))<EOL>super(Conv2d, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

2D convolutional layer.

Args:
    size: Number of filters
    window: Convolution window size, either an integer or pair of integers.
    stride: Convolution stride, either an integer or pair of integers.
    padding: Convolution padding, one of 'VALID' or 'SAME'
    bias: If true, a bias is added
    activation: Type of nonlinearity, or dict with name & arguments
    l2_regularization: L2 regularization weight
    l1_regularization: L1 regularization weight


def __init__(self, size, dropout=None, lstmcell_args={}, named_tensors=None, scope='<STR_LIT>', summary_labels=()):

self.size = size<EOL>self.dropout = dropout<EOL>self.lstmcell_args = lstmcell_args<EOL>super(InternalLstm, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>

LSTM layer.

Args:
    size: LSTM size.
    dropout: Dropout rate.


def __init__(self, size, dropout=None, named_tensors=None, scope='<STR_LIT>', summary_labels=(), return_final_state=True):

self.size = size<EOL>self.dropout = dropout<EOL>self.return_final_state = return_final_state<EOL>super(Lstm, self).__init__(named_tensors=named_tensors, scope=scope, summary_labels=summary_labels)<EOL>


def __init__(self, scope='<STR_LIT>', summary_labels=None):

self.summary_labels = set(summary_labels or ())<EOL>self.variables = dict()<EOL>self.all_variables = dict()<EOL>self.named_tensors = dict()<EOL>def custom_getter(getter, name, registered=False, **kwargs):<EOL><INDENT>variable = getter(name=name, registered=True, **kwargs)<EOL>if registered:<EOL><INDENT>pass<EOL><DEDENT>elif name in self.all_variables:<EOL><INDENT>assert variable is self.all_variables[name]<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>assert variable is self.variables[name]<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>self.all_variables[name] = variable<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>self.variables[name] = variable<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>return variable<EOL><DEDENT>self.apply = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_apply,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.regularization_loss = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_regularization_loss,<EOL>custom_getter_=custom_getter<EOL>)<EOL>


def tf_apply(self, x, internals, update, return_internals=False):

raise NotImplementedError<EOL>

Creates the TensorFlow operations for applying the network to the given input.

Args:
    x: Network input tensor or dict of input tensors.
    internals: List of prior internal state tensors
    update: Boolean tensor indicating whether this call happens during an update.
    return_internals: If true, also returns posterior internal state tensors

Returns:
    Network output tensor, plus optionally list of posterior internal state tensors


return None<EOL>

Creates the TensorFlow operations for the network regularization loss.

Returns:
    Regularization loss tensor


return dict()<EOL>

Returns the internal states specification.

Returns:
    Internal states specification


Returns the TensorFlow variables used by the network.

Returns:
    List of variables


if name in self.named_tensors:<EOL><INDENT>return True, self.named_tensors[name]<EOL><DEDENT>else:<EOL><INDENT>return False, None<EOL><DEDENT>

Returns a named tensor if available.

Returns:
    valid: True if named tensor found, False otherwise
    tensor: If valid, will be a tensor, otherwise None


return list(self.named_tensors)<EOL>

Returns a list of the names of tensors available.

Returns:
    List of the names of tensors available.


network = util.get_object(<EOL>obj=spec,<EOL>default_object=LayeredNetwork,<EOL>kwargs=kwargs<EOL>)<EOL>assert isinstance(network, Network)<EOL>return network<EOL>

Creates a network from a specification dict.


def __init__(self, scope='<STR_LIT>', summary_labels=()):

super(LayerBasedNetwork, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>self.layers = list()<EOL>


def __init__(self, layers, scope='<STR_LIT>', summary_labels=()):

self.layers_spec = layers<EOL>super(LayeredNetwork, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>self.parse_layer_spec(layer_spec=self.layers_spec, layer_counter=Counter())<EOL>

Single-stack layered network.

Args:
    layers: List of layer specification dicts.


pass<EOL>

Resets this preprocessor to some initial state. This method is called whenever an episode ends.
This could be useful if the preprocessor stores certain episode-sequence information to do the processing
and this information has to be reset after the episode terminates.


return tensor<EOL>

Process state (tensor).

Args:
    tensor (tf.Tensor): The Tensor to process.

Returns: The pre-processed Tensor.


return shape<EOL>

Shape of preprocessed state given original shape.

Args:
    shape (tuple): The original (unprocessed) shape.

Returns: The processed tensor shape.


return [self.variables[key] for key in sorted(self.variables)]<EOL>

Returns the TensorFlow variables used by the preprocessor.

Returns:
    List of variables.


fetches = []<EOL>for processor in self.preprocessors:<EOL><INDENT>fetches.extend(processor.reset() or [])<EOL><DEDENT>return fetches<EOL>

Calls `reset` on all our Preprocessor objects.

Returns:
    A list of tensors to be fetched.


for processor in self.preprocessors:<EOL><INDENT>tensor = processor.process(tensor=tensor)<EOL><DEDENT>return tensor<EOL>

Process state.

Args:
    tensor: tensor to process

Returns: processed state


for processor in self.preprocessors:<EOL><INDENT>shape = processor.processed_shape(shape=shape)<EOL><DEDENT>return shape<EOL>

Shape of preprocessed state given original shape.

Args:
    shape: original state shape

Returns: processed state shape


if isinstance(spec, dict):<EOL><INDENT>spec = [spec]<EOL><DEDENT>stack = PreprocessorStack()<EOL>for preprocessor_spec in spec:<EOL><INDENT>preprocessor_kwargs = copy.deepcopy(kwargs)<EOL>preprocessor = util.get_object(<EOL>obj=preprocessor_spec,<EOL>predefined_objects=tensorforce.core.preprocessors.preprocessors,<EOL>kwargs=preprocessor_kwargs<EOL>)<EOL>assert isinstance(preprocessor, Preprocessor)<EOL>stack.preprocessors.append(preprocessor)<EOL><DEDENT>return stack<EOL>

Creates a preprocessing stack from a specification dict.


def __init__(self, shape, length=<NUM_LIT:2>, add_rank=False, scope='<STR_LIT>', summary_labels=()):

<EOL>self.length = length<EOL>self.add_rank = add_rank<EOL>self.reset_op = None<EOL>super(Sequence, self).__init__(shape=shape, scope=scope, summary_labels=summary_labels)<EOL>

Args:
    length (int): The number of states to concatenate. In the beginning, when no previous state is available,
        concatenate the given first state with itself `length` times.
    add_rank (bool): Whether to add another rank to the end of the input with dim=length-of-the-sequence.
        This could be useful if e.g. a grayscale image of w x h pixels is coming from the env
        (no color channel). The output of the preprocessor would then be of shape [batch] x w x h x [length].


def __init__(self, shape, weights=(<NUM_LIT>, <NUM_LIT>, <NUM_LIT>), remove_rank=False, scope='<STR_LIT>', summary_labels=()):

self.weights = weights<EOL>self.remove_rank = remove_rank<EOL>super(Grayscale, self).__init__(shape=shape, scope=scope, summary_labels=summary_labels)<EOL>

Args:
    weights (tuple): The weights to multiply each color channel with (in order: red, blue, green).
    remove_rank (bool): If True, will remove the color channel rank from the input tensor.


def __init__(self, network, scope='<STR_LIT>', summary_labels=()):

self.network = Network.from_spec(<EOL>spec=network,<EOL>kwargs=dict(summary_labels=summary_labels)<EOL>)<EOL>assert len(self.network.internals_spec()) == <NUM_LIT:0><EOL>self.linear = Linear(size=<NUM_LIT:1>, bias=<NUM_LIT:0.0>, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>super(NetworkBaseline, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Network baseline.

Args:
    network_spec: Network specification dict


def __init__(self, conv_sizes, dense_sizes, scope='<STR_LIT>', summary_labels=()):

network = []<EOL>for size in conv_sizes:<EOL><INDENT>network.append(dict(type='<STR_LIT>', size=size))<EOL><DEDENT>network[<NUM_LIT:0>]['<STR_LIT>'] = <NUM_LIT:5><EOL>network.append(dict(type='<STR_LIT>'))  <EOL>for size in dense_sizes:<EOL><INDENT>network.append(dict(type='<STR_LIT>', size=size))<EOL><DEDENT>super(CNNBaseline, self).__init__(network=network, scope=scope, summary_labels=summary_labels)<EOL>

CNN baseline.

Args:
    conv_sizes: List of convolutional layer sizes
    dense_sizes: List of dense layer sizes


self.summary_labels = set(summary_labels or ())<EOL>self.variables = dict()<EOL>self.all_variables = dict()<EOL>def custom_getter(getter, name, registered=False, **kwargs):<EOL><INDENT>variable = getter(name=name, registered=True, **kwargs)<EOL>if registered:<EOL><INDENT>pass<EOL><DEDENT>elif name in self.all_variables:<EOL><INDENT>assert variable is self.all_variables[name]<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>assert variable is self.variables[name]<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>self.all_variables[name] = variable<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>self.variables[name] = variable<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>return variable<EOL><DEDENT>self.predict = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_predict,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.reference = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_reference,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.loss = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_loss,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.regularization_loss = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_regularization_loss,<EOL>custom_getter_=custom_getter<EOL>)<EOL>


def tf_predict(self, states, internals, update):

raise NotImplementedError<EOL>

Creates the TensorFlow operations for predicting the value function of given states.
Args:
    states: Dict of state tensors.
    internals: List of prior internal state tensors.
    update: Boolean tensor indicating whether this call happens during an update.
Returns:
    State value tensor


def tf_reference(self, states, internals, reward, update):

return None<EOL>

Creates the TensorFlow operations for obtaining the reference tensor(s), in case of a
comparative loss.

Args:
    states: Dict of state tensors.
    internals: List of prior internal state tensors.
    reward: Reward tensor.
    update: Boolean tensor indicating whether this call happens during an update.

Returns:
    Reference tensor(s).


def tf_loss(self, states, internals, reward, update, reference=None):

prediction = self.predict(states=states, internals=internals, update=update)<EOL>return tf.nn.l2_loss(t=(prediction - reward))<EOL>

Creates the TensorFlow operations for calculating the L2 loss between predicted
state values and actual rewards.

Args:
    states: Dict of state tensors.
    internals: List of prior internal state tensors.
    reward: Reward tensor.
    update: Boolean tensor indicating whether this call happens during an update.
    reference: Optional reference tensor(s), in case of a comparative loss.

Returns:
    Loss tensor


return None<EOL>

Creates the TensorFlow operations for the baseline regularization loss/

Returns:
    Regularization loss tensor


if include_nontrainable:<EOL><INDENT>return [self.all_variables[key] for key in sorted(self.all_variables)]<EOL><DEDENT>else:<EOL><INDENT>return [self.variables[key] for key in sorted(self.variables)]<EOL><DEDENT>

Returns the TensorFlow variables used by the baseline.

Returns:
    List of variables


baseline = util.get_object(<EOL>obj=spec,<EOL>predefined_objects=tensorforce.core.baselines.baselines,<EOL>kwargs=kwargs<EOL>)<EOL>assert isinstance(baseline, Baseline)<EOL>return baseline<EOL>

Creates a baseline from a specification dict.


def __init__(self, sizes, scope='<STR_LIT>', summary_labels=()):

network = []<EOL>for size in sizes:<EOL><INDENT>network.append(dict(type='<STR_LIT>', size=size))<EOL><DEDENT>super(MLPBaseline, self).__init__(network=network, scope=scope, summary_labels=summary_labels)<EOL>

Multi-layer perceptron baseline.

Args:
    sizes: List of dense layer sizes


def __init__(self, baselines, scope='<STR_LIT>', summary_labels=()):

self.baselines = dict()<EOL>for name in sorted(baselines):<EOL><INDENT>self.baselines[name] = Baseline.from_spec(<EOL>spec=baselines[name],<EOL>kwargs=dict(summary_labels=summary_labels))<EOL><DEDENT>self.linear = Linear(size=<NUM_LIT:1>, bias=<NUM_LIT:0.0>, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>super(AggregatedBaseline, self).__init__(scope, summary_labels)<EOL>

Aggregated baseline.

Args:
    baselines: Dict of per-state baseline specification dicts


def __init__(self, shape, probability=<NUM_LIT:0.5>, scope='<STR_LIT>', summary_labels=()):

self.shape = shape<EOL>action_size = util.prod(self.shape)<EOL>self.logit = Linear(size=action_size, bias=log(probability), scope='<STR_LIT>', summary_labels=summary_labels)<EOL>super(Bernoulli, self).__init__(shape=shape, scope=scope, summary_labels=summary_labels)<EOL>

Bernoulli distribution.

Args:
    shape: Action shape.
    probability: Optional distribution bias.


def __init__(self, shape, min_value, max_value, alpha=<NUM_LIT:0.0>, beta=<NUM_LIT:0.0>, scope='<STR_LIT>', summary_labels=()):

assert min_value is None or max_value > min_value<EOL>self.shape = shape<EOL>self.min_value = min_value<EOL>self.max_value = max_value<EOL>action_size = util.prod(self.shape)<EOL>self.alpha = Linear(size=action_size, bias=alpha, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>self.beta = Linear(size=action_size, bias=beta, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>super(Beta, self).__init__(shape=shape, scope=scope, summary_labels=summary_labels)<EOL>

Beta distribution.

Args:
    shape: Action shape.
    min_value: Minimum value of continuous actions.
    max_value: Maximum value of continuous actions.
    alpha: Optional distribution bias for the alpha value.
    beta: Optional distribution bias for the beta value.


def __init__(self, shape, num_actions, probabilities=None, scope='<STR_LIT>', summary_labels=()):

self.num_actions = num_actions<EOL>action_size = util.prod(shape) * self.num_actions<EOL>if probabilities is None:<EOL><INDENT>logits = <NUM_LIT:0.0><EOL><DEDENT>else:<EOL><INDENT>logits = [log(prob) for _ in range(util.prod(shape)) for prob in probabilities]<EOL><DEDENT>self.logits = Linear(size=action_size, bias=logits, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>super(Categorical, self).__init__(shape=shape, scope=scope, summary_labels=summary_labels)<EOL>

Categorical distribution.

Args:
    shape: Action shape.
    num_actions: Number of discrete action alternatives.
    probabilities: Optional distribution bias.


def __init__(self, shape, scope='<STR_LIT>', summary_labels=None):

self.shape = shape<EOL>self.scope = scope<EOL>self.summary_labels = set(summary_labels or ())<EOL>self.variables = dict()<EOL>self.all_variables = dict()<EOL>def custom_getter(getter, name, registered=False, **kwargs):<EOL><INDENT>variable = getter(name=name, registered=True, **kwargs)<EOL>if registered:<EOL><INDENT>pass<EOL><DEDENT>elif name in self.all_variables:<EOL><INDENT>assert variable is self.all_variables[name]<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>assert variable is self.variables[name]<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>self.all_variables[name] = variable<EOL>if kwargs.get('<STR_LIT>', True):<EOL><INDENT>self.variables[name] = variable<EOL>if '<STR_LIT>' in self.summary_labels:<EOL><INDENT>tf.contrib.summary.histogram(name=name, tensor=variable)<EOL><DEDENT><DEDENT><DEDENT>return variable<EOL><DEDENT>self.parameterize = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_parameterize,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.sample = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_sample,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.log_probability = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_log_probability,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.entropy = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_entropy,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.kl_divergence = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_kl_divergence,<EOL>custom_getter_=custom_getter<EOL>)<EOL>self.regularization_loss = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_regularization_loss,<EOL>custom_getter_=custom_getter<EOL>)<EOL>

Distribution.

Args:
    shape: Action shape.


raise NotImplementedError<EOL>

Creates the tensorFlow operations for parameterizing a distribution conditioned on the
given input.

Args:
    x: Input tensor which the distribution is conditioned on.

Returns:
    tuple of distribution parameter tensors.


def tf_sample(self, distr_params, deterministic):

raise NotImplementedError<EOL>

Creates the tensorFlow operations for sampling an action based on a distribution.

Args:
    distr_params: tuple of distribution parameter tensors.
    deterministic: Boolean input tensor indicating whether the maximum likelihood action
        should be returned.

Returns:
    Sampled action tensor.


def tf_log_probability(self, distr_params, action):

raise NotImplementedError<EOL>

Creates the tensorFlow operations for calculating the log probability of an action for a
distribution.

Args:
    distr_params: tuple of distribution parameter tensors.
    action: Action tensor.

Returns:
    KL divergence tensor.


raise NotImplementedError<EOL>

Creates the tensorFlow operations for calculating the entropy of a distribution.

Args:
    distr_params: tuple of distribution parameter tensors.

Returns:
    Entropy tensor.


def tf_kl_divergence(self, distr_params1, distr_params2):

raise NotImplementedError<EOL>

Creates the tensorFlow operations for calculating the KL divergence between two
distributions.

Args:
    distr_params1: tuple of parameter tensors for first distribution.
    distr_params2: tuple of parameter tensors for second distribution.

Returns:
    KL divergence tensor.


return None<EOL>

Creates the tensorFlow operations for the distribution regularization loss.

Returns:
    Regularization loss tensor.


Returns the tensorFlow variables used by the distribution.

Returns:
    List of variables.


distribution = util.get_object(<EOL>obj=spec,<EOL>predefined_objects=tensorforce.core.distributions.distributions,<EOL>kwargs=kwargs<EOL>)<EOL>assert isinstance(distribution, Distribution)<EOL>return distribution<EOL>

Creates a distribution from a specification dict.


def __init__(self, shape, mean=<NUM_LIT:0.0>, log_stddev=<NUM_LIT:0.0>, scope='<STR_LIT>', summary_labels=()):

self.shape = shape<EOL>action_size = util.prod(self.shape)<EOL>self.mean = Linear(size=action_size, bias=mean, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>self.log_stddev = Linear(size=action_size, bias=log_stddev, scope='<STR_LIT>', summary_labels=summary_labels)<EOL>super(Gaussian, self).__init__(shape=shape, scope=scope, summary_labels=summary_labels)<EOL>

Categorical distribution.

Args:
    shape: Action shape.
    mean: Optional distribution bias for the mean.
    log_stddev: Optional distribution bias for the standard deviation.


def __init__(self, scope='<STR_LIT>', summary_labels=None):

self.summary_labels = set(summary_labels or ())<EOL>self.variables = dict()<EOL>def custom_getter(getter, name, registered=False, **kwargs):<EOL><INDENT>variable = getter(name=name, registered=True, **kwargs)<EOL>if registered:<EOL><INDENT>pass<EOL><DEDENT>elif name in self.variables:<EOL><INDENT>assert variable is self.variables[name]<EOL><DEDENT>else:<EOL><INDENT>assert not kwargs['<STR_LIT>']<EOL>self.variables[name] = variable<EOL><DEDENT>return variable<EOL><DEDENT>self.step = tf.make_template(<EOL>name_=(scope + '<STR_LIT>'),<EOL>func_=self.tf_step,<EOL>custom_getter=custom_getter<EOL>)<EOL>


def tf_step(self, time, variables, **kwargs):

raise NotImplementedError<EOL>

Creates the TensorFlow operations for performing an optimization step on the given variables, including
actually changing the values of the variables.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    **kwargs: Additional arguments depending on the specific optimizer implementation.
        For instance, often includes `fn_loss` if a loss function is optimized.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def apply_step(self, variables, deltas):

if len(variables) != len(deltas):<EOL><INDENT>raise TensorForceError("<STR_LIT>")<EOL><DEDENT>return tf.group(<EOL>*(tf.assign_add(ref=variable, value=delta) for variable, delta in zip(variables, deltas))<EOL>)<EOL>

Applies the given (and already calculated) step deltas to the variable values.

Args:
    variables: List of variables.
    deltas: List of deltas of same length.

Returns:
    The step-applied operation. A tf.group of tf.assign_add ops.


def minimize(self, time, variables, **kwargs):

<EOL>deltas = self.step(time=time, variables=variables, **kwargs)<EOL>with tf.control_dependencies(control_inputs=deltas):<EOL><INDENT>return tf.no_op()<EOL><DEDENT>

Performs an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    **kwargs: Additional optimizer-specific arguments. The following arguments are used
        by some optimizers:
    - arguments: Dict of arguments for callables, like fn_loss.
    - fn_loss: A callable returning the loss of the current model.
    - fn_reference: A callable returning the reference values, in case of a comparative
        loss.
    - fn_kl_divergence: A callable returning the KL-divergence relative to the
        current model.
    - sampled_loss: A sampled loss (integer).
    - return_estimated_improvement: Returns the estimated improvement resulting from
        the natural gradient calculation if true.
    - source_variables: List of source variables to synchronize with.
    - global_variables: List of global variables to apply the proposed optimization
        step to.

Returns:
    The optimization operation.


return [self.variables[key] for key in sorted(self.variables)]<EOL>

Returns the TensorFlow variables used by the optimizer.

Returns:
    List of variables.


optimizer = util.get_object(<EOL>obj=spec,<EOL>predefined_objects=tensorforce.core.optimizers.optimizers,<EOL>kwargs=kwargs<EOL>)<EOL>assert isinstance(optimizer, Optimizer)<EOL>return optimizer<EOL>

Creates an optimizer from a specification dict.


def __init__(self, optimizer, scope='<STR_LIT>', summary_labels=(), **kwargs):

self.optimizer = Optimizer.from_spec(spec=optimizer, kwargs=kwargs)<EOL>super(MetaOptimizer, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Creates a new meta optimizer instance.

Args:
    optimizer: The optimizer which is modified by this meta optimizer.


def __init__(self, optimizer, scope=None, summary_labels=(), **kwargs):

self.tf_optimizer_type = optimizer<EOL>self.tf_optimizer = TFOptimizer.tf_optimizers[optimizer](**kwargs)<EOL>super(TFOptimizer, self).__init__(scope=(scope or optimizer), summary_labels=summary_labels)<EOL>

Creates a new optimizer instance of a TensorFlow optimizer.

Args:
    optimizer: The name of the optimizer. Must be one of the keys of the tf_optimizers dict.
    **kwargs: Arguments passed on to the TensorFlow optimizer constructor as **kwargs.


arguments = kwargs["<STR_LIT>"]<EOL>fn_loss = kwargs["<STR_LIT>"]<EOL>loss = fn_loss(**arguments)<EOL>with tf.control_dependencies(control_inputs=(loss,)):<EOL><INDENT>previous_variables = [variable + <NUM_LIT:0.0> for variable in variables]<EOL><DEDENT>with tf.control_dependencies(control_inputs=previous_variables):<EOL><INDENT>applied = self.tf_optimizer.minimize(loss=loss, var_list=variables)  <EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>return [<EOL>variable - previous_variable<EOL>for variable, previous_variable in zip(variables, previous_variables)<EOL>]<EOL><DEDENT>

Keyword Args:
    arguments: Dict of arguments for passing to fn_loss as **kwargs.
    fn_loss: A callable taking arguments as kwargs and returning the loss op of the current model.


def __init__(self, optimizer, fraction=<NUM_LIT:0.1>, scope='<STR_LIT>', summary_labels=()):

assert isinstance(fraction, float) and fraction > <NUM_LIT:0.0><EOL>self.fraction = fraction<EOL>super(SubsamplingStep, self).__init__(optimizer=optimizer, scope=scope, summary_labels=summary_labels)<EOL>

Creates a new subsampling-step meta optimizer instance.

Args:
    optimizer: The optimizer which is modified by this meta optimizer.
    fraction: The fraction of instances of the batch to subsample.


<EOL>arguments_iter = iter(arguments.values())<EOL>some_argument = next(arguments_iter)<EOL>try:<EOL><INDENT>while not isinstance(some_argument, tf.Tensor) or util.rank(some_argument) == <NUM_LIT:0>:<EOL><INDENT>if isinstance(some_argument, dict):<EOL><INDENT>if some_argument:<EOL><INDENT>arguments_iter = iter(some_argument.values())<EOL><DEDENT>some_argument = next(arguments_iter)<EOL><DEDENT>elif isinstance(some_argument, list):<EOL><INDENT>if some_argument:<EOL><INDENT>arguments_iter = iter(some_argument)<EOL><DEDENT>some_argument = next(arguments_iter)<EOL><DEDENT>elif some_argument is None or util.rank(some_argument) == <NUM_LIT:0>:<EOL><INDENT>some_argument = next(arguments_iter)<EOL><DEDENT>else:<EOL><INDENT>raise TensorForceError("<STR_LIT>")<EOL><DEDENT><DEDENT><DEDENT>except StopIteration:<EOL><INDENT>raise TensorForceError("<STR_LIT>")<EOL><DEDENT>batch_size = tf.shape(input=some_argument)[<NUM_LIT:0>]<EOL>num_samples = tf.cast(<EOL>x=(self.fraction * tf.cast(x=batch_size, dtype=util.tf_dtype('<STR_LIT:float>'))),<EOL>dtype=util.tf_dtype('<STR_LIT:int>')<EOL>)<EOL>num_samples = tf.maximum(x=num_samples, y=<NUM_LIT:1>)<EOL>indices = tf.random_uniform(shape=(num_samples,), maxval=batch_size, dtype=tf.int32)<EOL>subsampled_arguments = util.map_tensors(<EOL>fn=(lambda arg: arg if util.rank(arg) == <NUM_LIT:0> else tf.gather(params=arg, indices=indices)),<EOL>tensors=arguments<EOL>)<EOL>return self.optimizer.step(<EOL>time=time,<EOL>variables=variables,<EOL>arguments=subsampled_arguments,<EOL>**kwargs<EOL>)<EOL>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    arguments: Dict of arguments for callables, like fn_loss.
    **kwargs: Additional arguments passed on to the internal optimizer.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def __init__(self, optimizer, num_steps=<NUM_LIT:10>, unroll_loop=False, scope='<STR_LIT>', summary_labels=()):

assert isinstance(num_steps, int) and num_steps > <NUM_LIT:0><EOL>self.num_steps = num_steps<EOL>assert isinstance(unroll_loop, bool)<EOL>self.unroll_loop = unroll_loop<EOL>super(MultiStep, self).__init__(optimizer=optimizer, scope=scope, summary_labels=summary_labels)<EOL>

Creates a new multi-step meta optimizer instance.

Args:
    optimizer: The optimizer which is modified by this meta optimizer.
    num_steps: Number of optimization steps to perform.


def tf_step(self, time, variables, arguments, fn_reference=None, **kwargs):

<EOL>arguments['<STR_LIT>'] = fn_reference(**arguments)<EOL>deltas = self.optimizer.step(time=time, variables=variables, arguments=arguments, **kwargs)<EOL>if self.unroll_loop:<EOL><INDENT>for _ in xrange(self.num_steps - <NUM_LIT:1>):<EOL><INDENT>with tf.control_dependencies(control_inputs=deltas):<EOL><INDENT>step_deltas = self.optimizer.step(time=time, variables=variables, arguments=arguments, **kwargs)<EOL>deltas = [delta1 + delta2 for delta1, delta2 in zip(deltas, step_deltas)]<EOL><DEDENT><DEDENT>return deltas<EOL><DEDENT>else:<EOL><INDENT>def body(iteration, deltas):<EOL><INDENT>with tf.control_dependencies(control_inputs=deltas):<EOL><INDENT>step_deltas = self.optimizer.step(time=time, variables=variables, arguments=arguments, **kwargs)<EOL>deltas = [delta1 + delta2 for delta1, delta2 in zip(deltas, step_deltas)]<EOL>return iteration + <NUM_LIT:1>, deltas<EOL><DEDENT><DEDENT>def cond(iteration, deltas):<EOL><INDENT>return iteration < self.num_steps - <NUM_LIT:1><EOL><DEDENT>_, deltas = tf.while_loop(cond=cond, body=body, loop_vars=(<NUM_LIT:0>, deltas))<EOL>return deltas<EOL><DEDENT>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    arguments: Dict of arguments for callables, like fn_loss.
    fn_reference: A callable returning the reference values, in case of a comparative loss.
    **kwargs: Additional arguments passed on to the internal optimizer.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def __init__(self, learning_rate, num_samples=<NUM_LIT:1>, unroll_loop=False, scope='<STR_LIT>', summary_labels=()):

assert isinstance(learning_rate, float) and learning_rate > <NUM_LIT:0.0><EOL>self.learning_rate = learning_rate<EOL>assert isinstance(num_samples, int) and num_samples > <NUM_LIT:0><EOL>self.num_samples = num_samples<EOL>assert isinstance(unroll_loop, bool)<EOL>self.unroll_loop = unroll_loop<EOL>super(Evolutionary, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Creates a new evolutionary optimizer instance.

Args:
    learning_rate: Learning rate.
    num_samples: Number of sampled perturbations.


unperturbed_loss = fn_loss(**arguments)<EOL>perturbations = [tf.random_normal(shape=util.shape(variable)) * self.learning_rate for variable in variables]<EOL>applied = self.apply_step(variables=variables, deltas=perturbations)<EOL>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>perturbed_loss = fn_loss(**arguments)<EOL>direction = tf.sign(x=(unperturbed_loss - perturbed_loss))<EOL>deltas_sum = [direction * perturbation for perturbation in perturbations]<EOL><DEDENT>if self.unroll_loop:<EOL><INDENT>previous_perturbations = perturbations<EOL>for sample in xrange(self.num_samples):<EOL><INDENT>with tf.control_dependencies(control_inputs=deltas_sum):<EOL><INDENT>perturbations = [tf.random_normal(shape=util.shape(variable)) * self.learning_rate for variable in variables]<EOL>perturbation_deltas = [<EOL>pert - prev_pert for pert, prev_pert in zip(perturbations, previous_perturbations)<EOL>]<EOL>applied = self.apply_step(variables=variables, deltas=perturbation_deltas)<EOL>previous_perturbations = perturbations<EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>perturbed_loss = fn_loss(**arguments)<EOL>direction = tf.sign(x=(unperturbed_loss - perturbed_loss))<EOL>deltas_sum = [delta + direction * perturbation for delta, perturbation in zip(deltas_sum, perturbations)]<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>def body(iteration, deltas_sum, previous_perturbations):<EOL><INDENT>with tf.control_dependencies(control_inputs=deltas_sum):<EOL><INDENT>perturbations = [tf.random_normal(shape=util.shape(variable)) * self.learning_rate for variable in variables]<EOL>perturbation_deltas = [<EOL>pert - prev_pert for pert, prev_pert in zip(perturbations, previous_perturbations)<EOL>]<EOL>applied = self.apply_step(variables=variables, deltas=perturbation_deltas)<EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>perturbed_loss = fn_loss(**arguments)<EOL>direction = tf.sign(x=(unperturbed_loss - perturbed_loss))<EOL>deltas_sum = [delta + direction * perturbation for delta, perturbation in zip(deltas_sum, perturbations)]<EOL><DEDENT>return iteration + <NUM_LIT:1>, deltas_sum, perturbations<EOL><DEDENT>def cond(iteration, deltas_sum, previous_perturbation):<EOL><INDENT>return iteration < self.num_samples - <NUM_LIT:1><EOL><DEDENT>_, deltas_sum, perturbations = tf.while_loop(cond=cond, body=body, loop_vars=(<NUM_LIT:0>, deltas_sum, perturbations))<EOL><DEDENT>with tf.control_dependencies(control_inputs=deltas_sum):<EOL><INDENT>deltas = [delta / self.num_samples for delta in deltas_sum]<EOL>perturbation_deltas = [delta - pert for delta, pert in zip(deltas, perturbations)]<EOL>applied = self.apply_step(variables=variables, deltas=perturbation_deltas)<EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>return [delta + <NUM_LIT:0.0> for delta in deltas]<EOL><DEDENT>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    arguments: Dict of arguments for callables, like fn_loss.
    fn_loss: A callable returning the loss of the current model.
    **kwargs: Additional arguments, not used.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def __init__(self, sync_frequency=<NUM_LIT:1>, update_weight=<NUM_LIT:1.0>, scope='<STR_LIT>', summary_labels=()):

assert isinstance(sync_frequency, int) and sync_frequency > <NUM_LIT:0><EOL>self.sync_frequency = sync_frequency<EOL>assert isinstance(update_weight, float) and update_weight > <NUM_LIT:0.0><EOL>self.update_weight = update_weight<EOL>super(Synchronization, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Creates a new synchronization optimizer instance.

Args:
    sync_frequency: The interval between optimization calls actually performing a
    synchronization step.
    update_weight: The update weight, 1.0 meaning a full assignment of the source
    variables values.


def tf_step(self, time, variables, source_variables, **kwargs):

assert all(util.shape(source) == util.shape(target) for source, target in zip(source_variables, variables))<EOL>last_sync = tf.get_variable(<EOL>name='<STR_LIT>',<EOL>shape=(),<EOL>dtype=tf.int64,<EOL>initializer=tf.constant_initializer(value=(-self.sync_frequency), dtype=tf.int64),<EOL>trainable=False<EOL>)<EOL>def sync():<EOL><INDENT>deltas = list()<EOL>for source_variable, target_variable in zip(source_variables, variables):<EOL><INDENT>delta = self.update_weight * (source_variable - target_variable)<EOL>deltas.append(delta)<EOL><DEDENT>applied = self.apply_step(variables=variables, deltas=deltas)<EOL>last_sync_updated = last_sync.assign(value=time)<EOL>with tf.control_dependencies(control_inputs=(applied, last_sync_updated)):<EOL><INDENT>return [delta + <NUM_LIT:0.0> for delta in deltas]<EOL><DEDENT><DEDENT>def no_sync():<EOL><INDENT>deltas = list()<EOL>for variable in variables:<EOL><INDENT>delta = tf.zeros(shape=util.shape(variable))<EOL>deltas.append(delta)<EOL><DEDENT>return deltas<EOL><DEDENT>do_sync = (time - last_sync >= self.sync_frequency)<EOL>return tf.cond(pred=do_sync, true_fn=sync, false_fn=no_sync)<EOL>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    source_variables: List of source variables to synchronize with.
    **kwargs: Additional arguments, not used.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def __init__(self, optimizer, clipping_value, scope='<STR_LIT>', summary_labels=()):

assert isinstance(clipping_value, float) and clipping_value > <NUM_LIT:0.0><EOL>self.clipping_value = clipping_value<EOL>super(ClippedStep, self).__init__(optimizer=optimizer, scope=scope, summary_labels=summary_labels)<EOL>

Creates a new multi-step meta optimizer instance.

Args:
    optimizer: The optimizer which is modified by this meta optimizer.
    clipping_value: Clip deltas at this value.


deltas = self.optimizer.step(time=time, variables=variables, **kwargs)<EOL>with tf.control_dependencies(control_inputs=deltas):<EOL><INDENT>clipped_deltas = list()<EOL>exceeding_deltas = list()<EOL>for delta in deltas:<EOL><INDENT>clipped_delta = tf.clip_by_value(<EOL>t=delta,<EOL>clip_value_min=-self.clipping_value,<EOL>clip_value_max=self.clipping_value<EOL>)<EOL>clipped_deltas.append(clipped_delta)<EOL>exceeding_deltas.append(clipped_delta - delta)<EOL><DEDENT><DEDENT>applied = self.apply_step(variables=variables, deltas=exceeding_deltas)<EOL>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>return [delta + <NUM_LIT:0.0> for delta in clipped_deltas]<EOL><DEDENT>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    **kwargs: Additional arguments passed on to the internal optimizer.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def __init__(self, optimizer, scope='<STR_LIT>', summary_labels=()):

super(GlobalOptimizer, self).__init__(optimizer=optimizer, scope=scope, summary_labels=summary_labels)<EOL>

Creates a new global optimizer instance.

Args:
    optimizer: The optimizer which is modified by this meta optimizer.


global_variables = kwargs["<STR_LIT>"]<EOL>assert all(<EOL>util.shape(global_variable) == util.shape(local_variable)<EOL>for global_variable, local_variable in zip(global_variables, variables)<EOL>)<EOL>local_deltas = self.optimizer.step(time=time, variables=variables, **kwargs)<EOL>with tf.control_dependencies(control_inputs=local_deltas):<EOL><INDENT>applied = self.optimizer.apply_step(variables=global_variables, deltas=local_deltas)<EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>update_deltas = list()<EOL>for global_variable, local_variable in zip(global_variables, variables):<EOL><INDENT>delta = global_variable - local_variable<EOL>update_deltas.append(delta)<EOL><DEDENT>applied = self.apply_step(variables=variables, deltas=update_deltas)<EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>return [local_delta + update_delta for local_delta, update_delta in zip(local_deltas, update_deltas)]<EOL><DEDENT>

Keyword Args:
    global_variables: List of global variables to apply the proposed optimization step to.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


self.solver = LineSearch(<EOL>max_iterations=ls_max_iterations,<EOL>accept_ratio=ls_accept_ratio,<EOL>mode=ls_mode,<EOL>parameter=ls_parameter,<EOL>unroll_loop=ls_unroll_loop<EOL>)<EOL>super(OptimizedStep, self).__init__(optimizer=optimizer, scope=scope, summary_labels=summary_labels)<EOL>

Creates a new optimized step meta optimizer instance.

Args:
    optimizer: The optimizer which is modified by this meta optimizer.
    ls_max_iterations: Maximum number of line search iterations.
    ls_accept_ratio: Line search acceptance ratio.
    ls_mode: Line search mode, see LineSearch solver.
    ls_parameter: Line search parameter, see LineSearch solver.
    ls_unroll_loop: Unroll line search loop if true.


<EOL>arguments['<STR_LIT>'] = fn_reference(**arguments)<EOL>loss_before = -fn_loss(**arguments)<EOL>with tf.control_dependencies(control_inputs=(loss_before,)):<EOL><INDENT>deltas = self.optimizer.step(<EOL>time=time,<EOL>variables=variables,<EOL>arguments=arguments,<EOL>fn_loss=fn_loss,<EOL>return_estimated_improvement=True,<EOL>**kwargs<EOL>)<EOL>if isinstance(deltas, tuple):<EOL><INDENT>if len(deltas) != <NUM_LIT:2>:<EOL><INDENT>raise TensorForceError("<STR_LIT>")<EOL><DEDENT>deltas, estimated_improvement = deltas<EOL>estimated_improvement = -estimated_improvement<EOL><DEDENT>else:<EOL><INDENT>estimated_improvement = None<EOL><DEDENT><DEDENT>with tf.control_dependencies(control_inputs=deltas):<EOL><INDENT>loss_step = -fn_loss(**arguments)<EOL><DEDENT>with tf.control_dependencies(control_inputs=(loss_step,)):<EOL><INDENT>def evaluate_step(deltas):<EOL><INDENT>with tf.control_dependencies(control_inputs=deltas):<EOL><INDENT>applied = self.apply_step(variables=variables, deltas=deltas)<EOL><DEDENT>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>return -fn_loss(**arguments)<EOL><DEDENT><DEDENT>return self.solver.solve(<EOL>fn_x=evaluate_step,<EOL>x_init=deltas,<EOL>base_value=loss_before,<EOL>target_value=loss_step,<EOL>estimated_improvement=estimated_improvement<EOL>)<EOL><DEDENT>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    arguments: Dict of arguments for callables, like fn_loss.
    fn_loss: A callable returning the loss of the current model.
    fn_reference: A callable returning the reference values, in case of a comparative loss.
    **kwargs: Additional arguments passed on to the internal optimizer.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


assert learning_rate > <NUM_LIT:0.0><EOL>self.learning_rate = learning_rate<EOL>self.solver = ConjugateGradient(<EOL>max_iterations=cg_max_iterations,<EOL>damping=cg_damping,<EOL>unroll_loop=cg_unroll_loop<EOL>)<EOL>super(NaturalGradient, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Creates a new natural gradient optimizer instance.

Args:
    learning_rate: Learning rate, i.e. KL-divergence of distributions between optimization steps.
    cg_max_iterations: Conjugate gradient solver max iterations.
    cg_damping: Conjugate gradient solver damping factor.
    cg_unroll_loop: Unroll conjugate gradient loop if true.


<EOL>kldiv = fn_kl_divergence(**arguments)<EOL>kldiv_gradients = tf.gradients(ys=kldiv, xs=variables)<EOL>def fisher_matrix_product(deltas):<EOL><INDENT>deltas = [tf.stop_gradient(input=delta) for delta in deltas]<EOL>delta_kldiv_gradients = tf.add_n(inputs=[<EOL>tf.reduce_sum(input_tensor=(delta * grad)) for delta, grad in zip(deltas, kldiv_gradients)<EOL>])<EOL>return tf.gradients(ys=delta_kldiv_gradients, xs=variables)<EOL><DEDENT>loss = fn_loss(**arguments)<EOL>loss_gradients = tf.gradients(ys=loss, xs=variables)<EOL>deltas = self.solver.solve(fn_x=fisher_matrix_product, x_init=None, b=[-grad for grad in loss_gradients])<EOL>delta_fisher_matrix_product = fisher_matrix_product(deltas=deltas)<EOL>constant = <NUM_LIT:0.5> * tf.add_n(inputs=[<EOL>tf.reduce_sum(input_tensor=(delta_F * delta))<EOL>for delta_F, delta in zip(delta_fisher_matrix_product, deltas)<EOL>])<EOL>def natural_gradient_step():<EOL><INDENT>lagrange_multiplier = tf.sqrt(x=(constant / self.learning_rate))<EOL>estimated_deltas = [delta / lagrange_multiplier for delta in deltas]<EOL>estimated_improvement = tf.add_n(inputs=[<EOL>tf.reduce_sum(input_tensor=(grad * delta))<EOL>for grad, delta in zip(loss_gradients, estimated_deltas)<EOL>])<EOL>applied = self.apply_step(variables=variables, deltas=estimated_deltas)<EOL>with tf.control_dependencies(control_inputs=(applied,)):<EOL><INDENT>if return_estimated_improvement:<EOL><INDENT>return [estimated_delta + <NUM_LIT:0.0> for estimated_delta in estimated_deltas], estimated_improvement<EOL><DEDENT>else:<EOL><INDENT>return [estimated_delta + <NUM_LIT:0.0> for estimated_delta in estimated_deltas]<EOL><DEDENT><DEDENT><DEDENT>def zero_step():<EOL><INDENT>if return_estimated_improvement:<EOL><INDENT>return [tf.zeros_like(tensor=delta) for delta in deltas], <NUM_LIT:0.0><EOL><DEDENT>else:<EOL><INDENT>return [tf.zeros_like(tensor=delta) for delta in deltas]<EOL><DEDENT><DEDENT>return tf.cond(pred=(constant > <NUM_LIT:0.0>), true_fn=natural_gradient_step, false_fn=zero_step)<EOL>

Creates the TensorFlow operations for performing an optimization step.

Args:
    time: Time tensor.
    variables: List of variables to optimize.
    arguments: Dict of arguments for callables, like fn_loss.
    fn_loss: A callable returning the loss of the current model.
    fn_kl_divergence: A callable returning the KL-divergence relative to the current model.
    return_estimated_improvement: Returns the estimated improvement resulting from the
        natural gradient calculation if true.
    **kwargs: Additional arguments, not used.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


self.max_grad_norm = max_grad_norm<EOL>self._lr = learning_rate<EOL>self._momentum = momentum<EOL>self._clip_kl = clip_kl<EOL>self._channel_fac = channel_fac<EOL>self._kfac_update = kfac_update<EOL>self._async = async_<EOL>self._async_stats = async_stats<EOL>self._epsilon = epsilon<EOL>self._stats_decay = stats_decay<EOL>self._blockdiag_bias = blockdiag_bias<EOL>self._approxT2 = approxT2<EOL>self._use_float64 = use_float64<EOL>self._factored_damping = factored_damping<EOL>self._cold_iter = cold_iter<EOL>if cold_lr == None:<EOL><INDENT>self._cold_lr = self._lr<EOL><DEDENT>else:<EOL><INDENT>self._cold_lr = cold_lr<EOL><DEDENT>self._stats_accum_iter = stats_accum_iter<EOL>self._weight_decay_dict = weight_decay_dict<EOL>self._diag_init_coeff = <NUM_LIT:0.><EOL>self._full_stats_init = full_stats_init<EOL>if not self._full_stats_init:<EOL><INDENT>self._stats_accum_iter = self._cold_iter<EOL><DEDENT>self.sgd_step = tf.Variable(<NUM_LIT:0>, name='<STR_LIT>', trainable=False)<EOL>self.global_step = tf.Variable(<EOL><NUM_LIT:0>, name='<STR_LIT>', trainable=False)<EOL>self.cold_step = tf.Variable(<NUM_LIT:0>, name='<STR_LIT>', trainable=False)<EOL>self.factor_step = tf.Variable(<EOL><NUM_LIT:0>, name='<STR_LIT>', trainable=False)<EOL>self.stats_step = tf.Variable(<EOL><NUM_LIT:0>, name='<STR_LIT>', trainable=False)<EOL>self.vFv = tf.Variable(<NUM_LIT:0.>, name='<STR_LIT>', trainable=False)<EOL>self.factors = {}<EOL>self.param_vars = []<EOL>self.stats = {}<EOL>self.stats_eigen = {}<EOL>super(KFAC, self).__init__(scope=scope, summary_labels=summary_labels)<EOL>

Initializes a KFAC optimizer.

For more information on arguments, see the Kfac Optimization paper https://arxiv.org/pdf/1503.05671.pdf


def updateAccumStats():<EOL><INDENT>if self._full_stats_init:<EOL><INDENT>return tf.cond(tf.greater(self.sgd_step, self._cold_iter), lambda: tf.group(*self._apply_stats(statsUpdates, accumulate=True, accumulateCoeff=<NUM_LIT:1.> / self._stats_accum_iter)), tf.no_op)<EOL><DEDENT>else:<EOL><INDENT>return tf.group(*self._apply_stats(statsUpdates, accumulate=True, accumulateCoeff=<NUM_LIT:1.> / self._stats_accum_iter))<EOL><DEDENT><DEDENT>def updateRunningAvgStats(statsUpdates, fac_iter=<NUM_LIT:1>):<EOL><INDENT>return tf.group(*self._apply_stats(statsUpdates))<EOL><DEDENT>if self._async_stats:<EOL><INDENT>update_stats = self._apply_stats(statsUpdates)<EOL>queue = tf.FIFOQueue(<NUM_LIT:1>, [item.dtype for item in update_stats], shapes=[<EOL>item.get_shape() for item in update_stats])<EOL>enqueue_op = queue.enqueue(update_stats)<EOL>def dequeue_stats_op():<EOL><INDENT>return queue.dequeue()<EOL><DEDENT>self.qr_stats = tf.train.QueueRunner(queue, [enqueue_op])<EOL>update_stats_op = tf.cond(tf.equal(queue.size(), tf.convert_to_tensor(<EOL><NUM_LIT:0>)), tf.no_op, lambda: tf.group(*[dequeue_stats_op(), ]))<EOL><DEDENT>else:<EOL><INDENT>update_stats_op = tf.cond(tf.greater_equal(<EOL>self.stats_step, self._stats_accum_iter), lambda: updateRunningAvgStats(statsUpdates), updateAccumStats)<EOL><DEDENT>self._update_stats_op = update_stats_op<EOL>return update_stats_op<EOL>

compute stats and update/apply the new stats to the running average


<EOL>with tf.device('<STR_LIT>'):<EOL><INDENT>def removeNone(tensor_list):<EOL><INDENT>local_list = []<EOL>for item in tensor_list:<EOL><INDENT>if item is not None:<EOL><INDENT>local_list.append(item)<EOL><DEDENT><DEDENT>return local_list<EOL><DEDENT>def copyStats(var_list):<EOL><INDENT>print("<STR_LIT>")<EOL>redundant_stats = {}<EOL>copied_list = []<EOL>for item in var_list:<EOL><INDENT>if item is not None:<EOL><INDENT>if item not in redundant_stats:<EOL><INDENT>if self._use_float64:<EOL><INDENT>redundant_stats[item] = tf.cast(<EOL>tf.identity(item), tf.float64)<EOL><DEDENT>else:<EOL><INDENT>redundant_stats[item] = tf.identity(item)<EOL><DEDENT><DEDENT>copied_list.append(redundant_stats[item])<EOL><DEDENT>else:<EOL><INDENT>copied_list.append(None)<EOL><DEDENT><DEDENT>return copied_list<EOL><DEDENT>stats_eigen = self.stats_eigen<EOL>computedEigen = {}<EOL>eigen_reverse_lookup = {}<EOL>updateOps = []<EOL>with tf.control_dependencies([]):<EOL><INDENT>for stats_var in stats_eigen:<EOL><INDENT>if stats_var not in computedEigen:<EOL><INDENT>eigens = tf.self_adjoint_eig(stats_var)<EOL>e = eigens[<NUM_LIT:0>]<EOL>Q = eigens[<NUM_LIT:1>]<EOL>if self._use_float64:<EOL><INDENT>e = tf.cast(e, tf.float32)<EOL>Q = tf.cast(Q, tf.float32)<EOL><DEDENT>updateOps.append(e)<EOL>updateOps.append(Q)<EOL>computedEigen[stats_var] = {'<STR_LIT:e>': e, '<STR_LIT>': Q}<EOL>eigen_reverse_lookup[e] = stats_eigen[stats_var]['<STR_LIT:e>']<EOL>eigen_reverse_lookup[Q] = stats_eigen[stats_var]['<STR_LIT>']<EOL><DEDENT><DEDENT><DEDENT>self.eigen_reverse_lookup = eigen_reverse_lookup<EOL>self.eigen_update_list = updateOps<EOL><DEDENT>return updateOps<EOL>

compute the eigen decomp using copied var stats to avoid concurrent read/write from other queue


fn_loss = kwargs["<STR_LIT>"]<EOL>if variables is None:<EOL><INDENT>variables = tf.trainable_variables<EOL><DEDENT>return tf.gradients(fn_loss, variables)<EOL>

Creates the TensorFlow operations for performing an optimization step on the given variables, including
actually changing the values of the variables.

Args:
    time: Time tensor. Not used for this optimizer.
    variables: List of variables to optimize.
    **kwargs:
        fn_loss : loss function tensor to differentiate.

Returns:
    List of delta tensors corresponding to the updates for each optimized variable.


def apply_step(self, variables, deltas, loss_sampled):

update_stats_op = self.compute_and_apply_stats(<EOL>loss_sampled, var_list=var_list)<EOL>grads = [(a, b) for a, b in zip(deltas, varlist)]<EOL>kfacOptim, _ = self.apply_gradients_kfac(grads)<EOL>return kfacOptim<EOL>

Applies the given (and already calculated) step deltas to the variable values.

Args:
    variables: List of variables.
    deltas: List of deltas of same length.
    loss_sampled : the sampled loss

Returns:
    The step-applied operation. A tf.group of tf.assign_add ops.


loss = kwargs["<STR_LIT>"]<EOL>sampled_loss = kwargs["<STR_LIT>"]<EOL>min_op, _ = self.minimize_(loss, sampled_loss, var_list=variables)<EOL>return min_op<EOL>

Performs an optimization step.

Args:
    time: Time tensor. Not used for this
    variables: List of variables to optimize.
    **kwargs:
        fn_loss : loss function tensor that is differentiated
        sampled_loss : the sampled loss from running the model.

Returns:
    The optimization operation.


def __init__(self, max_iterations, accept_ratio, mode, parameter, unroll_loop=False):

assert accept_ratio >= <NUM_LIT:0.0><EOL>self.accept_ratio = accept_ratio<EOL>if mode not in ('<STR_LIT>', '<STR_LIT>'):<EOL><INDENT>raise TensorForceError("<STR_LIT>".format(mode))<EOL><DEDENT>self.mode = mode<EOL>self.parameter = parameter<EOL>super(LineSearch, self).__init__(max_iterations=max_iterations, unroll_loop=unroll_loop)<EOL>

Creates a new line search solver instance.

Args:
    max_iterations: Maximum number of iterations before termination.
    accept_ratio: Lower limit of what improvement ratio over $x = x'$ is acceptable
        (based either on a given estimated improvement or with respect to the value at
        $x = x'$).
    mode: Mode of movement between $x_0$ and $x'$, either 'linear' or 'exponential'.
    parameter: Movement mode parameter, additive or multiplicative, respectively.
    unroll_loop: Unrolls the TensorFlow while loop if true.