text: ******************************
  Containers: Layers & Operators
  ******************************
  
  Containers are the building blocks of Kur models. Each entry in a Kur model is
  a container, and containers may accept other containers as input. There are two
  types of containers: *layers* and *operators*. "Layers" are Kur's
  representation of deep learning primitives (like convolutions). "Operators" are
  like "meta-layers": they are used to modify how other layers are interpretted
  (like "for" loops).
  
  Layers
  ======
  
  Layers are your fundamental building blocks for complex deep learning models.
  
  Dense
  -----
  
  **Description**. A fully-connected layer of nodes (affine transformation), in
  which each node is connected to each node from the previous layer.
  
  **Purpose**. Dense layers are very common for "mixing" information across the
  entire tensor. So in image classification, you usually want one or more dense
  layers following your convolutions. They are also the building blocks of
  multi-layer perceptrons.
  
  **Note**. Dense layers do not include activations. If you want a non-linearity
  between successive dense layers, you must explicitly insert them.
  
  **Usage**::
  
  	dense: SIZE
  
  or::
  
  	dense:
  	  size: SIZE
  
  - ``SIZE``: an integer or list of integers. If a single integer, it indicates
    the number of nodes in this dense layer. If a list, it is treated as a series
    of dense layers, one for each entry in ``SIZE``.
  
  Convolution (CNN)
  -----------------
  
  **Description**. A convolution is a locally-connected layer (as opposed to a
  dense / fully-connected layer).
  
  **Purpose**. Convolutions are useful for "smoothing" out information, and is
  used to abstract lower-level features into a higher-level representation. These
  are very common in image recognition, where groups of pixels are combined to
  produce some "high-level" meaning, which in turn is fed into deeper layers.
  
  **Usage**::
  
  	convolution:
  	  kernels: KERNELS
  	  size: SIZE
  	  strides: STRIDES
  	  border: BORDER
  
  - ``KERNELS``: the number of convolutional filters to apply.
  - ``SIZE``: an integer (for 1-D convolutions) or a list of integers which
    specify the size of the receptive field of the convolution. For image
    classification, for example, this would be ``[width, height]``, in pixels,
    of the convolution.
  - ``STRIDES``: an integer or list of integers which indicate the stride (step
    size) of the convolution. If this is an integer, the stride is applied in
    each dimension; if it is a list, it indicates the stride in each dimension.
    This is optional; its default value is `1`.
  - ``BORDER``: one of "valid" or "same", which tells Kur how to treat the edges
    of the input. Its default value is "same".
  
  Recurrent (RNN)
  ---------------
  
  **Description**. A recurrent layer for learning sequences of data.
  
  **Purpose**. Recurrent layers are used to learn sequences.
  
  **Usage**::
  
  	recurrent:
  	  size: SIZE
  	  type: {lstm | gru}
  	  sequence: SEQUENCE
  	  bidirectional: BIDIRECTIONAL
  	  merge: {multiply | add | average | concat}
  	  outer_activation: ACTIVATION
  
  - ``SIZE``: the number of recurrent nodes in the layer. This is the number of
    features that are kept *per timestep*. If ``SEQUENCE`` is true, then there
    are as many outputs as inputs (because the number of timesteps doesn't
    change), and at each timestep, the feature vector has length ``SIZE``. If
    ``SEQUENCE`` is false, then the output of the RNN is length ``SIZE`` (because
    only the last timestep is kept).
  - ``TYPE``: the type of the recurrent cells (defaults to ``gru``).
  - ``SEQUENCE``: boolean. If true, returns the entire sequences of RNN outputs.
    If false, only the last element of the sequence is returned. Defaults to
    True.
  - ``BIDIRECTIONAL``: boolean. If true, a bidirectional RNN is constructed (one
    which learns both the forward and backward sequences of data).
  - ``MERGE``: if ``BIDIRECTIONAL`` is true, then this determines how the outputs
    of the forward and backward RNNs is merged. If bidirectional is not set, then
    "merge" cannot be used. The default value is ``concat``.
  - ``ACTIVATION``: an activation to apply within the RNN (it is not equivalent
    to simply following the RNN layer with an Activation layer). Defaults to
    "relu".
  
  Activation
  ----------
  
  **Description**. An activation layers applies a simple non-linearity to each
  element of the tensor.
  
  **Purpose**. Activations are non-linear, so they are an important part of any
  deep learning model; without them, complex operations could be reduced to
  simple multiplications. You usually want them following your convolutions, and
  you typically want them as a final layer in your model.
  
  **Usage**::
  
  	activation: {softmax | relu | tanh | sigmoid | none}
  
  Output
  ------
  
  **Description**. A convenience layer for explicitly marking a point in the model
  as an output.
  
  **Purpose**. Any layer can be marked as an output using the ``sink: True``
  value. But this is often awkward, especially if preceding layers are nested
  deeply within a ``for`` or ``parallel`` block. This can be used to resolve
  those ambiguities, as well as make your model easier to read.
  
  **Usage**::
  
  	output: NAME
  
  or::
  
  	output:
  	  name: NAME
  
  - ``NAME``: the name of the output layer. It will be associated with the
    tensors produced by the most recent layers.
  
  Pooling
  -------
  
  **Description**. A pooling layer is used to reduce the size of the model's
  representation in a local manner.
  
  **Purpose**. Pooling layers are useful for "shrinking" the model by only
  keeping the "most important" local information in the tensor. You often use
  them in a convolutional stack: convolutions can increase the size of the
  representation if its ``kernels`` value is higher than the previous layers, and
  so pooling layers can reduce the size again, keeping only the most important
  local information. This reduces training time and overfitting.
  
  **Usage**::
  
  	pool:
  	  size: SIZE
  	  strides: STRIDES
  	  type: {max | average}
  
  - ``SIZE``: an integer (for 1-D pools) or a list of integers which specify the
    size of the pooling layer's receptive field. For image classification, for
    example, this is the ``[width, height]``, in pixels, of the pool.
  - ``STRIDES``: an integer or list of integers which indicate the stride (step
    size) of the pool. If this is an integer, the stride is applied in each
    dimension; if it is a list, it indicates the stride in each dimension.  This
    is optional; its default value is `1`.
  
  Batch Normalization
  -------------------
  
  **Description**. A layer which directs Kur to normalize activations across the
  previous layer.
  
  **Purpose**. Batch normalization is useful to help prevent bad initializations
  and improve convergence.
  
  **Usage**::
  
  	batch_normalization
  
  or::
  
  	batch_normalization:
  	  axis: AXIS
  
  - ``AXIS``: an integer indicating which axis to apply the normalization to.
    Defaults to -1 (the last axis).
  
  Expand
  ------
  
  **Description**. An "expand" layer inserts a new axis into the tensor.
  
  **Purpose**. An expand layer is useful for manipulating the shape of your data
  or tensors to fit them into the rest of your model. For example, if you have
  a bunch of ``[width, height]`` single-channel (monochrome) images, you can't
  simply pass them into a two-dimensional convolution, since N-dimensional
  convolutions expect (N+1)-dimensional data. So you can insert an expand layer
  to make your image data have shape ``[width, height, 1]``. Three-channel (RGB)
  color images are already this shape ``[width, height, 3]``, so they don't need
  an expand layer.
  
  **Usage**::
  
  	expand: DIMENSION
  
  or::
  
  	expand:
  	  dimension: DIMENSION
  
  - ``DIMENSION``: the index to insert the new length-1 dimension at (zero-based
    index). So if ``DIMENSION`` is zero, it will reshape your data to be ``[1,
    ...]``. Negative numbers count from the back, so if ``DIMENSION`` is -1, your
    data will be shaped to be ``[..., 1]``.
  
  Squeeze
  -------
  
  **Description**. A "squeeze" layer removes a length 1 dimension from the data
  tensor.
  
  **Purpose**. Like the expand layer, a squeeze layer is also useful for
  manipulating shapes in your model. In fact, it is operational the inverse of
  the expand layer. If you have data tensor of shape ``[width, height, 1]``, as
  might come out of a 2D convolution with 1 kernel, you can cast it to a tensor
  of shape ``[width, height]`` with a ``squeeze: -1`` (or ``squeeze: 2``) layer.
  
  **Usage**::
  
  	squeeze: DIMENSION
  
  or::
  
  	squeeze:
  	  dimension: DIMENSION
  
  - ``DIMENSION``: the index to remove a length-1 dimension at (zero-based
    index). Negative numbers count from the back, so if ``DIMENSION`` is -1, the
    last dimension of the data will be removed if it is length 1.
  
  Flatten
  -------
  
  **Description**. A flatten layer reduces the dimension of a tensor to 1-D.
  
  **Purpose**. Flatten layers are used when you are no longer interested in the
  dimensionality of your data, and are ready to let the data "mix." You typically
  put them immediately before the first dense layer of your model, since dense
  layers require 1-D data.
  
  **Usage**::
  
  	flatten
  
  or::
  
  	flatten:
  
  Parallel
  --------
  
  **Description**. A parallel layer applies a series of tensor operations to each
  top-level element of your data. This is like a `map function
  <https://en.wikipedia.org/wiki/Map_(higher-order_function)>`_.
  
  **Purpose**. Here are a few (equivalent) ways to think about parallel layers:
  
  	- Map function. Parallel layers apply a function to each (top-level)
  	  element of your data, converting *[x0, x1, x2, ...]* to *[f(x0), f(x1),
  	  f(x2), ...]*. An ``activation`` layer applies a function to each
  	  *bottom-level* element of your data (you might say that an ``activation``
  	  layer is applied element-wise, in some sense) and only applies primitive
  	  operations. A parallel layer can apply very sophisticated deep learning
  	  operations to each element of your data.
  	- Sub-model. You can think of parallel layers as sub-models or nested
  	  models.  They are models inside the deep learning model itself. You get
  	  to define your sub-model within the ``parallel`` layer, and then it gets
  	  applied to each element of your data.
  	- Distributed operations. If you think of your data as a time series, where
  	  each element represents another time step, then a ``parallel`` layer
  	  applies the *same* operation (with the *same* weights) to each time step.
  
  **Usage**::
  
  	parallel:
  	  apply:
  	    - CONTAINER_1
  	    - CONTAINER_2
  	    ...
  
  The ``apply`` key is a list of layers (or operations) that define the
  "parallel" / "sub-model" / "time-distributed" operation. Each container in the
  ``apply`` list is applied, in turn, to each element of the input data.
  
  Reuse
  -----
  
  **Description**. A reuse layer is a weight-sharing layer; it simply re-applies
  another layer in the model without declaring new weights.
  
  **Purpose**. A reuse layer is useful when you want to have a single tensor
  operation (with learnable weights) that you apply in multiple places in your
  model (as opposed to multiple tensor operations in multiple places).
  
  **Usage**::
  
  	reuse:
  	  target: TARGET
  
  or::
  
  	reuse: TARGET
  
  - ``TARGET``: the name of the layer to re-apply.
  
  Tranpose
  --------
  
  **Description**. A transpose layer shuffles the dimensions of the data tensor.
  
  **Purpose**. Sometimes you need to reorder your dimensions so that downstream
  layers can act on your data in different ways. The Transpose layer helps you do
  this.
  
  **Usage**::
  
  	transpose: PERMUTATION
  
  or::
  
  	transpose:
  	  axes: PERMUTATION
  	  include_batch: INCLUDE_BATCH
  
  - ``PERMUTATION``: a list of axes which specifies the permutation to apply to
    the data tensor. You must specify the complete permutation (one entry for
    each dimension in the data tensor). If ``INCLUDE_BATCH`` is False (the
    default), then the axes are zero-based, starting with the first dimension
    after the batch. If ``INCLUDE_BATCH`` is True, then the axes are zero-based,
    but start with the batch dimension itself.
  - ``INCLUDE_BATCH``: a boolean which indicates whether or not to include the
    batch dimension in the permutation. By default, it is False. This should
    almost certainly remain False for most applications, since you probably don't
    want your layers to act differently depending on batch size!
  
  Merge
  -----
  
  **Description**. A merge layer combines multiple inputs into a single tensor.
  
  **Purpose**. If your model has branches (a single layer is used as input by
  multiple layers) or multiple inputs, then at some point you may want to combine
  information from different layers. The Merge layer takes multiple inputs and
  produces a single output.
  
  **Usage**::
  
  	merge: MODE
  
  or::
  
  	merge:
  	  mode: MODE
  	  axis: AXIS
  
  - ``MODE``: the function to use to combine input data. It can be one of these:
  
  	- ``average``: Computes the mean of the input tensors.
  	- ``multiply``: Computes the product of the input tensors.
  	- ``add``: Computes the sum of the input tensors.
  	- ``concat``: Concatenates the data tensors along ``AXIS`` axis. This is
  	  the default.
  
  - ``AXIS``: the axis to concatenate input tensors along. Only used if ``MODE``
    is ``concat``. Defaults to -1 (the last axis).
  
  .. _ref_placeholder:
  
  Placeholder
  -----------
  
  **Description**. A placeholder layer just declares an input to the model. They
  do not do anything other than declare where the model "starts."
  
  **Purpose**. There are two purposes for input layers. The first is to help with
  authoring, so you can keep track of your model's inputs. The second is to help
  with debugging: input layers can have explicit sizes associated with them, and
  if you try to apply your model to data of a different size, you'll get an
  error.
  
  **Usage**::
  
  	input: NAME
  
  or::
  
  	input:
  	  shape: SHAPE
  
  - ``NAME``. The name of the model input. It must match the name of one of the
    data sources in the model specification. The first form is just a convenience
    form; names can be specified in the second form using the standard
    specification, like this::
  
  	input:
  	  shape: SHAPE
  	name: NAME
  
  - ``SHAPE``: A list of dimensions describing the expected shape of the data.
    This is useful for catching data problems early (the model will refuse to
    train/evaluate if it is given data of the wrong shape), and you can use the
    ``kur build`` command to test that your model "fits together" if the shapes
    of all inputs are specified. However, the shape is optional; if omitted, it
    will be inferred from the data source.
  
  Repeat
  ------
  
  **Description**. Repeats an input vector a given number of times.
  
  **Purpose**. Repeat layers are great for taken 1D input (like the output of a
  ``dense`` or ``flatten`` layer) and producing 2D output. It does this by simply
  duplicating the input a fixed number of times, taking input ``X`` and producing
  ``[X, X, X, ..., X]``.
  
  **Usage**::
  
  	repeat: COUNT
  
  or::
  
  	repeat:
  	  count: COUNT
  
  - ``COUNT``: An integer. The number of times to repeat in input vector.
  
  Embedding
  ---------
  
  **Description**. A learnable embedding layer.
  
  **Purpose**. Embedding layers learn to map integer labels onto a
  higher-dimensional vocabulary.
  
  **Usage**::
  
  	embedding:
  	  vocab_size: VOCAB_SIZE
  	  size: SIZE
  
  - ``VOCAB_SIZE``: the size of the input vocabulary. An embedding layer takes
    integer labels as inputs, corresponding to vocabulary words. Thus, the inputs
    should be zero-based indices which are < ``VOCAB_SIZE``.
  - ``SIZE``: The dimensionality of the embedding layer to learn. Each input
    label is mapped to a ``SIZE``-dimensional vector.
  
  Dropout
  -------
  
  **Description**. A dropout layer which drops a random set of activations on the
  previous layer during every forward/backward pass.
  
  **Purpose**. Dropout layers help to control overtraining by forcing the network
  to use different "neural pathways" to solve the problem. It is similar to
  generating a large number of superimposed models.
  
  **Usage**::
  
  	dropout: FRACTION
  
  - ``FRACTION``: float between 0 and 1. The fraction of activations to drop at
    each pass.
  
  Operators
  =========
  
  Operators are used to manipulate deep learning operations at a higher level.
  They are used to build more general, parameterized models.
  
  For Loop
  --------
  
  **Description**. A traditional ``for`` loop, which can create an arbitrary
  number of layers.
  
  **Purpose**. The ``for`` loop is perfect when you want some part of your model
  to be repeated a fixed number of times, but you want to parameterize that
  number (e.g., turn it into a variable instead of simply copy/pasting the layers
  over and over again).
  
  **Usage**::
  
  	for:
  	  range: RANGE
  	  with_index: INDEX
  	  iterate:
  	    - CONTAINER_1
  	    - CONTAINER_2
  	    ...
  
  - ``RANGE``: the number of times to iterate.
  - ``INDEX``: the name of the local variable to create. This is optional and
    defaults to ``index``.
  
  The ``for`` loop adds each container under ``iterate`` during each iteration
  (there are ``RANGE`` iterations total). Unless the containers are ``reuse``
  containers, each resulting container is independent and has its own weights.
  
  .. _meta_containers:
  
  Meta-Containers
  ---------------
  
  **Description**. A user-defined template.
  
  **Purpose**. Meta-containers are instantiations of :ref:`user-defined templates
  <template_spec>`. They do not have a well-defined name, unlike other
  containers.  Instead, they are created by simply referencing the template
  definition, and then defining any variables that need to be passed to the
  template.
  
  **Usage**:
  
  Given a previous template definition::
  
  	templates:
  	  cnn_stack:
  	    - for:
  	        range: "{{ depth }}"
  	        iterate:
  	          - convolution: "{{ cnn }}"
  
  one could use the template like this::
  
  	cnn_stack:
  	  depth: 3
  	  cnn:
  	    size: [2, 2]
  	    kernels: 64
  
  As you can see, the template parameters ``depth`` and ``cnn`` where defined and
  passed into the container.
  
  **Note**: Some template parameters may be defined outside the meta-container
  layer (as with indices inside a loop); therefore, not all template parameters
  need to be defined as arguments to the meta-container.
  
  Debug
  -----
  
  **Description**. A debug message printer.
  
  **Purpose**. The debug layer does not affect the model in any way. It is used
  solely for outputting information to the console to aid in debugging your
  models. You can print out variable values, for example.
  
  **Usage**::
  
  	debug: MESSAGE
  
  **Note**: you must have debug-level (``kur -vv ...``) log messages enabled to
  see this.
  
  Assert
  ------
  
  *Description*. A debug-style assertion checker.
  
  **Purpose**. The assert layer does not affect the model in any way, but simply
  provides a debug-style "assert" condition that will immediately raise an
  AssertionError if the condition evaluates to False. This can be useful for
  checking that your model is not being constructed incorrectly and that it's shape
  is correct.
  
  **Usage**::
  
  	assert: CONDITION
  
  - ``CONDITION``: the condition to assert must be True. It will be evaluated by
    an Engine before being checked for true/false.
  