Package: nuage-metroae-config
Nuage MetroAE ConfigurationNuage MetroAE configuration template engine.Version 1.1.0This tool reads JSON or Yaml files of templates and user-data to
write a configuration to a VSD or to revert (remove) said configuration.OverviewMetroAE config is a Python-based engine which can apply configuration to VSDs
via templates.  The templates provide an abstraction around the VSD
configuration model simplifying and validating the required data.  Data is
provided by the user in the form of Yaml or JSON files and is transformed and
applied to the VSD through the tool's templates.  The required data and proper
format for each template is defined by standardized JSON schema specifications.InstallationVia Docker ContainerThere are several modes of operation for MetroAE config.  For most users, the
tool should be consumed via the MetroAE Docker container.  Using this method
ensures that all of the libraries are properly installed and themetroaecommand-line tool can be utilized for config (as well as deploy).  Instructions
for the Docker container installation are available at:https://github.com/nuagenetworks/nuage-metroae/blob/master/Documentation/DOCKER.mdIf using the Docker container, all remaining installation instructions can be
skipped.As Python LibraryThe MetroAE config tool is completely implemented as a Python library.  As such,
the functionality of the tool can be integrated into other Python tools or
libraries through the use of the APIs.  To do this, the library can be installed
via Python pip package from pypi.org:pip install nuage-metroae-configThe pip installation of the package will pull in all dependent libraries
automatically.  The following section of installing from Github can be skipped.
The use of the Library APIs is documented in thePython Library APIsection.From Github Python SourceUsing the MetroAE config tool without the Docker container can be accomplished
by pulling the source from Github.  You may want to use this method if you are
comfortable working with Python environments and don't want the overhead of a
container.   The instructions for this case is described in the next section.
If using this method, you would be responsible for installing all of the
necessary prerequiste libraries and to obtain the VSD API specs and templates.Installation of MetroAE Config from GithubThis section describes how to install the MetroAE config tool from the Github
repo.  These steps will be necessary to use the metroae_config command-line
tool directly.  If using the MetroAE Docker container, this section should be
skipped.The primary requirements for running MetroAE config are:Installation of Python and the necessary packages.Installation of the config engine python librariesAccess to the Configuration templatesAccess to the VSD API SpecificationInstallation of Python and required packages.Most linux distributions come with python bundled into the operating system.
Python 2.7 is required. Currently Python 3.0 is not supported.To verify that python is available we can do the following:Ubunturoot@ubuntu:~# python --version
Python 2.7.5RHEL/Centos[root@rhel]# python --version
Python 2.7.5To manage the python package installation we will use PIP. Install PIP via
apt-get or yum:Ubunturoot@ubuntu:~# apt-get install python-pip
...RHEL/Centos[root@rhel]# yum install python-pip
...Most packages required by MetroAE config are available as part of the base
python functionality. However we will need to install the following:BambouJinja2 (min version 2.10)PyYAMLlark-parserAdditional packages for unit-testmockpytestrequestsrequests-mocklark_parserThe following packages are installed via the same method on both Ubuntu and RHEL/Centos.Ubuntu and RHEL/Centos[root@rhel]# pip install -r requirements.txt
...Installation of configuration engineCurrently MetroAE config is available on github and will be installed via git
clone. Check the MetroAE home page for latest master branch location.First we need to install git.Ubunturoot@ubuntu:~# apt-get install gitRHEL/Centos[root@rhel]# yum install gitThe repository is available via Github:https://github.com/nuagenetworks/nuage-metroae-configDepending on the authentication method with github, ie. SSH Key, or
Username/Password the git clone command line may change.For username/password authenticationUbuntu and RHEL/Centos[root@rhel]# export GIT_SSL_NO_VERIFY=false
[root@rhel]# git clone https://github.com/nuagenetworks/nuage-metroae-config.git
Cloning into 'levistate'...
Username for 'https://github.com/nuagenetworks/nuage-metroae-config.git': sfiddian
Password for 'https://sfiddian@github.mv.usa.alcatel.com':
remote: Counting objects: 619, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 619 (delta 1), reused 0 (delta 0), pack-reused 610
Receiving objects: 100% (619/619), 178.01 KiB | 0 bytes/s, done.
Resolving deltas: 100% (398/398), done.Installation of the VSD API SpecificationsMetroAE Config requires a description of the VSD API to create, read, update and
delete the template contents into the VSD. We do this via reading the published
VSD API Specification. This specificaiton is opensourced and is available on
the public Nuage Networks github repository.Ubuntu and RHEL/Centos[root@rhel]# git clone https://github.com/nuagenetworks/vsd-api-specifications.git
Cloning into 'vsd-api-specifications'...
remote: Counting objects: 23428, done.
remote: Total 23428 (delta 0), reused 0 (delta 0), pack-reused 23428
Receiving objects: 100% (23428/23428), 4.98 MiB | 1013.00 KiB/s, done.
Resolving deltas: 100% (18163/18163), done.Command-line ToolWhen using the MetroAE Docker container, all commands are accessed through themetroaecommand-line tool as:metroae config ...If you are using the Python source obtained through Github, then any reference
tometroae configshould substitute the equivalent Python source tool:python metroae_config.py ...ParametersMetroAE Config command-line tool usage:usage: metroae config [-h]
                      {create,revert,validate,list,schema,example,upgrade-templates,version,help}
                      ...

Version 1.0 - This tool reads JSON or Yaml files of templates and user-data to
write a configuration to a VSD or to revert (remove) said configuration. See
README.md for more.

positional arguments:
  {create,revert,validate,list,schema,example,upgrade-templates,version,help}

optional arguments:
  -h, --help            show this help message and exit


usage: metroae config create [-h] [-tp TEMPLATE_PATH] [--version]
                             [-sp SPEC_PATH] [-dp DATA_PATH] [-d DATA]
                             [-v VSD_URL] [-u USERNAME] [-p PASSWORD]
                             [-e ENTERPRISE] [-lg]
                             [datafiles [datafiles ...]]

positional arguments:
  datafiles             Optional datafile

optional arguments:
  -h, --help            show this help message and exit
  -tp TEMPLATE_PATH, --template_path TEMPLATE_PATH
                        Path containing template files. Can also set using
                        environment variable TEMPLATE_PATH
  --version             Displays version information
  -sp SPEC_PATH, --spec_path SPEC_PATH
                        Path containing object specifications. Can also set
                        using environment variable VSD_SPECIFICATIONS_PATH
  -dp DATA_PATH, --data_path DATA_PATH
                        Path containing user data. Can also set using
                        environment variable USER_DATA_PATH
  -d DATA, --data DATA  Specify user data as key=value
  -v VSD_URL, --vsd_url VSD_URL
                        URL to VSD REST API. Can also set using environment
                        variable VSD_URL
  -u USERNAME, --username USERNAME
                        Username for VSD. Can also set using environment
                        variable VSD_USERNAME
  -p PASSWORD, --password PASSWORD
                        Password for VSD. Can also set using environment
                        variable VSD_PASSWORD
  -e ENTERPRISE, --enterprise ENTERPRISE
                        Enterprise for VSD. Can also set using environment
                        variable VSD_ENTERPRISE
  -lg, --logs           Show logs after runExample UsageApply enterprise, domain and ACLs to VSD.$ metroae config create -tp sample/templates -sp ~/vsd-api-specifications -v https://localhost:8443 sample/user_data/acls.yaml

Configuration
    Enterprise
        name = 'test_enterprise'
        DomainTemplate
            name = 'template_public'
            [store id to name domain_template_id]
        Domain
            name = 'public'
            templateID = [retrieve domain_template_id (DomainTemplate:id)]
            IngressACLTemplate
                priority = 100
                defaultAllowNonIP = False
                allowAddressSpoof = False
                name = 'test_acl'
                defaultAllowIP = True
                IngressACLEntryTemplate
                    networkID = ''
                    stateful = True
                    protocol = 6
                    description = 'Test ACL'
                    etherType = '0x0800'
                    statsLoggingEnabled = True
                    DSCP = '*'
                    priority = 200
                    action = 'FORWARD'
                    locationID = ''
                    destinationPort = '*'
                    locationType = 'ANY'
                    sourcePort = 80
                    networkType = 'ANY'
                    flowLoggingEnabled = True
            EgressACLTemplate
                priority = 100
                defaultInstallACLImplicitRules = True
                defaultAllowNonIP = False
                name = 'test_acl'
                defaultAllowIP = True
                EgressACLEntryTemplate
                    networkID = ''
                    stateful = True
                    protocol = 6
                    description = 'Test ACL'
                    etherType = '0x0800'
                    statsLoggingEnabled = True
                    DSCP = '*'
                    priority = 200
                    action = 'FORWARD'
                    locationID = ''
                    destinationPort = '*'
                    locationType = 'ANY'
                    sourcePort = 80
                    networkType = 'ANY'
                    flowLoggingEnabled = TrueRevert (remove) objects configured during application, use -r option$ metroae config revert -tp sample/templates -sp ~/vsd-api-specifications sample/user_data/acls.yamlUser DataThe templates within the MetroAE config tool are applied using data provided by the
user.  Each set of data values provided allows the creation of an instance of
configuration on the VSD.  The required format for the data of each template is
defined by a JSON schema describing the required fields, types and other
constraints on the data.File FormatThe user data for MetroAE config can be in either Yaml or JSON format.  Each entry
in the file defines a template and the value sets to use against that template
to instantiate configuration.- template: Enterprise
  values:
    - enterprise_name: my_first_enterprise
      description: The first

    - enterprise_name: my_second_enterprise
      description: The secondIn this example, two data sets are provided for the Enterprise template and the
result will be that the two enterprises will be created on the VSD.  Only the
enterprise_name is required, but description can be provided optionally.  The
JSON schema for the Enterprise template defines the data format.ChildrenLet's now add domains to the enterprise objects.  In the Domain template, note
that in addition to a domain_name, it also needs the enterprise_name to be
specified so that the domain can be associated with the correct enterprise.  We
could define our user data as follows:- template: Enterprise
  values:
    - enterprise_name: my_first_enterprise
      description: The first

    - enterprise_name: my_second_enterprise
      description: The second

- template: Domain
  values:
    - enterprise_name: my_first_enterprise
      description: The first
      domain_name: my_first_domain

    - enterprise_name: my_second_enterprise
      description: The second
      domain_name: my_second_domainThis user data is fully valid and will create two enterprises each with one
domain.  However, we are duplicating the enterprise names and it would be
tedious and error prone if the names needed to change.  In order to simplify
nested data, children can be utilized as follows:- template: Enterprise
  values:
    - enterprise_name: my_first_enterprise
      description: The first
  children:
    - template: Domain
      values:
        - domain_name: my_first_domain

- template: Enterprise
  values:
    - enterprise_name: my_second_enterprise
      description: The second
  children:
    - template: Domain
      values:
        - domain_name: my_second_domain
          description: Overriden descriptionIn this case, each Domain is a child of a parent Enterprise.  As such, all of
the values from the parent are inherited in the children.  The children
domains automatically inherit the enterprise_name of the parent.  This also
applies for the description.Note that values can be overriden if necessary as with the description of the second
domain.  The most specific value takes precendence.  To prevent ambiguity, any
template with child templates can only define one set of values.GroupsAnother mechanism for reducing duplication in data is by using groups.  A group
entry is very similar to a template, except that no actual template is
instantiated.  Thus, the group defines a stand-alone data set.  Groups are
useful when common data sets need to be defined and referenced elsewhere.- group: first
  values:
    - enterprise_name: my_first_enterprise
      domain_name: my_first_domain

- group: second
  values:
    - enterprise_name: my_second_enterprise
      domain_name: my_second_domain

- group: ssh
  values:
    - protocol: tcp
      port: 22

- group: html
  values:
    - protocol: tcp
      port: 80

- template: Acl
  values:
    - acl_name: http_acl
      $group_domain: first
      $group_traffic: http
      action: permit

    - acl_name: ssh_acl
      $group_domain: second
      $group_traffic: ssh
      action: denyIn the above example, data sets for domains and traffic types are defined as
groups.  These are being applied to the http_acl and ssh_acl templates using
$group fields and referencing by group name.  The template will inherit all of
the values from any group that is referenced.  The result in this case is that
http_acl is created in the first domain and permits http traffic (tcp port 80)
and the ssh_acl is created in the second domain and denies ssh traffic (tcp
port 22).Group definitions can be children or have children.  However, they should only
define one set of values.  The field name for group references must start with
the string "$group", but can have any suffix following it.  The purpose of the
suffix is to allow multiple group references without name collision.  Although
any suffix can be specified, it is recommended to choose one that is
descriptive of the reference meaning.Field and Value ListsAn alternative way of specifying values is by using a field list and set of value
lists.  This format is similar to Comma Separated Values (CSV) data and
can efficiently specify a large amount of data.- template: Acl
  fields: ['acl_name', '$group_domain', 'protocol', 'port', 'action']
  values:
      -   ['acl1',     'first',         'tcp',      22,     'deny']
      -   ['acl2',     'first',         'udp',      5000,   'permit']
      -   ['acl3',     'second',        'tcp',      80,     'deny']
      -   ['acl4',     'second',        'udp',      5002,   'permit']A fields list must be specified to define the field name for each position of
the value lists.  The values data sets must be specified as lists with the same
length as the fields list.Listing TemplatesThe templates that have been loaded into the MetroAE config tool can be listed
with the following:$ metroae config -tp sample/templates --list

Domain
Enterprise
Subnet
ZoneGenerating User Data ExamplesAn example of user data for any template can be provided using the following:metroae config example -tp sample/templates Domain

# First template set - Create a L3 Domain
- template: Domain
  values:
    - enterprise_name: ""                      # (reference)
      domain_name: ""                          # (string)
      description: ""                          # (opt string)
      underlay_enabled: enabled                # (['enabled', 'disabled', 'inherited'])
      address_translation: enabled             # (['enabled', 'disabled', 'inherited'])Generating JSON SchemasA JSON schema can be generated for the user data required for any template.
These schemas conform to the json-schema.org standard specification:metroae config schema -tp sample/templates Domain

{
  "title": "Schema validator for Nuage Metro config template Domain",
  "$id": "urn:nuage-metro:config:template:domain",
  "required": [
    "enterprise_name",
    "domain_name",
    "underlay_enabled",
    "address_translation"
  ],
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "underlay_enabled": {
      "enum": [
        "enabled",
        "disabled",
        "inherited"
      ],
      "title": "Underlay enabled"
    },
    "address_translation": {
      "enum": [
        "enabled",
        "disabled",
        "inherited"
      ],
      "title": "Address translation"
    },
    "domain_name": {
      "type": "string",
      "title": "Domain name"
    },
    "enterprise_name": {
      "type": "string",
      "title": "Enterprise name"
    },
    "description": {
      "type": "string",
      "title": "Description"
    }
  }
}Query ToolDescriptionMetroAE config can be used as a generic data querying tool to extract
information from devices.  At the time of this writing VSD and ES are
supported.  The tool uses a simple language to describe the data that is to be
retrieved.  The data will be displayed to the screen in a YAML like format, but
it could (in addition) be written to a file.  The results are also returned as
a Python list when using query tool as a library.UsageThe query tool is used by specifying "query" as the action tometroae_config.
The parameters are mostly the same as other actions supported bymetroae_config.  The VSD specification path-spis required if querying a
VSD.  The other VSD parameters (i.e. user, pass, URL, etc) also apply for
query in the same way as configuration creation actions.The query tool also supports queries to ElasticSearch (ES).  In this case, a
new-esparameter has been added to specify the address of the ES to query.Thequery languagecan be specified to the tool directly on the command-line
for easy manual use using the-qquery parameter.  An example follows:metroae config query -v https://localhost:8443 -q 'enterprise.name'

Device: Nuage Networks VSD 20.5.1
- Shared Infrastructure
- public
- private
>>> All actions successfully appliedMultiple queries can be specified on a single line using semicolon;:metroae config query -v https://localhost:8443 -q 'apps = SAASApplicationType.name; ents = enterprise.name'

Device: Nuage Networks VSD 20.5.1
apps:
- Office365
- SalesForce
- WebEx
ents:
- Shared Infrastructure
- public
- private
>>> All actions successfully appliedThe language definition for query tool can also be specifed in a file to be
read.  One or more of these files (or a directory containing .query files) can
be specifed on the command-line for the tool to read and execute each file.
Newlines in files separate commands.  However, semicolons are still
supported in files as well.example.query:

apps = SAASApplicationType.name
ents = enterprise.name

metroae config query -v https://localhost:8443 example.query

Device: Nuage Networks VSD 20.5.1
apps:
- Office365
- SalesForce
- WebEx
ents:
- Shared Infrastructure
- public
- private
>>> All actions successfully appliedLanguage DefinitionThe query tool uses aquery languageto specify the data to be reteived from
devices.  This language is intended to be simple to read and use and is easily
extendable.General FormattingEach query to be performed is specified one per-line.  Alternatively and
equivalently, semicolons;can be used to separate multiple queries on a
single line.  Any query beginning with a hash#indicates a comment and will
be ignored.  Outside of strings, any whitespace is ignored.Data RetrievalData retrieval commands are specified by providing nested object names
separated by dot.with the attribute as the last identifier.  A retrieval
traverses all instances of each level of the specified object tree and returns
the attribute value for every instance found.The result is always a flat list of the attribute values.  Any heirarchy is
flattened.  If no results are found or there is an error (such as missing child
object) then an empty list is returned.  With this procedure, processing
results is consistent as there will always be a list.For VSD:Enterprise.Domain.Zone.Subnet.nameThe above (reading from right to left) returns a list of the names from all
subnets under all zones under all domains under all enterprises.  For VSD
queries, each object is a configuration object specified by entity name.  The
names here are case-insensitve although note that case may matter for other
devices (like ES).For ElasticSearch:nuage_sysmon.disks.availableThe above (reading from right to left) returns the available values for all
disks under the nuage_sysmon index.  For ES, the first object is the ES index
and then subsequent object names traverse the JSON objects stored in each ES
record.  Since ES encodes using JSON, the object names and attribures are all
case-sensitive as required by the format.AttributesThe last identifier on a data retrieval is the attribute.  It determines which
value to extract from the objects traversed.  In previous examples, a single
value was extracted, however multiple attributes can be gathered by specifying
a comma-separated list in curly braces{}.  When retrieving multiple
attributes, the result becomes a list of dictionaries with key/value pairs
for each of the attributes.  A star*can be used to get all available
attributes.Single attribute:Enterprise.name

- Shared Infrastructure
- public
- privateMultiple attributes:Enterprise.{name,id}

- name: Shared Infrastructure
  id: abcd-0001
- name: public
  id: abcd-0002
- name: private
  id: abcd-0003All attributes:Enterprise.{*}

- name: Shared Infrastructure
  id: abcd-0001
  bgpenabled: false
  dhcpleaseinterval: 24
  (Whole bunch of stuff omitted for brevity...)
- name: public
  id: abcd-0002
  bgpenabled: false
  dhcpleaseinterval: 24
  (Whole bunch of stuff omitted for brevity...)
- name: private
  id: abcd-0003
  bgpenabled: false
  dhcpleaseinterval: 24
  (Whole bunch of stuff omitted for brevity...)FiltersFilters can be applied to each object of the data retrieval.  Filters are
enclosed by square brackets[]and are separated by and&characters.  The
choice of&as a separator enforces that filters are combined together as
logical AND operations.The filter applies only to the level where it is specified.  The filters
manipulate the results from the level, usually by removing unwanted records.
Any subsequent object levels would operate on the filtered list.  For example:Enterprise[name=public].domain.nameThe filter on enterprise will limit enterprise results to only a single
enterprise object (with namepublic).  However, let's say thepublicenterprise had 10 domains.  Then the 10 name results for those domains would be
returned.Range FilterA range filter can be used to limit the start and max number of results at the
level.  The filter has a start index and an end index separated by colon:.
Indexes are 0-based positions of the list.  Thus 0 is the first and 4 would be
the 5th item in the list.  The items from the query are returned starting at
the start index and ending one before the end index.  Any range out of bounds
of the results are omitted.  Thus if the start index is after the end of the
results, an empty list is returned.If the start index is omitted, a value of 0 is assumed (start at beginning).
If the end index is omitted, then the entire remainder of the list is returned.A single index (without colon specified) can be used to return the single
item from the results at the specified index.  If this is out of bounds of the
results, an empty list is returned.Negative indicies are allowed and count from the end of the list.  Thus -1 is
the last item and -5 is the 5th from last.The behavior of the range filter is exactly the same as Python list slicing.Note that ES index queries do not support negative range indicies.  Use a
reverse sort and positive indicies instead.Examples using enterpises:e1, e2, e3, e4, e5enterprise[1:5].name
- e2
- e3
- e4

enterprise[1:99].name
- e2
- e3
- e4
- e5

enterprise[:5].name
- e1
- e2
- e3
- e4

enterprise[3:].name
- e4
- e5

enterprise[99:].name
[]

enterprise[3].name
- e4

enterprise[99].name
[]

enterprise[-3:-1].name
- e3
- e4Attribute FilterObject results can be filtered by attribute values.  Only results where the
field value of the object matches the filter value will be returned.  The
format for these filters is the field name equal=value.  A list of values
can be provided enclosed in square brackets[]and separated by comma,.
This denotes that any result matching any of the list values will be returned.Given these enterprises:- name: e1
  BGPEnabled: true
- name: e2
  BGPEnabled: false
- name: e3
  BGPEnabled: trueThe following examples produce:enterprise[BGPEnabled=true].name
- e1
- e3

enterprise[name=[e2, e3]].name
- e2
- e3

enterprise[name=[e2, e3] & BGPEnabled=false].name
- e2

enterprise[name=[e1, e3] & BGPEnabled=false].name
[]Sort FilterThe sort filter can be used to sort the results by an object field.  Note that
the sorting occurs only at the level where it is specified.  If there are
child objects under a sort fitler the children would not be sorted, but they
would be queried in the parent's sorted order.Sorting is specified by the%sort=keyword followed by the field name of the
object to sort by.  There is also%sort_desc=to sort in descending order.Given these enterprises:- name: e3
  dhcpleaseinterval: 1
- name: e1
  dhcpleaseinterval: 2
- name: e2
  dhcpleaseinterval: 3The following examples produce:enterprise[%sort=name].name
- e1
- e2
- e3

enterprise[%sort=DHCPLeaseInterval].name
- e3
- e1
- e2

enterprise[%sort_desc=name].name
- e3
- e2
- e1Group FilterA group filter can be specified to combine together results by the value of
another field.  This makes it easier to identify subsets of results given that
the query tool collapses them all into flat lists.The group filter is specified by%group=followed by the field name to group
by.  When using groups, results are returned as a list of pairs.  The first
item of the pair is the group field value and the second is the list of results
matching the group field value.For example, normally results are flattened and thus it is difficult to
determine which domain goes with which enterprise:enterprise.domain.name

- domain1
- domain2
- domain3
- domain4However, using grouping the relationships can be determined:enterprise[%group=name].domain.name

- - public
  - - domain1
    - domain2
    - domain3
- - private
  - - domain4
- - Shared Infrastructure
  - []VariablesResults can be stored in variables for identification and use in later queries.
Variables are restricted to C-like names which cannot start with a number and
can contain capital letters, lower case letters, numbers or underscore_.Variable assignments are performed by specifying the variable name then equal=then the expression or data type to store in the variable.  The output of
assigments changes to a dictionary.enterprise_names = enterprise.name

enterprise_names:
- public
- private
- Shared InfrastructureVariables are dynamically typed and can contain any data type.  More
information about data types will be provided in a later section.  Variables
can be dereferenced in later queries by specifying dollar$and the variable
name.  Variables that have been dereferenced but not yet defined will cause an
error.Variables can be filtered:enterprises = enterprise.{name,id}

enterprises:
- name: public
  id: abcd-0001
- name: private
  id: abcd-0002

$enterprises[%sort=name]

- name: private
  id: abcd-0002
- name: public
  id: abcd-0001

$enterprises[%sort=name].id

- abcd-0002
- abcd-0001Variables can also be used as the values for filters:enterprises = enterprise.name

enterprises:
- e1
- e2
- e3
- e4

start = 2

start: 2

$enterprises[$start:]
- e3
- e4

name = "e2"

name: e2

enterprise[name=$name].{name,id}
- name: e2
  id: abcd-0002Variable values can be specified on the command-line using the-doption.
The format is variable name=value. i.e.-d name=enterprise1.  When
variables are defined on the command-line, they override any assignment
within queries.  This allows defaults to be set inside query files and-dto
override with a specific value.Data TypesThe query language defines various data types to be specified.  Variables are
dynamically typed and can accept any of these types.Strings are defined enclosed with single'or double"quotes.  Multi-line
blocks can be defined using triple single'''or double"""quotes.string = 'this is a string'
string = "another has ' in it"
multiline = """
this has
multiple lines
"""Numbers can be specified as positive or negative integers.  Floating point
numbers are not supported.integer = 42
negative = -4For Booleans,truespecifies true andfalsespecifies false.is_true = true
is_false = falseLists can be formed using square brackets[]and separating items by comma,.  Lists can contain any of the above data types even in combinations.
However, lists of lists are not supported.enterprise_names = ['e1', 'e2', 'e3']FunctionsThere are a set of functions that can operate on expressions such as data
retrieval or variables.  These come in the form of function name followed by
the expression enclosed in parens().Thecountfunction returns the number of items in a list.  An error is
produced if the expression provided to count is not a list.names = enterprise.name

names:
- e1
- e2
- e3

count($names)
3Thereversefunction returns the items of a list in backward order.  An error
is produced if the expression provided to reverse is not a list.reverse(enterprise.name)
- e3
- e2
- e1CombineThere is acombineoperator which combines expressions together depending on
the data types.  The format is expression1 plus+expression2.  The
expressions can be variables, data type constants or data retrieval.If both expressions are lists, then the combine operator will return a list
with all of the items of both.names = enterprise.name

names:
- e1
- e2
- e3

$names + ["new1", "new2"]
- e1
- e2
- e3
- new1
- new2If both expressions are integers, then the combine operator will add the
values.count(enterprise.name) + 1
4If both expressions are strings, then the strings will be concatenated.filename = "output"
filename: output

path = "/tmp/" + $filename + ".txt"
path: /tmp/output.txtAny other combination of data types will result in an error from combine.ActionsThere are a set of actions that can change the state of later queries or
perform a task.  These come in the form of action name followed by a list of
comma-separated arguments enclosed in parens().  Actions do not return any
value and they cannot be part of an assignment or expression.Theconnectaction sets up a new session with a device.  All subsequent
queries are directed to the new device.  The first argument to connect is the
device type, then the remaining parameters are specific to the device.  Using
connect allows data to be gathered from multiple sources in the same query
execution.For VSD:connect("VSD", url, username, password, enterprise, cert_file, cert_key_file)

Where:
    url (required)           : URL of VSD to connect to
    username (optional)      : Username to be used to connect (default "csproot")
    password (optional)      : Password to be used to connect (default "csproot")
    enterprise (optional)    : Enterprise to be used to connect (default "csp")
    cert_file (optional)     : Path to a certificate file (instead of password)
    cert_key_file (optional) : Path to a certificate key file (required if cert_file specified)For ElasticSearch:connect("ES", address, port)

Where:
    address (required) : Address of ElasticSearch to connect to
    port (optional)    : Port of ElasticSearch (default 9200)Theredirect_to_fileaction causes all subsequent queries to write results
to the specified file in addition to echoing to the screen.redirect_to_file("/tmp/output.txt")Theechoaction turns on and off the printing of output of subsequent queries
to the screen.  This only affects the display of the queries and they still
operate normally in all other ways.  Variables would still be assigned and
output would still be written to file if redirected.  Echoing is on by default.
it can be turned off with the stringoffand turned back on with the stringon.  Disabling echoing is useful when huge amounts of data are expected to be
returned but it would clutter the screen.echo("off")Theoutputaction turns on and off the writing of results to the screen,
redirect file and Python returned results.  Variables are still assigned when
output is off.  Turning off output is useful for gathering all intermediate
variables and then turning it on to write out a single final report.  Output is
on by default.  It can be turned off with the stringoffand turned back on
with the stringon.output("off")Therender_templateaction outputs the text of a variable substituted
template.  The action takes aPython Jinjatemplate string as an argument.  Jinja templates contain tags enclosed by
double curly braces{{}}that reference variables to be substituted into
the text.  The variables assigned from queries are all available to be injected
into the text output.  Jinja templates support loops, conditionals, math and
other filters.  See thePython Jinjadocumentation for more. Using Jinja templates, many file formats can be output
including nice text reports, HTML pages, CSV or JSON.Therender_yaml_templateaction is exactly the same asrender_templateexcept that substituted variables are formatted properly for YAML/JSON output.
For example strings are quoted, booleans are converted to true/false, etc.Example query file:output("off")

connect("VSD", "https://localhost:8443")

report_file = "report.txt"
ent_names = enterprise.name
ent_count = count($ent_names)
licenses_expires = License.expirytimestamp

template = """============== VSD Health Report ===========
VSD:
    Enterprises: {{ ent_names | join(', ') }} ({{ ent_count }})
    License expiry: {{ ((licenses_expires[0] / 1000 - now) / (3600 * 24)) | int }} days
===========================================
"""

redirect_to_file($report_file)
output("on")
render_template($template)Resulting output toreport.txt:============== VSD Health Report ===========
VSD:
    Enterprises: public, private, Shared Infrastructure (3)
    License expiry: 90 days
===========================================Python Library APIThe MetroAE Config engine is written entirely using Python.  It can be integrated
into other Python-based tools using the library APIs via the pip packagenuage-metroae-config.  The following sections describe the use of the APIs.Template Store ClassTheTemplateStoreclass is a repository for all of the templates loaded into
the engine.  It will be used by theConfigurationclass to apply template
actions to a device via aDeviceWriterBaseclass.from nuage_metroae_config.template import TemplateStoreReads and parses configuration templates.read_templates(path_or_file_name)Reads and parses templates from either all templates in a
  directory path, or a single template specified by filename.
  Both yaml (.yml) and JSON (.json) files are supported.add_template(template_string, filename=None)Parses the specified string as a template in Yaml or JSON format.get_template_names(software_type=None, software_version=None)Returns a list of all template names currently loaded in store.
  If software_version and/or software_type is provided, names will
  be filtered by the specified version/type.get_template(name, software_type=None, software_version=None)Returns a Template object of the specified name.  If software_version
  and/or software_type is provided, template of specified version/type
  will be returned.Template ClassTheTemplateclass is a read-only class which provides information about a
template that was read in via theTemplateStore.from nuage_metroae_config.template import TemplateConfiguration template.  This class is read-only.get_name()Returns the name of this template.get_template_version()Returns the template versionget_software_version()Returns a dictionary of {"software_version": "xxx", "software_type": "xxx"}get_schema()Returns the schema for the template variables in json-schema form.get_example()Returns example user-data for template variables in YAML format.get_documentation()Returns template documentation in MarkDown format.validate_template_data(template_data)Validates that the template_data provided matches the variables schema.
  Returns True if ok, otherwise an exception is raised.VSD Writer ClassTheVsdWriterclass is a derived class of theDeviceWriterBasethat applies
configuration to a Nuage Networks VSD.  The writers are modular such that
new devices could be configured by defining derived device writer classes based
off of the commonDeviceWriterBasebase class.  The remainder of the engine
APIs use the generic base class.  This provides great extendability to
configure many different devices.  Note that theVsdWriterclass also acts as
a reader for theQueryclass.from nuage_metroae_config.vsd_writer import VsdWriterWrites configuration to a VSD.  This class is a derived class from
the DeviceWriterBase Abstract Base Class.set_session_params(url, username="csproot", password=None, enterprise="csp", certificate=None)Sets the parameters necessary to connect to the VSD.  This must
  be called before writing or an exception will be raised.get_version():Returns the version running on the VSD in format:
      {"software_version": "xxx", "software_type": "xxx"}set_software_version(software_version):Sets the software version of the VSD.  This can be obtained via the
  get_version() method to get from the VSD itself or overwriten
  to a specific value.read_api_specifications(path_or_file_name):Reads the VSD configuration API specifications from JSON files
  in the specified path or file name.  This must be called before
  writing or an exception will be raised.set_logger(logger):Set a custom logger for actions taken.  This should be based on the
  logging Python library.  It will need to define an 'output' log level
  which is intended to print to stdout.User Data Parser ClassTheUserDataParserclass parses the YAML-based user-data format defined in
theUser Datasection of this document.  It is able to return the data in
template_name/data_dictionary pairs.  This data can be fed into theConfigurationclass to render templates and apply configuration.from nuage_metroae_config.user_data_parser import UserDataParserParses Yaml or JSON files containing template user dataread_data(path_or_file_name):Reads and parses user data from either all files in a
  directory path, or a single file specified by filename.
  Both yaml (.yml) and JSON (.json) files are supported.add_data(user_data_string):Parses the specified string as user data in Yaml or JSON format.get_template_name_data_pairs():Returns parsed data as a list of (template_name, data_dict) pairs.Configuration ClassTheConfigurationclass puts all the above pieces together to configure a
device.  It requires aTemplateStoreobject with loaded templates.  User data
is applied to theConfigurationobject which can be taken directly from theUserDataParserclass or from pure Python data via another source.  Finally,
the configuration is applied, updated or reverted to a device using one of the
derived writer classes.from nuage_metroae_config.configuration import ConfigurationContainer for template instances.set_software_version(software_type=None, software_version=None)Sets the current software version of templates that is desired.
  If not called, the latest software version of templates will be
  used.add_template_data(template_name, template_data)Adds template data (user data) for the specified template name.
  Data is specified in a kwargs dictionary with keys as the
  attribute/variable name.  The data is validated against the
  corresponding template schema.  An id is returned for reference.apply(writer)Applies this configuration to the provided device
  writer.  Returns True if ok, otherwise an exception is
  raised.update(writer)Applies this configuration to the provided device
  writer as an update.  This means objects that exist will
  not be considered conflicts.  Returns True if ok, otherwise
  an exception is raised.revert(writer)Reverts (removes or undo) this configuration from the
  provided device writer.  Returns True if ok, otherwise
  an exception is raised.set_logger(logger):Set a custom logger for actions taken.  This should be based on the
  logging Python library.  It will need to define an 'output' log level
  which is intended to print to stdout.Query ClassTheQueryclass allows information to be retrieved from a device through
device readers based on theDeviceReaderBase.  Note that theVsdWriterclass is also a reader derived off of this base and can be used for query.  The
information to be retrieved is defined by a "query language" which is described
in theLanguage Definitionsection of this document.  Results are returned
from theexecute()of the query language text.  Also, any variables set during
the query can be retrieved withget_variables().from nuage_metroae_config.query import QueryA class which can retrieve information from a device using a
DeviceReaderBase reader object.  The information to be retrieved is defined
by query language text.set_reader(reader)Set the primary reader object for performing queries on the device.
  The reader needs to be derived from the DeviceReaderBase class.get_variables()Returns a dictionary of all variables and values set during queries.register_reader(reader_type, reader)Register a reader object for each 'connect' type for performing queries
  on the device.  This allows devices to be switched within the query
  text.  The readers need to be derived from the DeviceReaderBase class.add_query_file(path_or_file_name)Reads and parses query sets from either all query files in a
  directory path, or a single query file specified by filename.
  Query files are expected to have .query extensionexecute(query_text=None, override_variables)Executes the query defined in query_text.  Variables in the query
  text can be overriden using the override_variables kwargs dict.  The
  results of the query are returned as a list with an entry for each
  query text line.  If query_text is None, then query text is pulled
  from query files defined by add_query_file.set_logger(logger)Set a custom logger for actions taken.  This should be based on the
  logging Python library.  It will need to define an 'output' log level
  which is intended to print to stdout.ES Reader ClassTheEsReaderclass can be used as another reader with theQueryclass.  It
is derived from theDeviceReaderBaseclass and can retrieve information from
ElasticSearch.from nuage_metroae_config.es_reader import EsReaderPerforms queries on ElasticSearch.  This class is a derived class from
the DeviceReaderBase Abstract Base Class.set_session_params(address, port=None):Sets the parameters necessary to connect to the ES.  This must
  be called before reading or an exception will be raised.Library API Example - Configfrom nuage_metroae_config.configuration import Configuration
from nuage_metroae_config.template import TemplateStore
from nuage_metroae_config.user_data_parser import UserDataParser
from nuage_metroae_config.vsd_writer import VsdWriter

ENGINE_VERSION = "1.1.0"
VSD_API_SPEC_PATH = "./vsd-api-specifications"
TEMPLATE_PATH = "./templates"
USER_DATA_PATH = "./user_data"
VSD_URL = "https://10.0.0.1:8443"
VSD_USERNAME = "username"
VSD_PASSWORD = "password"
VSD_ENTERPRISE = "csp"


def setup_template_store(template_path):
    store = TemplateStore(ENGINE_VERSION)

    store.read_templates(template_path)

    return store


def setup_vsd_writer(vsd_url):
    vsd_writer = VsdWriter()
    vsd_writer.add_api_specification_path(VSD_API_SPEC_PATH)

    vsd_writer.set_session_params(vsd_url,
                                  username=VSD_USERNAME,
                                  password=VSD_PASSWORD,
                                  enterprise=VSD_ENTERPRISE)

    device_version = vsd_writer.get_version()
    vsd_writer.set_software_version(device_version)

    return vsd_writer


def setup_configuration(template_store, user_data_path):
    config = Configuration(template_store)

    parser = UserDataParser()
    template_data_pairs = parser.read_data(user_data_path)

    for data in template_data_pairs:
        template_name = data[0]
        template_data = data[1]
        config.add_template_data(template_name, **template_data)

    return config


# Setup
template_store = setup_template_store(TEMPLATE_PATH)
vsd_writer = setup_vsd_writer(VSD_URL)
config = setup_configuration(template_store, USER_DATA_PATH)

# Apply the configuration from user data
config.apply(vsd_writer)

# Revert the configuration from user data
config.revert(vsd_writer)Library API Example - Queryfrom nuage_metroae_config.es_reader import EsReader
from nuage_metroae_config.query import Query
from nuage_metroae_config.vsd_writer import VsdWriter

VSD_API_SPEC_PATH = "./vsd-api-specifications"
VSD_URL = "https://10.0.0.1:8443"
VSD_USERNAME = "username"
VSD_PASSWORD = "password"
VSD_ENTERPRISE = "csp"

ES_ADDRESS = "10.0.0.2"
ES_PORT = 9200


def setup_vsd_query_tool(vsd_url):
    vsd_reader = VsdWriter()
    vsd_reader.add_api_specification_path(VSD_API_SPEC_PATH)

    vsd_reader.set_session_params(vsd_url,
                                  username=VSD_USERNAME,
                                  password=VSD_PASSWORD,
                                  enterprise=VSD_ENTERPRISE)

    device_version = vsd_writer.get_version()
    vsd_reader.set_software_version(device_version)

    vsd_query = Query()
    vsd_query.set_reader(vsd_reader)

    return vsd_query


def setup_es_query_tool(es_address):
    es_reader = EsReader()
    es_reader.set_session_params(es_address, ES_PORT)

    es_query = Query()
    es_query.set_reader(es_reader)

    return es_query


def get_subnetwork_usage(vsd_query):

    query_string = """
    subnets = Enterprise.Domain.Zone.Subnet.{name, netmask}
    zones = Enterprise.Domain.Zone[%group=name].Subnet.{name, netmask}
    d_count = count(Enterprise.Domain)
    s_count = count(Enterprise.Domain.Zone.Subnet)
    z_count = count(Enterprise.Domain.Zone)
    """
    vsd_query.execute(query_string)

    results = vsd_query.get_variables()

    domain_dict = dict()
    domain_dict_agg = dict()

    # [{"name": <subnet_name>, "netmask": <netmask>}, ...]
    domain_dict["subnets"] = results["subnets"]
    # [[<zone_name>, [{"name": <subnet_name>, "netmask": <netmask>}, ...]], ...]
    domain_dict["zones"] = results["zones"]
    domain_dict_agg["d_count"] = results["d_count"]
    domain_dict_agg["s_count"] = results["s_count"]
    domain_dict_agg["z_count"] = results["z_count"]

    return domain_dict, domain_dict_agg


def get_enterprises_with_alarms(vsd_query):

    query_string = """
    Enterprise[%group=name].Alarm[severity="CRITICAL"].description
    """
    results = vsd_query.execute(query_string)

    # [[<enterprise_name>, [{"description": <alarm_descr>}, ...]], ...]

    enterprises_with_critical_alarms = dict()
    count = 0

    for result in results:
        ent_name = result[0]
        alarms = result[1]
        if len(alarms) > 0:
            enterprises_with_critical_alarms[ent_name] = alarms
            count += 1

    enterprises_with_critical_alarms['total_enterprises_with_alarms'] = count

    return enterprises_with_critical_alarms


# Setup query
vsd_query = setup_vsd_query_tool(VSD_URL)

# Perform queries
domain_dict, domain_dict_agg = get_subnetwork_usage(vsd_query)
enterprises_with_critical_alarms = get_enterprises_with_alarms(vsd_query)