Package: grifter
# GrifterPython library to build large scale Vagrant topologies for the networkingspace. Can also be used the build small scale labs for networking/computedevices.[![Build Status](https://travis-ci.org/bobthebutcher/grifter.svg?branch=master)](https://travis-ci.org/bobthebutcher/grifter.svg?branch=master)[![Coverage Status](https://coveralls.io/repos/github/bobthebutcher/grifter/badge.svg?branch=master)](https://coveralls.io/github/bobthebutcher/grifter?branch=master)NOTE: Python 3.6+ is required to make use of this library.```*****************************************************************This project is currently in beta and stability is not currentlyguaranteed. Breaking API changes can be expected.*****************************************************************```## VagrantWhat is Vagrant? From the Vagrant [website](https://www.vagrantup.com/docs/index.html)```A command line utility for managing the lifecycle of virtual machines```## Vagrant LibvirtWhat is Vagrant Libvirt? From the `vagrant-libvirt` github [page](https://github.com/vagrant-libvirt/vagrant-libvirt)```A Vagrant plugin that adds a Libvirt provider to Vagrant, allowing Vagrant to control and provision machines via Libvirt toolkit.```## WhyWhen simulating large topologies Vagrantfiles can become thousandsof lines long. Getting all the configuration correct is often afrustrating, error riddled process especially for those not familiarwith Vagrant. Grifter aims to help simplify that process.##### Additional project goals- Generate topology.dot files for use with PTM :heavy_check_mark:- Generate Inventory files for tools such as Ansible, NornirNOTE: Only a `vagrant-libvirt` compatible `Vagrantfile` forVagrant version `>= 2.1.0` will be generated.Support for Virtualbox or any other provider type is not supported oron the road map.## DependenciesGrifter requires the help of the following awesome projects from the Pythoncommunity.- [Cerberus](http://docs.python-cerberus.org/en/stable/) - Schema validation- [Click](https://click.palletsprojects.com/) - CLI utility- [Jinja2](http://jinja.pocoo.org/docs) - Template engine- [PyYAML](https://pyyaml.org/) - YAML all the things## InstallationThere is currently no PyPI release for this project. Grifter can beinstalled directly from source using PIP.Create and activate virtualenv.```mkdir ~/test && cd ~/testpython3 -m venv .venvsource .venv/bin/activate```Install `grifter` with `pip````# Install the master branch.pip install https://github.com/bobthebutcher/grifter/archive/master.zip```Releases are distributed via Github Releases.```# Install the latest release.pip install https://github.com/bobthebutcher/grifter/archive/v0.2.11.zip```## Quick StartCreate a `guests.yml` file.```tee guests.yml > /dev/null << "EOF"srv01:vagrant_box:name: "centos/7"EOF```Generate a Vagrantfile```grifter create guests.yml```Let Vagrant do its magic```vagrant up```## Config FileA file named `config.yml` is required to define the base settings ofeach box managed within the grifter environment. The default `config.yml`file can be found [here](grifter/config.yml)### Box NamingGrifter expects Vagrant boxes to be named according to the following list.##### Custom Boxes- arista/veos- cisco/csr1000v- cisco/iosv- cisco/xrv- juniper/vmx-vcp- juniper/vmx-vfp- juniper/vqfx-pfe- juniper/vqfx-re- juniper/vsrx- juniper/vsrx-packetmode##### Vagrant Cloud Boxes- CumulusCommunity/cumulus-vx- centos/7- generic/ubuntu1804- opensuse/openSUSE-15.0-x86_64#### guest_configThe `guest_config` section defines characteristics about the Vagrant boxesused with grifter.#### Required Parameters.- data_interface_base- data_interface_offset- max_data_interfaces- management_interfaceNote: `data_interface_base` cannot be an empty string. If the box does nothave any data interfaces the suggested value is "NA". This field will beignored so it can be anything as long as it is not empty.```yamlguest_config:example/box:data_interface_base: "eth" # String pattern for data interfaces.data_interface_offset: 0 # Number of first data interface ie: 0, 1, 2, etc..internal_interfaces: 0 # Used for inter-box connections for multi-vm boxes.max_data_interfaces: 8management_interface: "ma1"reserved_interfaces: 0 # Interfaces that are required but cannot be used.arista/veos:data_interface_base: "eth"data_interface_offset: 1internal_interfaces: 0max_data_interfaces: 24management_interface: "ma1"reserved_interfaces: 0juniper/vsrx-packetmode:data_interface_base: "ge-0/0/"data_interface_offset: 0internal_interfaces: 0max_data_interfaces: 16management_interface: "fxp0.0"reserved_interfaces: 0```#### guest_pairsThe `guest_pairs` section is used the define boxes that need two VMs tobe fully functional. Some examples are the Juniper vMX and vQFX whereone box is used for the control-plane and another for the forwarding-plane.NOTE: This functionality will be added in a future release.#### Custom config filesA default config file ships with the grifter python package.This file can be customized with your required parameters by creating a`config.yml` file in the following locations.- `/opt/grifter/`- `~/.grifter/`- `./`Parameters in a users `config.yml` file will be merged with the default`config.yml` file with the user-defined parameters taking preference.## Usage#### CLI UtilityGrifter ships with a CLI utility. Execute `grifter -h` todiscover all the CLI options.```grifter -hUsage: grifter [OPTIONS] COMMAND [ARGS]...Create a Vagrantfile from a YAML data input file.Options:--version   Show the version and exit.-h, --help  Show this message and exit.Commands:create   Create a Vagrantfile.example  Print example file declaration.```#### Create Vagrantfile```grifter create guests.yml```### Guests DatafileGuest VMs characteristics and interface connections are defined in a YAML file.This file can be named anything, but the recommended naming convention is`guests.yml`.#### Guest SchemaJinja2 is used a the templating engine to generate the Vagrantfiles.Guests definition within a guests file must use the followingschema as it is required to ensure templates render correctly andwithout errors. The guest data will be validated against the schemausing the Cerberus project.```yamlsome-guest: # guest namevagrant_box: # vagrant_box parametersname: # string - requiredversion: # string - optional | default: ""url: # string - optional | default: ""provider: # string - optional | default: "libvirt"guest_type: # string - optional | default: ""boot_timeout: # integer - optional | default: 0throttle_cpu: # integer - optional | default: 0ssh: # dict - optionalusername: # string - optional | default: ""password: # string - optional | default: ""insert_key: # boolean - optional | default: Falsesynced_folder: # dict - optionalenabled: # boolean - default: Falseid: # string - default: "vagrant-root"src: # string - default: "."dst: # string - default: "/vagrant"provider_config: # dict - optionalrandom_hostname: # boolean - optional | default: Falsenic_adapter_count: # integer - optional | default: 0disk_bus: # string - optional | default: ""cpus: # integer - optional | default: 1memory: # integer - optional | default: 512huge_pages: # boolean - optional | default: Falsestorage_pool: # string - optional | default: ""additional_storage_volumes: # list - optional# For each list element the following is required.- location: # stringtype: # stringbus: # stringdevice: # stringnic_model_type: # string - optional | default: ""management_network_mac: # string - optional | default: ""internal_interfaces: # list - optional# For each list element the following is required.- local_port: # integerremote_guest: # stringremote_port: # integerdata_interfaces: # list - optional# For each list element the following is required.- local_port: # integerremote_guest: # stringremote_port: # integer```#### Example DatafileThe following example datafile defines two `arista/veos` switches connectedtogether on ports 1 and 2.```yamlsw01:vagrant_box:name: "arista/veos"version: "4.20.1F"guest_type: "tinycore"provider: "libvirt"ssh:insert_key: Falsesynced_folder:enabled: Falseprovider_config:nic_adapter_count: 2disk_bus: "ide"cpus: 2memory: 2048data_interfaces:- local_port: 1remote_guest: "sw02"remote_port: 1- local_port: 2remote_guest: "sw02"remote_port: 2sw02:vagrant_box:name: "arista/veos"version: "4.20.1F"guest_type: "tinycore"provider: "libvirt"ssh:insert_key: Falsesynced_folder:enabled: Falseprovider_config:nic_adapter_count: 2disk_bus: "ide"cpus: 2memory: 2048data_interfaces:- local_port: 1remote_guest: "sw01"remote_port: 1- local_port: 2remote_guest: "sw01"remote_port: 2```#### Generated Vagrantfile```ruby# -*- mode: ruby -*-# vi: set ft=ruby :def get_mac(oui="28:b7:ad")"Generate a MAC address"nic = (1..3).map{"%0.2x"%rand(256)}.join(":")return "#{oui}:#{nic}"endcwd = Dir.pwd.split("/").lastusername = ENV['USER']domain_prefix = "#{username}_#{cwd}"domain_uuid = "1f22b55d-2d7e-5a24-b4fa-3a8878df5cc5"Vagrant.require_version ">= 2.1.0"Vagrant.configure("2") do |config|config.vm.define "sw01" do |node|guest_name = "sw01"node.vm.box = "arista/veos"node.vm.box_version = "4.20.1F"node.vm.guest = :tinycorenode.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: truenode.ssh.insert_key = falsenode.vm.provider :libvirt do |domain|domain.default_prefix = "#{domain_prefix}"domain.cpus = 2domain.memory = 2048domain.disk_bus = "ide"domain.nic_adapter_count = 2endnode.vm.network :private_network,# sw01-eth1 <--> sw02-eth1:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.146.53.1",:libvirt__tunnel_local_port => 10001,:libvirt__tunnel_ip => "127.146.53.2",:libvirt__tunnel_port => 10001,:libvirt__iface_name => "sw01-eth1-#{domain_uuid}",auto_config: falsenode.vm.network :private_network,# sw01-eth2 <--> sw02-eth2:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.146.53.1",:libvirt__tunnel_local_port => 10002,:libvirt__tunnel_ip => "127.146.53.2",:libvirt__tunnel_port => 10002,:libvirt__iface_name => "sw01-eth2-#{domain_uuid}",auto_config: falseendconfig.vm.define "sw02" do |node|guest_name = "sw02"node.vm.box = "arista/veos"node.vm.box_version = "4.20.1F"node.vm.guest = :tinycorenode.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: truenode.ssh.insert_key = falsenode.vm.provider :libvirt do |domain|domain.default_prefix = "#{domain_prefix}"domain.cpus = 2domain.memory = 2048domain.storage_pool_name = "disk1"domain.disk_bus = "ide"domain.nic_adapter_count = 2endnode.vm.network :private_network,# sw02-eth1 <--> sw01-eth1:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.146.53.2",:libvirt__tunnel_local_port => 10001,:libvirt__tunnel_ip => "127.146.53.1",:libvirt__tunnel_port => 10001,:libvirt__iface_name => "sw02-eth1-#{domain_uuid}",auto_config: falsenode.vm.network :private_network,# sw02-eth2 <--> sw01-eth2:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.146.53.2",:libvirt__tunnel_local_port => 10002,:libvirt__tunnel_ip => "127.146.53.1",:libvirt__tunnel_port => 10002,:libvirt__iface_name => "sw02-eth2-#{domain_uuid}",auto_config: falseendend```### Defaults Per-Guest TypeIt is possible to define default values per guest group type. Grifter willlook for a file named `guest-defaults.yml` in the following locations fromthe least to most preferred:- `/opt/grifter/`- `~/.grifter/`- `./````yamlarista/veos:vagrant_box:version: "4.20.1F"guest_type: "tinycore"ssh:insert_key: Falsesynced_folder:enabled: Falseprovider_config:nic_adapter_count: 24cpus: 2memory: 2048disk_bus: "ide"juniper/vsrx-packetmode:vagrant_box:version: "18.3R1-S1.4"provider: "libvirt"guest_type: "tinycore"ssh:insert_key: Falsesynced_folder:enabled: Falseprovider_config:nic_adapter_count: 2disk_bus: "ide"cpus: 2memory: 4096```Group variables can be over-written by variables at the guest variable level.The values of the group and guest variables will be merged prior to buildinga `Vagrantfile` with the guest variables taking precedence over the groupvariables.This means you can have a much more succinct guests file by reducinga lot of duplication. Here is an example of a simplified guest file. Thevalues from the `arista/veos` guest type in the `guest-defaults.yml` filewill be used to fill in the parameters for the guests.```yamlsw01:vagrant_box:name: "arista/veos"provider_config:nic_adapter_count: 2data_interfaces:- local_port: 1remote_guest: "sw02"remote_port: 1- local_port: 2remote_guest: "sw02"remote_port: 2sw02:vagrant_box:name: "arista/veos"provider_config:nic_adapter_count: 2data_interfaces:- local_port: 1remote_guest: "sw01"remote_port: 1- local_port: 2remote_guest: "sw01"remote_port: 2```The generated `Vagrantfile` below is the same as the one above, but with amuch cleaner guest definition file.```ruby# -*- mode: ruby -*-# vi: set ft=ruby :def get_mac(oui="28:b7:ad")"Generate a MAC address"nic = (1..3).map{"%0.2x"%rand(256)}.join(":")return "#{oui}:#{nic}"endcwd = Dir.pwd.split("/").lastusername = ENV['USER']domain_prefix = "#{username}_#{cwd}"domain_uuid = "d35fb1b6-ecdc-5412-be22-185446af92d6"Vagrant.require_version ">= 2.1.0"Vagrant.configure("2") do |config|config.vm.define "sw01" do |node|guest_name = "sw01"node.vm.box = "arista/veos"node.vm.box_version = "4.20.1F"node.vm.guest = :tinycorenode.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: truenode.ssh.insert_key = falsenode.vm.provider :libvirt do |domain|domain.default_prefix = "#{domain_prefix}"domain.cpus = 2domain.memory = 2048domain.disk_bus = "ide"domain.nic_adapter_count = 2endnode.vm.network :private_network,# sw01-eth1 <--> sw02-eth1:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.127.145.1",:libvirt__tunnel_local_port => 10001,:libvirt__tunnel_ip => "127.127.145.2",:libvirt__tunnel_port => 10001,:libvirt__iface_name => "sw01-eth1-#{domain_uuid}",auto_config: falsenode.vm.network :private_network,# sw01-eth2 <--> sw02-eth2:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.127.145.1",:libvirt__tunnel_local_port => 10002,:libvirt__tunnel_ip => "127.127.145.2",:libvirt__tunnel_port => 10002,:libvirt__iface_name => "sw01-eth2-#{domain_uuid}",auto_config: falseendconfig.vm.define "sw02" do |node|guest_name = "sw02"node.vm.box = "arista/veos"node.vm.box_version = "4.20.1F"node.vm.guest = :tinycorenode.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: truenode.ssh.insert_key = falsenode.vm.provider :libvirt do |domain|domain.default_prefix = "#{domain_prefix}"domain.cpus = 2domain.memory = 2048domain.storage_pool_name = "disk1"domain.disk_bus = "ide"domain.nic_adapter_count = 2endnode.vm.network :private_network,# sw02-eth1 <--> sw01-eth1:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.127.145.2",:libvirt__tunnel_local_port => 10001,:libvirt__tunnel_ip => "127.127.145.1",:libvirt__tunnel_port => 10001,:libvirt__iface_name => "sw02-eth1-#{domain_uuid}",auto_config: falsenode.vm.network :private_network,# sw02-eth2 <--> sw01-eth2:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.127.145.2",:libvirt__tunnel_local_port => 10002,:libvirt__tunnel_ip => "127.127.145.1",:libvirt__tunnel_port => 10002,:libvirt__iface_name => "sw02-eth2-#{domain_uuid}",auto_config: falseendend```## Example FilesExamples of the `config.yml`, `guests-defaults.yml` and `guests.yml` filescan be found [here](grifter/examples)## InterfacesThere are 3 types of interfaces that can be defined.- internal_interfaces- data_interfaces- reserved_interfaces#### Internal InterfacesConfig location: `guests.yml`Used for an inter-vm communication channel for multi-vm boxes.Known examples are the vMX and vQFX.#### data_interfacesConfig location: `guests.yml`Revenue ports that are used to pass data traffic.#### reserved_interfacesConfig location: `config.yml`Interfaces that need to be defined because 'reasons' but cannot beused. The only known example is the `juniper/vqfx-re`. The number ofreserved_interfaces is defined per-box type in the `config.yml` file.Grifter builds out the interface definitions automatically as ablackhole interfaces.#### Blackhole InterfacesInterfaces defined in the Vagratfile relate to interfaceson the guest vm on a first to last basis. This can be undesirable whentrying to accurately simulate a production environment when devicescan have 48+ ports.Grifter will automatically create `blackhole interfaces` to fill outundefined `data_interfaces` ports up to the box types`max_data_interfaces` parameter in the `config.yml` file.#### Vagrantfile Interface OrderInterfaces are added to the Vagrantfile in the following order.- internal_interfaces- reserved_interfaces- data_interfacesInterfaces are configured using the udp tunneling type. Thiswill create a 'pseudo' layer 1 connection between VM ports.##### Example interface definition```yamldata_interfaces:- local_port: 1remote_guest: "sw02"remote_port: 1```##### Rendered Vagrantfile interface```rubynode.vm.network :private_network,# sw01-eth1 <--> sw02-eth1:mac => "#{get_mac()}",:libvirt__tunnel_type => "udp",:libvirt__tunnel_local_ip => "127.255.255.1",:libvirt__tunnel_local_port => 10001,:libvirt__tunnel_ip => "127.255.255.2",:libvirt__tunnel_port => 10001,:libvirt__iface_name => "sw01-eth1-#{domain_uuid}",auto_config: false```#### NIC Adapter CountConfig location: `guests.yml`Defines the total number of `data_interfaces` to create on the VM.Any undefined `data_interfaces` will be added as a blackhole interface.The total is calculated against the sum of the `internal_interfaces`, `reserved_interfaces` and `data_interfaces` parameters after blackholeinterfaces have been added automatically by the template system.