Package: autoshell
AutoshellA simple, fully programmable, shell-based network automation utilityQuick Install:pip install autoshellGet started fast with theAutoshell TutorialVERSIONThe version of Autoshell documented here is:0.0.45TABLE OF CONTENTSWhat is AutoshellRequirementsInstallationGetting StartedContributingAutoshell TutorialWHAT IS AUTOSHELLAutoshell is a Command-Line Utility and Python Library built to make quick automation tasks of a network easy.Autoshell is a "shell" application in that it handles a part of the automation process, mainly parsing user arguments and connecting to hosts. It then depends on bundled and/or user-written modules to perform desired tasks on the connected hosts. Since Autoshell does the heavy lifting (connecting to hosts, importing modules, sorting through credentials, providing module libraries, etc) it allows the modules to be very short and simple while making their effects quite powerful.INSTALLATIONThe installation process is also covered in theAutoshell TutorialPrepare Your OSWindowsIf you have not yet installed Python on your Windows OS, then download and install the latest Python2 or Python3 fromPython Downloads PageMake sure to check the box during installation which adds Python to PATH. Labeled something likeAdd Python 3.7 to PATHOnce Python is installed, you should be able to open a command window, typepython, hit ENTER, and see a Python prompt opened. Typequit()to exit it. You should also be able to run the commandpipand see its options. If both of these work, then move on to install Autoshell.If this does not work, you will need to add the Python installation directory path to the Windows PATH variableThe easiest way to do this is to find the new shortcut for Python in your start menu, right-click on the shortcut, and find the folder path for thepython.exefileFor Python2, this will likely be something likeC:\Python27For Python3, this will likely be something likeC:\Users\<USERNAME>\AppData\Local\Programs\Python\Python37Open your Advanced System Settings window, navigate to the "Advanced" tab, and click the "Environment Variables" buttonCreate a new system variable:Variable name:PYTHON_HOMEVariable value: <your_python_installation_directory>Now modify the PATH system variable by appending the text;%PYTHON_HOME%\;%PYTHON_HOME%;%PYTHON_HOME%\Scripts\to the end of it.Close out your windows, open a command window and make sure you can run the commandspythonandpipMacOSMacOS often comes with a native version of Python, but we likely want to upgrade that and install PIP. The best way to do this is with a MacOS Linux-like package manager calledHomebrew. You can visit the below pages to walk you through installing Homebrew and an updated Python interpreter along with itOpen Terminal and run:xcode-select --install. This will open a window. Click'Get Xcode'and install it from the app store.Install Homebrew. Run:/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"Install latest Python2:brew install python@2Install latest Python3:brew install pythonLink your default Python path to the new installbrew link --overwrite pythonClose Terminal and reopen it. You should see Python 2.7.15 when you runpython -VOnce Python is installed, you should be able to open Terminal, typepython, hit ENTER, and see a Python prompt opened. Typequit()to exit it. You should also be able to run the commandpipand see its options. If both of these work, then move on toInstall AutoshellAdditional resources onInstalling Python 2 on Mac OS XAdditional resources onInstalling Python 3 on Mac OS XLinuxInstall required OS packagesRaspberry Pimay need Python and PIPsudo apt install -y python-pipas well assudo apt-get install libffi-devDebian (Ubuntu)distributions may need Python and PIPInstall Python and PIP:sudo apt install -y python-pipRHEL (CentOS)distributions usually need PIPInstall the EPEL package:sudo yum install -y epel-releaseInstall PIP:sudo yum install -y python-pipInstall Autoshell using PIP (Recommended)Install Autoshell using pip by running the commandpip install autoshellOnce the install completes, you should be able to run the commandautoshell -hand see the help menu. Autoshell is now ready to use.Install Autoshell from source (Advanced)Retrieve the source code repository using one of the two below methodsMethod #1: Install a Git client (process differs depending on OS) and clone the Autoshell repository using Gitgit clone https://github.com/packetsar/autoshell.gitChange to the branch you want to install usinggit checkout <branch_name>Method #2: Download and extract the repository files from theGithub RepoMake sure to download the branch you want to installMove into the autoshell project directorycd autoshellRun the setup.py file to build the package into the ./build/ directorypython setup.py buildUse PIP to install the packagepip install .Once the install completes, you should be able to run the commandautoshell -hand see the help menu. Autoshell is now ready to use.GETTING STARTEDBelow are some nuts and bolts to help you understand how to use Autoshell. If you want a quick 'how-to', then check out theAutoshell Tutorial.Once you have installed Autoshell, you can see the command guide withautoshell -h.Since modules are able to add their own arguments into the argument parser when they are imported, you can see a module's help by importing it (ie:-m crawl) and then adding the-hargument. You can see this by tryingautoshell -m crawl -h, there you will see a section of argument descriptions for the Crawl module.The simplest way to connect to a host is to point Autoshell directly at it (autoshell 192.168.1.1). If you do not provide credentials, it will prompt you for them. It is easy to add them inline with-c username:password. You are able to add as many hosts and credentials as you want and they are processed in the order you provide them.CredentialsCredentials can be provided to Autoshell in a few different ways:At the command line as a string.The default full format for credentials is-c <username>:<password>:<secret>@<host_type>. This default format has optional values included. A credential string can have just one value (ie:-c admin) and Autoshell will use thatadminvalue for the username, password, and secret; it will leave the host_type blank unless provided. You can instead provide-c admin:passwordand Autoshell will use the provided password for both the password and secret values. More examples are provided in the command help guide at the command line.As a structured JSON or YAML file.You can use theexamples/example_structured_credentials_file.jsonandexamples/example_structured_credentials_file.ymlfiles as examples, then reference them from the command-line like-c example_structured_credentials_file.json. You can reference as many credential files as you want.As an unstructured file.Seeexamples/example_unstructured_credentials.txtfor an example. In the unstructured format, each line in the file will contain a credential string in the standard command-line format. You can then reference the file from the command-line like-c example_unstructured_credentials.txt. You can reference as many credential files as you want.AddressesAddresses are provided to Autoshell at the command-line using positional arguments (without a prepended-xor--xxxxtag). You are able to provide as many addresses as you want here and they will processed in the order you give them. Below are some examples of how you can provide addresses.As a simple string at the command-line.Addresses use the format of<address>@<host_type>where the host_type value is optional. A simple example is just using the IP address or dns-name like192.168.1.1or with a host_type like192.168.1.1@cisco_ios.As a structured JSON or YAML file.You can use theexamples/example_structured_addresses_file.jsonandexamples/example_structured_addresses_file.ymlfiles as examples, then reference them from the command-line likeexample_structured_addresses_file.json. You can reference as many address files as you want.As an unstructured file.Seeexamples/example_unstructured_addresses_file.txtfor an example. In the unstructured format, each line in the file will contain an address string in the standard command-line format. You can then reference the file from the command-line likeexample_unstructured_addresses_file.txt. You can reference as many address files as you want.Using Config FilesSince you may often need to define many command-line arguments, it is often easier to provide command-line arguments using a config file. You can use a structured JSON or YAML config file (examples can be found atexamples/example_config_file.jsonandexamples/example_config_file.yml) to define your arguments and reference the config file using-for--config_fileat the command-line (like-f example_config_file.json). The example config files can also be found in theexamples project folder.You can define known or unknown arguments in the config file and both will be passed into the program and into any modules you import. You can also define multiple config files which will either overwrite or append to each others settings (depending on if the argument can have multiple values).Arguments can still be defined at the command-line even when using config files. If the argument is a single value (like the debugging level) then the command-line value will overwrite any config file values. If the argument allows multiple entries (like credentials) then the command-line values will prepend any config-file values.TIP:If you want to take your working autoshell command and convert the inline options to a config file, you can run your command with level 4 (-dddd) or level 5 (-ddddd) debugging enabled and look for the###### INPUT ARGUMENTS #######section. You can copy that JSON data into a file and use that file as your config file. Remember to readjust your debugging level in the config file as it will show as 4 or 5.MODULESThe power and flexibility of Autoshell is realized by the use of modules. Modules are python files or libraries which are used to manipulate the hosts connected to by Autoshell. Autoshell includes some modules bundled in the installation. These bundled modules can be found in themodules folderat pathautoshell/modules/. Modules can (and often should) also be user-written using Python to do whatever the user wants. Modules can be imported in two different ways:Using a command-line argument like-m neighbors.Using a config-file. See theUsing Config Filessection for more info.Modules may [optionally] introduce their own arguments into the argument parser when they are imported. These module-specific arguments are recognized by the argparser and will have a help-menu displayed when using the-hargument. You can see an example of this by issuingautoshell -m crawl -h.Bundled ModulesBundled modules are included in the default installation of Autoshell and can be imported for use immediately after installation. The bundled module files can be found in themodules folderat pathautoshell/modules/. You can use these bundled modules as a reference when writing your own module as they too must adhere to theAutoshell Module API.cmd:cmdis a bundled module and can be imported with-m cmd. You can also use-m cmd -hto see all options and switches related to it. When no options are used, thecmdmodule will prompt the user for a command to execute on all connected hosts. It will then execute the command and return the output.The-Coption can be used to run one or more commands without user interaction.You can prepend the termconfig:to a command to have it run in config mode on the device. Example:config: router ospf 1The-Ooption can be used to write host output (from all hosts) to a filepath.The-Poption can be used to write each hosts output to a different file. In this option you can use the Jinja2 language to templatize the names of the files/folders where the output is written. For example:-P /root/{{hostname}}.txtwill write the output for each host into a file named from the hosts hostname. All attributes from thehost.infodictionary are available here as well as thenowfunction from thedatetimelibrary. This allows you to structure file/folder names with a timestamp like-P /root/{{now.strftime('%Y-%m-%d_%H.%M.%S')}}.txt.User-Written ModulesIf you are not able to accomplish the automation tasks you want using the bundled modules (which is common), then you can write your own module to accomplish your task. Autoshell makes this quite easy since much of the difficult work will have been done by the time the code in your module is called. User-written modules can be imported using its file path (ie:-m mymods/mymodule.py) or you can reference the file name in a config-file.Autoshell Module APIAutoshell will attempt to call any imported module at three (3) points during execution.add_parser_options(parser)(optional): The first call against your module will be done against youradd_parser_optionsfunction. This is anoptionalcall and will only be done if theadd_parser_optionsfunction exists in your code. This call will hand program control over to your module and allow you to populate custom arguments into the Autoshell argument parser. This call is made before the arguments are parsed and before we connect to hosts (this is how you can see a module's help info when using the-hswitch).load(ball)(optional): The second call against your module will be done against yourloadfunction. This is anoptionalcall and will only be done if theloadfunction exists in your code. This call will hand program control over to your module to allow you to perform input checks on user-provided arguments before Autoshell starts connecting to hosts. Theloadcall exists to permit your module to perform input checks and throw warnings/errors early on in the program instead of waiting until all hosts have been connected and all other modules have run. You can also use theloadfunction to do some pre-processing of user inputs and store them in a namespace object to be used later when the module is run.run(ball)(required): The third and final call against your module will be done against yourrunfunction. This is arequiredcall and your module must include arunfunction in order to operate. Therunfunction is called after all hosts have been processed/connected and after all previously imported modules (since modules are processed in the order in which they are referenced from the CLI) have completed. Therunfunction is where you want to perform your custom tasks on connected hosts.Threading: Autoshell includes a queueing/threading library calledautoqueuewhich can be very useful when wanting to perform your custom tasks on many hosts in parallel. Autoqueue can be imported and used in your module by importing the autoshell library (import autoshell) and instantiating an autoqueue object (autoshell.common.autoqueue.autoqueue(<arguments>)). You can reference the example modules for examples on how to use autoqueue.Once the module'srunfunction returns control of the main thread back to the Autoshell program, Autoshell will call therunfunction of the next module if there is a module in order after this one. Once all modules complete and the last module returns control, Autoshell will perform a final processing of all active hosts by gracefully disconnecting from them and quitting the program.The Autoqueue LibraryAutoqueue is an Autoshell library which makes threading and queueing much easier than using the threading and queue libraries separately. Autoqueue also provides functions for blocking the main thread until the queue is empty and the threads are idle.To use Autoqueue in a module. You can import autoshell as a library (import autoshell) and referece autoqueue atqueue = autoshell.common.autoqueue.autoqueue(thread_count, worker_func, worker_args).Autoqueue Argumentsthread_count: A integer defining how many worker threads you want to use. All threads are supervised and managed by Autoqueue and you don't need to worry about dealing with them.worker_func: The worker function which will be run inside each thread. Since each thread is monitored and maintained by autoqueue's supervisor, any returned object will be discarded. If you need to return any object/data out of your worker function, you will have pass in and write them to a namespace object or something similar.worker_args: A tuple containing any arguments you want to have passed into your worker function by the thread supervisor in addition to the default arguments which get passed in (parent_objectandqueue_item)Once the queue has been instantiated, it will begin waiting for items to be put into the queue. When items begin being.put(item)into the queue, the thread supervisors will begin calling the worker functions, passing the queue items into them.EXAMPLE FILESBelow are some example files you can use for reference. You can find these files in theexamples project folder.examples/example_structured_credentials_file.json[
  {
    "username": "file_admin1",
    "password": "file_password1"
  },
  {
    "username": "file_admin2",
    "password": "file_password2"
  }
]examples/example_structured_credentials_file.yml- username: file_admin3
  password: file_password3
- username: file_admin4
  password: file_password4examples/example_unstructured_credentials.txtunst_admin1:unst_password1
unst_admin2:unst_password2@cisco_iosexamples/example_structured_addresses_file.json[
  {
    "address": "192.168.1.1",
    "type": "cisco_ios"
  },
  {
    "address": "192.168.1.2"
  }
]examples/example_structured_addresses_file.yml- address: 192.168.1.1
  type: cisco_ios
- address: 192.168.1.2examples/example_unstructured_addresses_file.txt192.168.1.10
192.168.1.11@cisco_iosexamples/example_config_file.json{
  "modules": [
    "neighbors",
    "cli"
  ],
  "debug": 5,
  "addresses": [
    "192.168.1.1",
    "192.168.1.2"
  ],
  "credentials": [
    "cfg_file_admin1:cfg_file_password1",
    "cfg_file_admin2:cfg_file_password2@cisco_ios",
    "examples/example_structured_credentials_file.json"
  ],
  "dump_hostinfo": false,
  "logfiles": [
    "autoshell_log1.log",
    "autoshell_log2.log"
  ]
}examples/example_config_file.ymlmodules:
  - neighbors
  - cli
debug: 5
addresses:
  - 192.168.1.1
  - 192.168.1.2
credentials:
  - admin1:password1
  - admin2:password2@cisco_ios
  - examples/example_structured_credentials_file.json
dump_hostinfo: false
logfiles:
  - autoshell_log1.log
  - autoshell_log2.log