npm package: @itentialopensource/automation-gateway-copy-files-from-git
Description: Copy Files from Gitlab or Github to IAG
Author: Itential
Latest version: 1.0.5

<!-- This is a comment in md (Markdown) format, it will not be visible to the end user -->

<!-- Update the below line with your Pre-Built name -->
# Automation Gateway Copy Files From Git

<!-- Leave TOC intact unless you've added or removed headers -->
## Table of Contents

- [Automation Gateway Copy Files From Git](#Automation-Gateway-Copy-Files-From-Git)
  - [Table of Contents](#table-of-contents)
  - [Overview](#overview)
  - [Supported IAP Versions](#supported-iap-versions)
  - [Getting Started](#getting-started)
    - [Prerequisites](#prerequisites)
    - [Capabilities](#capabilities)
    - [How to Install](#how-to-install)
    - [Testing](#testing)
  - [Using this Pre-Built](#using-this-prebuilt)
    - [Input Schema](#input-schema)
    - [Output Schema](#output-schema)
  - [Additional Information](#additonal-information)
## Overview

<table><tr><td>
<img  src="./images/automation-studio.png"  alt="Automation Studio"  width="600px">
</td></tr></table>

This Pre-Built Automation enables users to copy a file from a Gitlab or GitHub repository to an IAG server.

_Estimated Run Time_: 0.5 min.

## Supported IAP Versions
Itential pre-builts are built and tested on particular versions of IAP. Please make sure to run on the following version:

* Itential Automation Platform
  * `^2023.1.X`


## Getting Started

These instructions will help you get a copy of the pre-built in your IAP instance for testing in your environment. Reading this section is also helpful for deployments as it provides you with pertinent information on prerequisites and capabilities.

### Prerequisites
This Pre-Built requires the following:

* The Automation Gateway Adapter must be installed
* Either the Gitlab adapter or the Github adapter must be installed

You must create an authentication token in Gitlab to put into the Gitlab adapter configuration or create an authentication token in Github to put into the Github adapter configuration.

Sample Gitlab adapter configuration:
```json
{
    "name": "gitlab",
    "model": "@itentialopensource/adapter-gitlab",
    "type": "Adapter",
    "properties": {
        "id": "gitlab",
        "type": "Gitlab",
        "properties": {
            "host": "gitlab.com",
            "port": 443,
            "base_path": "/api",
            "version": "v4",
            "cache_location": "local",
            "save_metric": true,
            "stub": false,
            "protocol": "https",
            "authentication": {
                "auth_method": "static_token",
                "username": "",
                "password": "",
                "auth_field": "header.headers.Private-Token",
                "auth_field_format": "{token}",
                "token": "************************",
                "invalid_token_error": 401,
                "token_timeout": 0,
                "token_cache": "local"
            },
            "healthcheck": {
                "type": "startup",
                "frequency": 300000
            },
            "request": {
                "number_retries": 3,
                "limit_retry_error": 401,
                "failover_codes": [
                    404,
                    405
                ],
                "attempt_timeout": 5000,
                "global_request": {
                    "payload": {},
                    "uriOptions": {},
                    "addlHeaders": {},
                    "authData": {}
                },
                "healthcheck_on_timeout": false,
                "return_raw": false,
                "archiving": false
            },
            "ssl": {
                "ecdhCurve": "",
                "enabled": false,
                "accept_invalid_cert": false,
                "ca_file": "",
                "secure_protocol": "",
                "ciphers": ""
            },
            "throttle": {
                "throttle_enabled": false,
                "number_pronghorns": 1,
                "sync_async": "sync",
                "max_in_queue": 1000,
                "concurrent_max": 1,
                "expire_timeout": 0,
                "avg_runtime": 200
            },
            "proxy": {
                "enabled": false,
                "host": "localhost",
                "port": 9999,
                "protocol": "http"
            },
            "mongo": {
                "host": "",
                "port": 0,
                "database": "",
                "username": "",
                "password": ""
            }
        },
        "brokers": [],
        "groups": []
    },
    "isEncrypted": false,
    "loggerProps": {
        "description": "Logging",
        "log_max_files": 100,
        "log_max_file_size": 1048576,
        "log_level": "info",
        "log_directory": "./logs",
        "log_filename": "pronghorn.log",
        "console_level": "info"    
        }
}
```

Sample GitHub adapter configuration:
```json
{
  "name": "github",
  "model": "@itentialopensource/adapter-github",
  "type": "Adapter",
  "properties": {
    "id": "github",
    "type": "GitHub",
    "brokers": [],
    "groups": [],
    "properties": {
      "host": "api.github.com",
      "port": 443,
      "base_path": "",
      "version": "",
      "cache_location": "none",
      "encode_pathvars": true,
      "encode_queryvars": true,
      "save_metric": false,
      "stub": false,
      "protocol": "https",
      "authentication": {
        "auth_method": "static_token",
        "username": "",
        "password": "",
        "token": "***token***",
        "token_timeout": 1800000,
        "token_cache": "local",
        "invalid_token_error": 401,
        "auth_field": "header.headers.Authorization",
        "auth_field_format": "Bearer {token}",
        "auth_logging": false,
        "client_id": "",
        "client_secret": "",
        "grant_type": ""
      },
      "healthcheck": {
        "type": "startup",
        "frequency": 300000,
        "query_object": {}
      },
      "request": {
        "number_redirects": 0,
        "number_retries": 3,
        "limit_retry_error": 401,
        "failover_codes": [],
        "attempt_timeout": 5000,
        "global_request": {
          "payload": {},
          "uriOptions": {},
          "addlHeaders": {
            "user-agent": "node.js"
          },
          "authData": {}
        },
        "healthcheck_on_timeout": true,
        "return_raw": false,
        "archiving": false,
        "return_request": false
      }
    }
  },
  "isEncrypted": true,
  "loggerProps": {
    "description": "Logging",
    "log_max_files": 10,
    "log_max_file_size": 10485760,
    "log_level": "debug",
    "log_directory": "/opt/itential/logs",
    "log_filename": "itential.log",
    "log_timezone_offset": 0,
    "console_level": "debug",
    "syslog": {
      "level": "warning",
      "host": "127.0.0.1",
      "port": 514,
      "protocol": "udp4",
      "facility": "local0",
      "type": "BSD",
      "path": "",
      "pid": "process.pid",
      "localhost": "",
      "app_name": "",
      "eol": ""
    }
  },
  "virtual": false
}
```

Copy this helper script to the destination IAG server and customize it for the environment. See overview of the [Script Execution Engine](https://docs.itential.com/docs/script-execution-engine) for more information on adding the script to IAG.  Name it "import_playbook.pl" and make it executable and make sure it is in a path that is configured as a script path in properties.yml for the IAG server. By default, this pre-built imports the file to ansible playbooks, but you can change the basePath to fit your specific needs.

```
#!/usr/bin/perl

use MIME::Base64;

# NOTE: adjust this to a path in the configured playbook paths in properties.yml
my $basePath = "/usr/share/ansible/playbooks/";
my $filename = $ARGV[0];
my $content = $ARGV[1];

my $fullfile = $basePath . $filename;

$content =~ s/(\^&)/'/g;

my $decoded = decode_base64(decode_base64($content));

print "Writing playbook to $fullfile\n";

open(FH, '>', $fullfile) or die	$!;
print FH $content;
close(FH);

print "Done.";

```

### Capabilities

The main benefits and features of the Pre-Built are outlined below.

* This will copy a file from a Git repository (Gitlab or Github) to an Itential Automation Gateway server.

## How to Install

To install the Pre-Built:

* Verify you are running a supported version of the Itential Automation Platform (IAP) as listed above in the [Requirements](#requirements) section in order to install the Pre-Built. 
* The Pre-Built can be installed from within App-Admin_Essential. Simply search for the name of your desired Pre-Built and click the install button (as shown below).

<table><tr><td>
<img  src="./images/install.png"  alt="install"  width="600px">
</td></tr></table>

* For more information, follow the instructions on the Itential Documentation site for [importing a pre-built](https://docs.itential.com/docs/importing-a-prebuilt-2).

### Testing

While Itential tests this pre-built and its capabilities, it is often the case the customer environments offer their own unique circumstances. Therefore, it is our recommendation that you deploy this pre-built into a development/testing environment in which you can test the pre-built.

## Using this Pre-Built

This pre-built should be run in an [automation](https://docs.itential.com/docs/operations-manager-ui-1).

**Note**: The entry automation to this pre-built is called `Automation Gateway Copy Files From Git`.

Use the following to run the Pre-Built:

Run the automation from Operations Manager, fill in the form values and select start.

Hover over the blue tool tips next to each form field for more details.

<table><tr><td>
<img  src="./images/automation-catalog-example-gitlab.png"  alt="Automation Catalog example"  width="400px">
</td></tr></table>

<table><tr><td>
<img  src="./images/automation-catalog-example-github.png"  alt="Automation Catalog example"  width="400px">
</td></tr></table>

When running this pre-built, it depends on being provided proper input so that git can retrieve the files and IAG can accept the files. The input to and possible outputs from this pre-built are described here.

### Input Schema

Example input to the github trigger form:

```json
formData: {
  "gitSettings": {
    "adapterName": "github",
    "filePath": "fileName.yml",
    "ref": "main"
  },
  "automationGatewaySettings": {
    "iagAdapter": "IAG-2021.2",
    "importScriptName": "import_playbook.pl",
    "destinationFilename": "exampleName.yml"
  },
  "repoName": "repoName",
  "repoOwner": "repoOwner"
}
```
The following table details the property keys of the input object.
| key                                           | type    | required | description                                                         |
|-----------------------------------------------|---------|----------|---------------------------------------------------------------------|
| gitSettings                                   | object  | yes      | Wrapper object for all git settings                                 |
| gitSettings.adapterName                       | string  | yes      | The name of the adapter configured in Admin Essentials              |
| gitSettings.filePath                          | string  | yes      | The file path found within your Version Control System              |
| gitSettings.ref                               | string  | yes      | The name of the branch found within git                             |
| automationGatewaySettings                     | object  | yes      | Wrapper object for all Automation Gateway settings                  |
| automationGatewaySettings.iagAdapter          | string  | yes      | The name of the adapter configured in Admin Essentials              |
| automationGatewaySettings.importScriptName    | string  | yes      | The name of the scrip used to import file (import_playbook.pl)      |
| automationGatewaySettings.destinationFilename | string  | yes      | The name of the file that will be created/updated in IAG            |
| repoName                                      | string  | yes      | The name of the repo found within github                            |
| repoOwner                                     | string  | yes      | The name of the owner/organization of the repo found within github  |


Example input to the gitlab trigger form:

```json
formData: {
  "gitSettings": {
    "adapterName": "gitlab",
    "filePath": "fileName.yml",
    "ref": "master"
  },
  "automationGatewaySettings": {
    "iagAdapter": "IAG-2021.2",
    "importScriptName": "import_playbook.pl",
    "destinationFilename": "exampleName.yml"
  },
  "repoId": "repoId",
}
```

The following table details the changed property keys of the input object from github to gitlab.
| key              | type    | required | description                                            |
|------------------|---------|----------|--------------------------------------------------------|
| repoId           | string  | yes      | The repository ID of the repo containing the file      |

<table><tr><td>
<img  src="./images/gitlab-project-id.png"  alt="Gitlab Project ID location"  width="600px">
</td></tr></table>

### Output Schema

The `ReturnStatus` job variable returned from the run pre-built workflow `Automation Gateway Copy File From Git` reports the success or failure of importing the script into IAG. 

```json
ReturnStatus {
  "status": "SUCCESS",
  "message" : "Successfully grabbed the file from git and imported it into IAG",
  "response" : {
    "getContents" : {},
    "importScript" : {},
    "refreshScripts" : {}
  },
  "errors" : []
}
```

The following table details the property keys of the outputReturnStatus object.
| key                                      | type    | description                                                         |
|------------------------------------------|---------|---------------------------------------------------------------------|
| status                                   | enum    | The status of the whole workflow (SUCCESS or FAILED)                |
| message                                  | string  | A message saying which tasks completed and their status             |
| response                                 | object  | An object consisting of objects which correspond to adapter calls   |
| errors                                   | array   | An array of errors that occured during the prebuilt run             |

## Additional Information

Please use your Itential Customer Success account if you need support when using this Pre-Built Transformation.