Package: pull-scp
Table of ContentsAbstractDescriptionWarningCreditUsageArgumentsInstallFrom dockerFrom PyPIRunDevelopmentExamplesUsing login credentials stored in the container’s.envfile:Using login credentials from the CLI:AbstractThis plugin application is used to recursively copy data from a remote host into an analysis root node.Descriptionpull_scpis aChRIS fs-typeapplication that produces data for an analysis tree by copying data off a remote host using (recursive)scp.Of course this assumes that the user executing this plugin has the correct login credentials to access the resource. Credentials are defined in either:a hard-coded .env file in the repo/containersetting appropriate environment variablesusing plugin CLI argumentsOther than login credentials, this plugin also needs afilepathin the remote user space. All files and directories rooted in this filefilepathare copied into this plugin’soutputdir.WarningThis plugin/app is not considered or purported to be secure! One deployment vector has the login credentials contained within an.envfile in cleartext and copied into the container. Deploying such a container will expose login credentials! For better security, supply login credentials from the CLI or from within environment variables:exportENVIRONMENT=developmentexportSSH_REMOTE_HOST=1.1.1.1exportSSH_USERNAME=yournameexportSSH_PASSWORD=yourpasswordexportSSH_KEY_FILEPATH=/usr/local/src/key.pubexportSCP_DESTINATION_FOLDER=/tmpCreditMost of the innards of this plugin are lightly adapted from the most excellent paramiko tutorial of Todd Birchard:https://hackersandslackers.com/automate-ssh-scp-python-paramikoUsagedocker run --rm fnndsc/pl-pull_scp pull_scp                     \
    [-h] [--help]                                               \
    [--json]                                                    \
    [--man]                                                     \
    [--meta]                                                    \
    [--savejson <DIR>]                                          \
    [-v <level>] [--verbosity <level>]                          \
    [--version]                                                 \
    [--username <username>]                                     \
    [--password <password>]                                     \
    [--host <hostname>]                                         \
    [--sshPubKeyFile <pubKeyFile>]                              \
    --filepath <filepath>                                       \
    <outputDir>Arguments--filepath <filepath>
The path in the <hostname>'s filesystem to pull. This is technically
the only required argument of this plugin. All user specific
credentials are assumed to be container in the .env file or set in
environment variables. Note of course that all the credentials can
be overriden with CLI flags.

[--username <username>]
The username in the remote host.

[--password <password>]
The <username>'s password to connect to the remote host.

[--sshPubKeyFile <pubKeyFile>]
The ssh public key file to use in this session.

[--host <hostname>]
The hostname to access.

[-h] [--help]
If specified, show help message and exit.

[--json]
If specified, show json representation of app and exit.

[--man]
If specified, print (this) man page and exit.

[--meta]
If specified, print plugin meta data and exit.

[--savejson <DIR>]
If specified, save json representation file to DIR and exit.

[-v <level>] [--verbosity <level>]
Verbosity level for app. Not used currently.

[--version]
If specified, print version number and exit.InstallInstallation is either viadocker(recommended) or directly fromPyPI(less recommended).From dockerdockerpullfnndsc/pl-pull_scpGetting inline help is:dockerrun--rmfnndsc/pl-pull_scppull_scp--manFrom PyPIpipinstallpull_scpFor this use case, copy your public key to/tmp/key.puband create an environment file/tmp/.envthat either contains your login data/credentials or contains “dummy” data. The.env_must_ exist for the app to work – regardless of the validity of its data.RunSincedockeris the recommended usagage deployment, the following instructions aredockerbased.Being an _FS_ plugin, you should specify an output directory using the-vflag todocker run.dockerrun--rm-u$(id-u)\-v$(pwd)/out:/outgoing\fnndsc/pl-pull_scppull_scp\--usernamejohnnyapple\--password'mysecret'\--hostcomputer.org\--sshPubKeyFile~/.ssh/rsa_pub.key\--filepath/home/johnnyapple/data\/outgoingDevelopmentBuild the Docker container:dockerbuild-tlocal/pl-pull_scp.Run unit tests:dockerrun--rmlocal/pl-pull_scpnosetestsExamplesUsing login credentials stored in the container’s.envfile:All relevant login credentials are stored in/tmp/.envin the container. Note this method embeds the.envfile in the container where it can be potentially expose credentials!mkdir out && chmod 777 out
docker run --rm -u $(id -u) --name=pl-pull_scp              \
            -v $PWD/out:/outgoing                           \
            -it                                             \
            local/pl-pull_scp pull_scp -v 1                 \
            --filepath /home/rudolphpienaar/Desktop         \
            outgoing

10-15-2021 16:25:38 | INFO: /tmp/key.pub uploaded to 192.168.1.216

             _ _
            | | |
 _ __  _   _| | |  ___  ___ _ __
| '_ \| | | | | | / __|/ __| '_ \
| |_) | |_| | | | \__ \ (__| |_) |
| .__/ \__,_|_|_| |___/\___| .__/
| |           ______       | |
|_|          |______|      |_|

Version: X.Y.Z
10-15-2021 16:25:38 | INFO: (remote): du -ksh /home/rudolphpienaar/Desktop: 142M    /home/  rudolphpienaar/Desktop
10-15-2021 16:25:39 | INFO: Pulling rudolphpienaar@192.168.1.216:/home/rudolphpienaar/Desktop...
10-15-2021 16:25:53 | INFO: Remote contents pulled to /outgoing
10-15-2021 16:25:53 | INFO: (NOTE: if running in a container, and doing a volume mapping,
                            the destination dir name might not match the host dirname!)
10-15-2021 16:25:53 | INFO: (local): du -ksh /outgoing: 142M        /outgoingUsing login credentials from the CLI:Credentials are supplied at run time. Contents of/tmp/.envare not used.docker run --rm -u $(id -u) --name=pl-pull_scp          \
            -v $PWD/out:/outgoing                       \
            -it                                         \
            local/pl-pull_scp pull_scp -v 1             \
            --filepath /home/chris/Pictures             \
            --host 192.168.1.200                        \
            --username chris                            \
            --password XXXXXXXXXXX                      \
            /outgoing
10-15-2021 17:05:13 | INFO: /tmp/key.pub uploaded to 192.168.1.200

             _ _
            | | |
 _ __  _   _| | |  ___  ___ _ __
| '_ \| | | | | | / __|/ __| '_ \
| |_) | |_| | | | \__ \ (__| |_) |
| .__/ \__,_|_|_| |___/\___| .__/
| |           ______       | |
|_|          |______|      |_|

Version: X.Y.Z
10-15-2021 17:05:14 | INFO: (remote): du -ksh /home/chris/Pictures: 81M     /home/chris/Pictures
10-15-2021 17:05:14 | INFO: Pulling chris@192.168.1.200:/home/chris/Pictures...
10-15-2021 17:06:01 | INFO: Remote contents pulled to /outgoing
10-15-2021 17:06:01 | INFO: (NOTE: if running in a container, and doing a volume mapping,
                            the destination dir name might not match the host dirname!)
10-15-2021 17:06:01 | INFO: (local): du -ksh /outgoing: 156M        /outgoing(note that in this example thelocaldirectory is larger than theremote. This occurs when the remote directory contains symbolic links – each symbolic link is actually translated into the target file when pulled)_-30-_