client.shared.base_packages: Add new fetcher GitFetcher

This adds git as a new fetcher.  Git is great for
fetching tarballs directly from its source tree
using git-archive. This method allows us to skip
packaging tests into tar.bz2 and instead grab
them directly from a branch or commit or tag.

Underneath the command looks like:

git archive --remote=git://<git-site>/users/dzickus/autotest-tests -o don.tar master testsuite

This would download everything under the directory
testsuite into don.tar and autotest would install
it locally for use. All the local control file would
need to do is add a git repo using

job.add_repository(git://<git-location>:[<branch>])

where <branch> is assumed to be master if not specified.

Also, make it possible to add directories to organize
tests better instead of the current behavior which is
a flat directory level.

Tested by creating a git tree with all the autotest
tests in there and using a local control file to fetch
them.

Signed-off-by: Don Zickus <dzickus@redhat.com>
Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
client.cmdparser: add fetch command

When playing with my changes for the Git Fetcher,
I thought it would be nice if we could just 'fetch'
the control file remotely.

This would make it easier for folks to create
random tests and host them, while letting others
just grab them with a url pointer (either git: or
http:).

It creates a new area to store the fetched tests
(based on the 'test_out_dir' variable) and hooks
this area into the 'list' and 'run' commands.

Tested by running:

autotest-local fetch git://<internal-git-site> testsuite/control.simple

and then run it with

autotest-local run testsuite/control.simple

(the control file adds the remote repo to fetch the tests from)

In order to completely abstract the use of the
job.add_repo to control files, just set fetch_location
in global_config.ini.

Signed-off-by: Don Zickus <dzickus@redhat.com>
base_utils: is_url - Include git in group of valid URL schemas

Include git in the list of valid schemas to determine
whether a string is a url, then remove all custom is_url
implementations.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
job: Move the site_tests area to the job tmp dir

In order to allow the package manager to install
test modules from http or git, move site_tests dir
to the autotest test output area. This area is
guaranteed to be writable by autotest in all use
cases, since all test locations so far are read-only.

Signed-off-by: Don Zickus <dzickus@redhat.com>
Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
global_config.ini: Add fetch_location for packaging

Add a commented out fetch_location key in global_config.ini,
which will help make this param more discoverable. From
this key autotest will pick up the repositories for
dependencies, tests, profilers and client.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
client.cmdparser: Use the logging system

When testing the new autotest-local fetch command, I've
noticed that a *lot* of debug output is thrown, which
might be confusing.

So, make the CmdParser code to use the logging code
used by other autotest entry points, making the INFO
level the default. This reduces the amount of
output of autotest-local using the fetch action
drastically.

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
client.shared.base_packages: documentation and style overhaul

To educate myself, and hopefuly other, about the inner workings and
interface of the package manager, repository fetchers and other
related classes and functions, improve the docstrings and general
code style of this module.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
client.shared.base_packages: move the RepositoryFetcher init code up

To the base class, making it share more code, and add the custom
code (branch handling) to GitFetcher.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
client.shared.base_packages: remove misleading comment

Signed-off-by: Cleber Rosa <crosa@redhat.com>
client.shared.base_packages: deal with different suffixes created by git-archive

It looks like some versions of git-archive may choose to add a suffix
according to the archive format, even when the filename is explictly
specified without it.

This makes the install package post code to deal with these different
suffix possibilities.

Signed-off-by: Cleber Rosa <crosa@redhat.com>