Repository: ARMmbed/yotta (python)

Issue:
Install yotta modules from github with git credentials
I'd like to do following but it fails:
```
$ yotta install git@github.com:ARMmbed/module-x.git
info: get versions for git
Fatal Exception, yotta=0.17.2
Traceback (most recent call last):
  File "/home/jaakor01/workspace/yotta_issue/venv/bin/yotta", line 4, in <module>
    yotta.main()
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/main.py", line 61, in wrapped
    return fn(*args, **kwargs)
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/main.py", line 46, in wrapped
    return fn(*args, **kwargs)
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/main.py", line 243, in main
    status = args.command(args, following_args)
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/install.py", line 62, in execCommand
    return installComponent(args)
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/install.py", line 202, in installComponent
    working_directory = path
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/access.py", line 385, in satisfyVersion
    name, version_required, working_directory, type=type, inherit_shrinkwrap = inherit_shrinkwrap
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/access.py", line 314, in satisfyVersionByInstalling
    v = latestSuitableVersion(name, version_required, _registryNamespaceForType(type))
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/access.py", line 159, in latestSuitableVersion
    local_clone = remote_component.clone()
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/git_access.py", line 114, in clone
    clone = vcs.Git.cloneToTemporaryDir(self.url)
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/vcs.py", line 62, in cloneToTemporaryDir
    return cls.cloneToDirectory(remote, tempfile.mkdtemp())
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/vcs.py", line 69, in cloneToDirectory
    cls._execCommands(commands)
  File "/home/jaakor01/workspace/yotta_issue/venv/local/lib/python2.7/site-packages/yotta/lib/vcs.py", line 146, in _execCommands
    raise VCSError("command failed: %s" % (err or out), returncode=returncode, command=cmd)
yotta.lib.vcs.VCSError: command failed: Cloning into '/tmp/tmpi_bJ_8'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

However, if I add the same address manually to `module.json` it works with `yotta update`.
```
"dependencies": {
    "module-x":"git@github.com:ARMmbed/module-x.git#master"
  },
```

Pull request description:
Git and GitHub support rework
This pull request fixes the following issues:

- Added support for Git and GitHub hash commits in package specifications
- Fixed the `install` command to correctly support all non-registry forms
- Fixed the saving of non-registry package specifications to `module.json`
- Additional tests for the above
- Updated docs for the above

Fixes #801, #691 and #566 

@bogdanm @marcuschangarm @yogpan01 @0xc0170 @bremoran @LiyouZhou @korjaa 

Patch:
diff --git a/docs/reference/buildsystem.md b/docs/reference/buildsystem.md
index e728c9f..6cc8bd9 100644
--- a/docs/reference/buildsystem.md
+++ b/docs/reference/buildsystem.md
@@ -30,7 +30,7 @@ The name of the library being built by the current module is available as
 No header needs to be included for this definition to be available.
 
 Use the [preprocessor stringification
-trick](https://gcc.gnu.org/onlinedocs/cpp/Stringizing.html) to get the
+trick](https://gcc.gnu.org/onlinedocs/cpp/Stringification.html) to get the
 module name as a string, if desired. Note that this definition is **not**
 currently available when compiling tests, and there are other circumstances
 where using custom CMake can make it unavailable.
diff --git a/docs/reference/commands.md b/docs/reference/commands.md
index 7269eca..30ee1bf 100755
--- a/docs/reference/commands.md
+++ b/docs/reference/commands.md
@@ -534,7 +534,7 @@ example:
 
 
 ## <a href="#yotta-uninstall" name="yotta-uninstall">#</a> yotta uninstall
-Synonyms: `yotta unlink`, `yotta rm`
+Synonyms: `yotta unlink`, `yotta rm`, `yotta un`
 #### Synopsis
 
 ```
diff --git a/docs/reference/config.md b/docs/reference/config.md
index d56a2f5..43be6bb 100644
--- a/docs/reference/config.md
+++ b/docs/reference/config.md
@@ -267,7 +267,7 @@ definitions will be produced:
 Note that string values are not quoted. If you want a quoted string,
 either embed escaped quotes (`\"`) in the string value, or use the preprocessor
 [stringification
-trick](https://gcc.gnu.org/onlinedocs/cpp/Stringizing.html).
+trick](https://gcc.gnu.org/onlinedocs/cpp/Stringification.html).
 
 JSON boolean values are converted to 1 or 0, and `null` values are converted to `NULL`.
 
diff --git a/docs/reference/module.md b/docs/reference/module.md
index 4bc38c5..57498c1 100755
--- a/docs/reference/module.md
+++ b/docs/reference/module.md
@@ -198,6 +198,10 @@ To specify a dependency on a github module, use one of the following forms:
 
     Uses the latest committed version on the specified branch.
 
+ * `"usefulmodule": "username/repositoryname#commit-id"`
+
+    Uses the specified commit ID.
+
 #### Depending on git Modules
 To specify a module available from a non-Github git server as a dependency, use
 a git URL:
@@ -206,8 +210,9 @@ a git URL:
  * `"usefulmodule": "git+ssh://somwhere.com/anything/anywhere#<version specification>"`
  * `"usefulmodule": "git+ssh://somwhere.com/anything/anywhere#<branch name>"`
  * `"usefulmodule": "git+ssh://somwhere.com/anything/anywhere#<tag name>"`
+ * `"usefulmodule": "git+ssh://somwhere.com/anything/anywhere#<commit id>"`
  * `"usefulmodule": "<anything>://somwhere.git"`
- * `"usefulmodule": "<anything>://somwhere.git#<version spec, tag, or branch name>"`
+ * `"usefulmodule": "<anything>://somwhere.git#<version spec, tag, branch name or commit id>"`
 
 #### Depending on hg Modules
 To specify a module available from a mercurial server as a dependency, use
diff --git a/docs/tutorial/privaterepos.md b/docs/tutorial/privaterepos.md
index 471d2de..630ad9a 100644
--- a/docs/tutorial/privaterepos.md
+++ b/docs/tutorial/privaterepos.md
@@ -24,18 +24,33 @@ Sometimes it may not be appropriate publish a module to the public package regis
 
 The shorthand GitHub URL is formed of two parts: `<username>/<reponame>` where `<username>` is the GitHub user or organisation name of the repository owner and `<reponame>` is the name of the repositiry. e.g. the `yotta` repositry can be found at `ARMmbed/yotta`.
 
-You can specify a particular branch or tag to use by providing it in the URL. The supported GitHub URL formats are:
+You can specify a particular branch, tag or commit to use by providing it in the URL. The supported GitHub URL formats are:
 
 ```
 username/reponame
 username/reponame#<versionspec>
 username/reponame#<branchname>
 username/reponame#<tagname>
+username/reponame#<commit>
+https://github.com/username/reponame
+https://github.com/username/reponame#<branchname>
+https://github.com/username/reponame#<tagname>
+https://github.com/username/reponame#<commit>
 ```
 
+If the GitHub repository is public, the dependency will simply be downloaded. If the GitHub repository is private and this is the first time you are downloading from a private GitHub repository, you will be prompted to log in to GitHub using a URL.
+
+If you have a private GitHub repository and you would prefer to download it using SSH keys, you can use the following dependency form:
+
+```
+git@github.com:username/reponame.git
+git@github.com:username/reponame.git#<branchname>
+git@github.com:username/reponame.git#<tagname>
+git@github.com:username/reponame.git#<commit>
+```
 
 ###Other ways to depend on private repositories 
-Using shorthand GitHub URLs is the easiest and reccomneded method of working with private repositories, however as not all projects are hosted on GitHub, `yotta` supports using git and hg URLs directly as well.
+Using shorthand GitHub URLs is the easiest and recommended method of working with private repositories, however as not all projects are hosted on GitHub, `yotta` supports using git and hg URLs directly as well.
 
 For example, to include a privately hosted git repository from example.com:
 
@@ -47,13 +62,13 @@ For example, to include a privately hosted git repository from example.com:
 ...
 ```
 
-Git URLs support branch, version and tags specifications:
+Git URLs support branch, version, tag and commit specifications:
 
 ```
 git+ssh://example.com/path/to/repo
-git+ssh://example.com/path/to/repo#<versionspec, branch or tag>
+git+ssh://example.com/path/to/repo#<versionspec, branch, tag or commit>
 anything://example.com/path/to/repo.git
-anything://example.com/path/to/repo.git#<versionspec, branch or tag>
+anything://example.com/path/to/repo.git#<versionspec, branch, tag or commit>
 ```
 
 Currently, mercurial URLs only support a version specification:
diff --git a/yotta/install.py b/yotta/install.py
index 28e2816..1ed7d8a 100644
--- a/yotta/install.py
+++ b/yotta/install.py
@@ -167,7 +167,12 @@ def installComponentAsDependency(args, current_component):
     # (if it is not already present), and write that back to disk. Without
     # writing to disk the dependency wouldn't be usable.
     if installed and not current_component.hasDependency(component_name):
-        saved_spec = current_component.saveDependency(installed)
+        vs = sourceparse.parseSourceURL(component_spec)
+        if vs.source_type == 'registry':
+            saved_spec = current_component.saveDependency(installed)
+        else:
+            saved_spec = current_component.saveDependency(installed, component_spec)
+
         current_component.writeDescription()
         logging.info('dependency %s: %s written to module.json', component_name, saved_spec)
     else:
diff --git a/yotta/lib/access.py b/yotta/lib/access.py
index ae8fcac..51dec3b 100644
--- a/yotta/lib/access.py
+++ b/yotta/lib/access.py
@@ -147,8 +147,14 @@ def latestSuitableVersion(name, version_required, registry='modules', quiet=Fals
             )
             if v:
                 return v
+
+            # we have passed a specific commit ID:
+            v = remote_component.commitVersion()
+            if v:
+                return v
+
             raise access_common.Unavailable(
-                'Github repository "%s" does not have any tags or branches matching "%s"' % (
+                'Github repository "%s" does not have any tags, branches or commits matching "%s"' % (
                     version_required, remote_component.tagOrBranchSpec()
                 )
             )
@@ -189,8 +195,14 @@ def latestSuitableVersion(name, version_required, registry='modules', quiet=Fals
             )
             if v:
                 return v
+
+            # we have passed a specific commit ID:
+            v = local_clone.commitVersion(remote_component.tagOrBranchSpec())
+            if v:
+                return v
+
             raise access_common.Unavailable(
-                '%s repository "%s" does not have any tags or branches matching "%s"' % (
+                '%s repository "%s" does not have any tags, branches or commits matching "%s"' % (
                     clone_type, version_required, spec
                 )
             )
diff --git a/yotta/lib/git_access.py b/yotta/lib/git_access.py
index f53e19f..4b2bbb1 100644
--- a/yotta/lib/git_access.py
+++ b/yotta/lib/git_access.py
@@ -73,8 +73,18 @@ class GitWorkingCopy(object):
 
 
     def tipVersion(self):
-        return GitCloneVersion('', '', self)
+        raise NotImplementedError
 
+    def commitVersion(self, spec):
+        ''' return a GithubComponentVersion object for a specific commit if valid
+        '''
+        import re
+
+        commit_match = re.match('^[a-f0-9]{7,40}$', spec, re.I)
+        if commit_match:
+            return GitCloneVersion('', spec, self)
+
+        return None
 
 class GitComponent(access_common.RemoteComponent):
     def __init__(self, url, tag_or_branch=None, semantic_spec=None):
diff --git a/yotta/lib/github_access.py b/yotta/lib/github_access.py
index e2d47a1..b1f293f 100644
--- a/yotta/lib/github_access.py
+++ b/yotta/lib/github_access.py
@@ -141,6 +141,12 @@ def _getTipArchiveURL(repo):
     repo = g.get_repo(repo)
     return repo.get_archive_link('tarball')
 
+@_handleAuth
+def _getCommitArchiveURL(repo, commit):
+    ''' return a string containing a tarball url '''
+    g = Github(settings.getProperty('github', 'authtoken'))
+    repo = g.get_repo(repo)
+    return repo.get_archive_link('tarball', commit)
 
 @_handleAuth
 def _getTarball(url, into_directory, cache_key, origin_info=None):
@@ -283,6 +289,19 @@ class GithubComponent(access_common.RemoteComponent):
             '', '', _getTipArchiveURL(self.repo), self.name, cache_key=None
         )
 
+    def commitVersion(self):
+        ''' return a GithubComponentVersion object for a specific commit if valid
+        '''
+        import re
+
+        commit_match = re.match('^[a-f0-9]{7,40}$', self.tagOrBranchSpec(), re.I)
+        if commit_match:
+            return GithubComponentVersion(
+                '', '', _getCommitArchiveURL(self.repo, self.tagOrBranchSpec()), self.name, cache_key=None
+            )
+
+        return None
+
     @classmethod
     def remoteType(cls):
         return 'github'
diff --git a/yotta/lib/sourceparse.py b/yotta/lib/sourceparse.py
index 1b1176e..0f451ad 100644
--- a/yotta/lib/sourceparse.py
+++ b/yotta/lib/sourceparse.py
@@ -51,39 +51,55 @@ class VersionSource(object):
             return self.semantic_spec.match(v)
 
 
-def _splitFragment(url):
-    parsed = urlsplit(url)
-    if '#' in url:
-        return url[:url.index('#')], parsed.fragment
-    else:
-        return url, None
-
-def _getGithubRef(source_url):
+def _getNonRegistryRef(source_url):
     import re
+
     # something/something#spec = github
-    defragmented, fragment = _splitFragment(source_url)
-    github_match = re.match('^[a-z0-9_-]+/([a-z0-9_-]+)$', defragmented, re.I)
+    # something/something@spec = github
+    # something/something spec = github
+    github_match = re.match('^([.a-z0-9_-]+/([.a-z0-9_-]+)) *[@#]?([.a-z0-9_\-\*\^\~\>\<\=]*)$', source_url, re.I)
     if github_match:
-        return github_match.group(1), VersionSource('github', defragmented, fragment)
+        return github_match.group(2), VersionSource('github', github_match.group(1), github_match.group(3))
 
-    # something/something@spec = github
-    alternate_github_match = re.match('([a-z0-9_-]+/([a-z0-9_-]+)) *@?([~^><=.0-9a-z\*-]*)$', source_url, re.I)
-    if alternate_github_match:
-        return alternate_github_match.group(2), VersionSource('github', alternate_github_match.group(1), alternate_github_match.group(3))
+    parsed = urlsplit(source_url)
+
+    # github
+    if parsed.netloc.endswith('github.com'):
+        # any URL onto github should be fetched over the github API, even if it
+        # would parse as a valid git URL
+        name_match = re.match('^/([.a-z0-9_-]+/([.a-z0-9_-]+?))(.git)?$', parsed.path, re.I)
+        if name_match:
+            return name_match.group(2), VersionSource('github', name_match.group(1), parsed.fragment)
+
+    if '#' in source_url:
+        without_fragment = source_url[:source_url.index('#')]
+    else:
+        without_fragment = source_url
+
+    # git
+    if parsed.scheme.startswith('git+') or parsed.path.endswith('.git'):
+        # git+anything://anything or anything.git is a git repo:
+        name_match = re.match('^.*?([.a-z0-9_-]+?)(.git)?$', parsed.path, re.I)
+        if name_match:
+            return name_match.group(1), VersionSource('git', without_fragment, parsed.fragment)
+
+    # mercurial
+    if parsed.scheme.startswith('hg+') or parsed.path.endswith('.hg'):
+        # hg+anything://anything or anything.hg is a hg repo:
+        name_match = re.match('^.*?([.a-z0-9_-]+?)(.hg)?$', parsed.path, re.I)
+        if name_match:
+            return name_match.group(1), VersionSource('hg', without_fragment, parsed.fragment)
 
     return None, None
 
+
 def parseSourceURL(source_url):
     ''' Parse the specified version source URL (or version spec), and return an
         instance of VersionSource
     '''
-    import re
-    parsed = urlsplit(source_url)
-
-    if '#' in source_url:
-        without_fragment = source_url[:source_url.index('#')]
-    else:
-        without_fragment = source_url
+    name, spec = _getNonRegistryRef(source_url)
+    if spec:
+        return spec
 
     try:
         url_is_spec = version.Spec(source_url)
@@ -94,22 +110,6 @@ def parseSourceURL(source_url):
         # if the url is an unadorned version specification (including an empty
         # string) then the source is the module registry:
         return VersionSource('registry', '', source_url)
-    elif parsed.netloc.endswith('github.com'):
-        # any URL onto github should be fetched over the github API, even if it
-        # would parse as a valid git URL
-        return VersionSource('github', parsed.path, parsed.fragment)
-    elif parsed.scheme.startswith('git+') or parsed.path.endswith('.git'):
-        # git+anything://anything or anything.git is a git repo:
-        return VersionSource('git', without_fragment, parsed.fragment)
-    elif parsed.scheme.startswith('hg+') or parsed.path.endswith('.hg'):
-        # hg+anything://anything or anything.hg is a hg repo:
-        return VersionSource('hg', without_fragment, parsed.fragment)
-
-    # something/something@spec = github
-    # something/something#spec = github
-    module_name, github_match = _getGithubRef(source_url)
-    if github_match:
-        return github_match
 
     raise InvalidVersionSpec("Invalid version specification: \"%s\"" % (source_url))
 
@@ -143,8 +143,8 @@ def parseTargetNameAndSpec(target_name_and_spec):
     import re
     # fist check if this is a raw github specification that we can get the
     # target name from:
-    name, spec = _getGithubRef(target_name_and_spec)
-    if name and spec:
+    name, spec = _getNonRegistryRef(target_name_and_spec)
+    if name:
         return name, target_name_and_spec
 
     # next split at the first @ or , if any
@@ -178,8 +178,8 @@ def parseModuleNameAndSpec(module_name_and_spec):
     import re
     # fist check if this is a raw github specification that we can get the
     # module name from:
-    name, spec = _getGithubRef(module_name_and_spec)
-    if name and spec:
+    name, spec = _getNonRegistryRef(module_name_and_spec)
+    if name:
         return name, module_name_and_spec
 
     # next split at the first @, if any
diff --git a/yotta/link.py b/yotta/link.py
index d13263d..8275e4f 100644
--- a/yotta/link.py
+++ b/yotta/link.py
@@ -11,9 +11,6 @@ def addOptions(parser):
     )
 
 def tryLink(src, dst):
-    # standard library modules, , ,
-    import logging
-
     # fsutils, , misc filesystem utils, internal
     from yotta.lib import fsutils
     try:
diff --git a/yotta/link_target.py b/yotta/link_target.py
index e67de6a..0ad10dd 100644
--- a/yotta/link_target.py
+++ b/yotta/link_target.py
@@ -11,9 +11,6 @@ def addOptions(parser):
     )
 
 def tryLink(src, dst):
-    # standard library modules, , ,
-    import logging
-    
     # fsutils, , misc filesystem utils, internal
     from yotta.lib import fsutils
     try:
diff --git a/yotta/main.py b/yotta/main.py
index c18cd72..12f6839 100644
--- a/yotta/main.py
+++ b/yotta/main.py
@@ -201,6 +201,7 @@ def main():
     short_commands = {
                 'up':subparser.choices['update'],
                 'in':subparser.choices['install'],
+                'un':subparser.choices['uninstall'],
                 'ln':subparser.choices['link'],
                  'v':subparser.choices['version'],
                 'ls':subparser.choices['list'],


Test patch:
diff --git a/yotta/test/test_sourceparse.py b/yotta/test/test_sourceparse.py
index 412f4ad..0b7af6f 100644
--- a/yotta/test/test_sourceparse.py
+++ b/yotta/test/test_sourceparse.py
@@ -11,50 +11,60 @@ import unittest
 # sourceparse, , parse version source urls, internal
 from yotta.lib import sourceparse
 
-
-Registry_URLs = [
-    '',
-    '*',
-    '1.2.3',
-    '>=1.2.3',
-    '^0.1.2',
-    '~0.1.2',
+# Shorthand URLs for GitHub
+ShortHand_URLs = [
+    'username/reponame',
 ]
 
+# Longhand URLs for GitHub
 Github_URLs = [
-    'username/reponame',
-    'username/reponame#1.2.3',
-    'username/reponame#^1.2.3',
-    'username/reponame#-1.2.3',
-    'username/reponame#branch-or-tag-name',
-    'username/reponame@1.2.3',
-    'username/reponame@^1.2.3',
-    'username/reponame@-1.2.3',
-    'username/reponame@branch-or-tag-name',
+    'https://github.com/username/reponame.git',
+    'git://github.com/username/reponame.git',
+    'git+http://username@github.com/username/reponame.git',
+    'git+https://username@github.com/username/reponame.git',
 ]
 
 Git_URLs = [
-    'git+ssh://somewhere.com/something/etc/etc',
-    'git+ssh://somewhere.com/something/etc/etc#1.2.3',
-    'git+ssh://somewhere.com/something/etc/etc#^1.2.3',
-    'git+ssh://somewhere.com/something/etc/etc#~1.2.3',
-    'git+ssh://somewhere.com/something/etc/etc#branch-name',
-    'ssh://somewhere.com/something/etc/etc.git',
-    'ssh://somewhere.com/something/etc/etc.git#^1.2.3',
-    'ssh://somewhere.com/something/etc/etc.git#~1.2.3',
-    'ssh://somewhere.com/something/etc/etc.git#branch-name',
-    'http://somewhere.something/something.git',
+    'http://somewhere.something/reponame.git',
+    'https://somewhere.something/reponame.git',
+    'ssh://somewhere.com/something/etc/reponame.git',
+    'git+ssh://somewhere.com/something/etc/reponame',
+    'git@github.com:username/reponame.git',
 ]
 
 HG_URLs = [
-    'hg+ssh://somewhere.com/something/etc/etc',
-    'hg+ssh://somewhere.com/something/etc/etc#1.2.3',
-    'hg+ssh://somewhere.com/something/etc/etc#^1.2.3',
-    'hg+ssh://somewhere.com/something/etc/etc#~1.2.3',
-    'ssh://somewhere.com/something/etc/etc.hg',
-    'ssh://somewhere.com/something/etc/etc.hg#^1.2.3',
-    'ssh://somewhere.com/something/etc/etc.hg#~1.2.3',
-    'http://somewhere.something/something.hg',
+    'http://somewhere.something/reponame.hg',
+    'https://somewhere.something/reponame.hg',
+    'ssh://somewhere.com/something/etc/reponame.hg',
+    'hg+ssh://somewhere.com/something/etc/reponame',
+]
+
+# We support version spec, branch name, tag name and commit id for GitHub and Git
+Git_Specs = [
+    '',
+    '1.2.3',
+    '^1.2.3',
+    '~1.2.3',
+    '-1.2.3',
+    'branch-or-tag-name',
+    'd5f5049',
+]
+
+# We support only version spec for HG
+HG_Specs = [
+    '',
+    '1.2.3',
+    '^1.2.3',
+    '~1.2.3',
+]
+
+Registry_Specs = [
+    '',
+    '*',
+    '1.2.3',
+    '>=1.2.3',
+    '^0.1.2',
+    '~0.1.2',
 ]
 
 test_invalid_urls = [
@@ -65,24 +75,56 @@ test_invalid_urls = [
 
 class TestParseSourceURL(unittest.TestCase):
     def test_registryURLs(self):
-        for url in Registry_URLs:
+        for url in Registry_Specs:
             sv = sourceparse.parseSourceURL(url)
             self.assertEqual(sv.source_type, 'registry')
 
+    def test_shorthandURLs(self):
+        for url in ShortHand_URLs:
+            for s in Git_Specs:
+                if len(s):
+                    # Shorthand URLs support '@' and ' ' as well as '#'
+                    for m in ['#', '@', ' ']:
+                        sv = sourceparse.parseSourceURL(url + m + s)
+                        self.assertEqual(sv.source_type, 'github')
+                        self.assertEqual(sv.spec, s)
+                else:
+                    sv = sourceparse.parseSourceURL(url)
+                    self.assertEqual(sv.source_type, 'github')
+                    self.assertEqual(sv.spec, s)
+
     def test_githubURLs(self):
         for url in Github_URLs:
-            sv = sourceparse.parseSourceURL(url)
-            self.assertEqual(sv.source_type, 'github')
+            for s in Git_Specs:
+                if len(s):
+                    source = url + '#' + s
+                else:
+                    source = url
+                sv = sourceparse.parseSourceURL(source)
+                self.assertEqual(sv.source_type, 'github')
+                self.assertEqual(sv.spec, s)
 
     def test_gitURLs(self):
         for url in Git_URLs:
-            sv = sourceparse.parseSourceURL(url)
-            self.assertEqual(sv.source_type, 'git')
+            for s in Git_Specs:
+                if len(s):
+                    source = url + '#' + s
+                else:
+                    source = url
+                sv = sourceparse.parseSourceURL(source)
+                self.assertEqual(sv.source_type, 'git')
+                self.assertEqual(sv.spec, s)
 
     def test_hgURLs(self):
         for url in HG_URLs:
-            sv = sourceparse.parseSourceURL(url)
-            self.assertEqual(sv.source_type, 'hg')
+            for s in HG_Specs:
+                if len(s):
+                    source = url + '#' + s
+                else:
+                    source = url
+                sv = sourceparse.parseSourceURL(source)
+                self.assertEqual(sv.source_type, 'hg')
+                self.assertEqual(sv.spec, s)
 
     def test_invalid(self):
         for url in test_invalid_urls:
@@ -101,38 +143,63 @@ class TestParseModuleNameAndSpec(unittest.TestCase):
             self.assertEqual(n, name)
             self.assertEqual(s, '*')
 
+    def test_ShorthandRefs(self):
+        for url in ShortHand_URLs:
+            for spec in Git_Specs:
+                if len(spec):
+                    # Shorthand URLs support '@' and ' ' as well as '#'
+                    for m in ['#', '@', ' ']:
+                        ns = url + m + spec
+                        n, s = sourceparse.parseModuleNameAndSpec(ns)
+                        self.assertEqual(n, 'reponame')
+                        self.assertEqual(s, ns)
+                else:
+                    n, s = sourceparse.parseModuleNameAndSpec(url)
+                    self.assertEqual(n, 'reponame')
+                    self.assertEqual(s, url)
+
     def test_GithubRefs(self):
         for url in Github_URLs:
-            n, s = sourceparse.parseModuleNameAndSpec(url)
-            self.assertEqual(n, 'reponame')
+            for spec in Git_Specs:
+                if len(spec):
+                    ns = url + '#' + spec
+                else:
+                    ns = url
+                n, s = sourceparse.parseModuleNameAndSpec(ns)
+                self.assertEqual(n, 'reponame')
+                self.assertEqual(s, ns)
+
+    def test_GitRefs(self):
+        for url in Git_URLs:
+            for spec in Git_Specs:
+                if len(spec):
+                    ns = url + '#' + spec
+                else:
+                    ns = url
+                n, s = sourceparse.parseModuleNameAndSpec(ns)
+                self.assertEqual(n, 'reponame')
+                self.assertEqual(s, ns)
+
+    def test_HGRefs(self):
+        for url in HG_URLs:
+            for spec in HG_Specs:
+                if len(spec):
+                    ns = url + '#' + spec
+                else:
+                    ns = url
+                n, s = sourceparse.parseModuleNameAndSpec(ns)
+                self.assertEqual(n, 'reponame')
+                self.assertEqual(s, ns)
 
     def test_atVersion(self):
         for name in Valid_Names:
-            for v in Registry_URLs:
+            for v in Registry_Specs:
                 if len(v):
                     nv = name + '@' + v
                     n, s = sourceparse.parseModuleNameAndSpec(nv)
                     self.assertEqual(n, name)
                     self.assertEqual(s, v)
 
-    def test_atGitURL(self):
-        for name in Valid_Names:
-            for v in Git_URLs:
-                nv = name + '@' + v
-                n, s = sourceparse.parseModuleNameAndSpec(nv)
-                self.assertEqual(n, name)
-                self.assertEqual(s, v)
-
-    def test_atHGURL(self):
-        for name in Valid_Names:
-            for v in HG_URLs:
-                nv = name + '@' + v
-                n, s = sourceparse.parseModuleNameAndSpec(nv)
-                self.assertEqual(n, name)
-                self.assertEqual(s, v)
-
 
 if __name__ == '__main__':
     unittest.main()
-
-