Repository: mkdocs/mkdocs

Issue:
Sometimes deploy to gh-pages fails
Encountered while fixing #966.  Originally added `--force` in #967, but it makes more sense to do some more research on the root cause.

Possibly:
- GH token permissions
- Same GH token used on multiple machines(?)
- Some git call is failing silently

Will investigate repro steps and update here.



Discussion:
Hm. There's a bit of a 🐰 hole here. For context, this is using the token approach from #632 to deploy updates using a CI server.

Looks like the root cause is in [these lines of code](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/utils/ghp_import.py#L72-L76).

Here's where I see the problem:
- Run `mkdocs gh-deploy --clean --remote-name https://{TOKEN}@github.com/owner/repo.git`
- When we call `git rev-list` with a URL as the remote, we would normally see the error `fatal: Invalid object name 'https'.`
- However, `try_rebase` returns True on line 76 if `rev-list` fails, so we just never do the rebase.
- Now when we try to `git push` we see an error (which before #967 would fail silently):

```
To https://{TOKEN}@github.com/owner/repo.git
! [rejected]        gh-pages -> gh-pages (fetch first)
error: failed to push some refs to 'https://{TOKEN}@github.com/owner/repo.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```

Some possible fixes:
- Add `--force` to the push command (cheap, well understood, but rebase still fails so the git history on gh-pages gets _real_ long).
- If remote name starts with 'http', run `git remote set-url origin {REMOTE-URL}` before the rebase. This should allow the git commands that can't hand a URL as a remote to function properly. This doesn't address the fact that the rebase fails.
- Go for a simpler git model that will _always_ destroy the gh-pages branch for real. E.g., check if branch `gh-pages` exists (`git branch -r`). If so, delete it locally (`git branch -D gh-pages`) and remotely (`git push origin :gh-pages`). This could replace [the `try_rebase` call in `ghp_import`](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/utils/ghp_import.py#L163-L164) and we could continue with `run_import` knowing the gh-pages branch is fresh.

Maybe I'm missing a better option, if so happy to do some more investigation if y'all have suggestions.

FWIW, it seems that for other similar tasks (e.g., using Sphinx or Pelican to push to gh-pages) people often include a force push after `ghp-import`:
- https://github.com/Syntaf/travis-sphinx/blob/master/travis_sphinx.py#L51
- http://lexi-lambda.github.io/blog/2015/07/18/automatically-deploying-a-frog-powered-blog-to-github-pages/
- http://blog.mathieu-leplatre.info/publish-your-pelican-blog-on-github-pages-via-travis-ci.html
- http://shaunagordon.com/blog/2016/01/13/pelican-travis-and-github/

@pjbull thanks for doing all the work on this. I agree using force is probably the best way to address this. However, as I understand it, this is not a problem for all users (perhaps only for those doing automated updates from CI servers), so the use of force should be an option. In fact, in all the examples you point to, the user is explicitly including the `--force` flag in their script.

That being the case, my inclination is to add a `--force` flag (off by default) to the [`gh-deploy`](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/__main__.py#L178) command which then gets passed along. Then you (and others) can simply include that flag in your automation script: `mkdocs gh-deploy --force ...`.

The only hesitation I have is that we might want to explore merging recent updates to ghp_import which are missing from our vendored copy. If we are going to do that, I think that should happen first. And perhaps we should offer to pass our local modifications to ghp_import back upstream. Maybe that project could then be better at both being a lib and a command line tool. If so,  we would no longer need to vendor it, which makes for simpler maintenance.

By the way, a possible workaround might be to use [git-subtree](https://github.com/apenwarr/git-subtree) instead of our `gh-deploy` command. It can copy (and preserve) the history of a subdirectory to a separate branch. It will only copy the commits which effected the specified sub-directory. Additionally, any commits which included changes in both the specified subdirectory and other parts of your source only include the changes to the subdirectory. I did a full write-up on [how to use it](https://gist.github.com/waylan/4505033) with GitHub Pages a while back.

Just be aware that if you have been using our `gh-deploy` command, you will need to delete the existing `gh-pages` branch both locally and remotely before using git-subtree the first time.

Personally, I prefer the approach of git-subtree and would like to use it under-the-hood for `gh-deploy`. However, it requires installation of a non-standard git extension which does not have a clean install path on all systems. The existing solution is just easier to get new users up-to-speed on.

👌 I'll start with `--force` as an option, since that seems easiest to implement and for existing users. I'll take a look at pulling the upstream changes to the vendored copy.

Just to get a couple more notes about other options in one place:
Looks like [`subtree` is now in git](https://github.com/apenwarr/git-subtree/blob/master/THIS-REPO-IS-OBSOLETE), so no need for a separate install (though we would take a git>=1.7.11 dependency).

Another option, which seems pretty clean comes from [this comment](https://gist.github.com/cobyism/4730490#gistcomment-1800932):
- Keep site in `.gitignore`
- On deploy, `git clone` gh-pages into the site directory
- Clean and build the site
- cd into the directory, `git add`, `git commit` and `git push`

> Looks like [`subtree`](https://github.com/apenwarr/git-subtree/blob/master/THIS-REPO-IS-OBSOLETE) is now in git...

Yes, I know, but as that page states:

> ... it's in the "contrib" subtree for now, so it's not installed by default.

Which is the same as **not** being included in git on most systems. So, yes, a separate install is still necessary.


Resolving patch:
diff --git a/mkdocs/__main__.py b/mkdocs/__main__.py
--- a/mkdocs/__main__.py
+++ b/mkdocs/__main__.py
@@ -86,6 +86,7 @@ def common_options(f):
                       "overrides the value specified in config")
 remote_name_help = ("The remote name to commit to for Github Pages. This "
                     "overrides the value specified in config")
+force_help = "Force the push to the repository."
 
 
 @click.group(context_settings={'help_option_names': ['-h', '--help']})
@@ -198,8 +199,9 @@ def json_command(clean, config_file, strict, site_dir):
 @click.option('-m', '--message', help=commit_message_help)
 @click.option('-b', '--remote-branch', help=remote_branch_help)
 @click.option('-r', '--remote-name', help=remote_name_help)
+@click.option('--force', is_flag=True, help=force_help)
 @common_options
-def gh_deploy_command(config_file, clean, message, remote_branch, remote_name):
+def gh_deploy_command(config_file, clean, message, remote_branch, remote_name, force):
     """Deploy your documentation to GitHub Pages"""
     try:
         cfg = config.load_config(
@@ -208,7 +210,7 @@ def gh_deploy_command(config_file, clean, message, remote_branch, remote_name):
             remote_name=remote_name
         )
         build.build(cfg, dirty=not clean)
-        gh_deploy.gh_deploy(cfg, message=message)
+        gh_deploy.gh_deploy(cfg, message=message, force=force)
     except exceptions.ConfigurationError as e:  # pragma: no cover
         # Avoid ugly, unhelpful traceback
         raise SystemExit('\n' + str(e))
diff --git a/mkdocs/commands/gh_deploy.py b/mkdocs/commands/gh_deploy.py
--- a/mkdocs/commands/gh_deploy.py
+++ b/mkdocs/commands/gh_deploy.py
@@ -49,7 +49,7 @@ def _get_remote_url(remote_name):
     return host, path
 
 
-def gh_deploy(config, message=None):
+def gh_deploy(config, message=None, force=False):
 
     if not _is_cwd_git_repo():
         log.error('Cannot deploy - this directory does not appear to be a git '
@@ -66,7 +66,7 @@ def gh_deploy(config, message=None):
              config['site_dir'], config['remote_branch'])
 
     result, error = ghp_import.ghp_import(config['site_dir'], message, remote_name,
-                                          remote_branch)
+                                          remote_branch, force)
     if not result:
         log.error("Failed to deploy to GitHub with error: \n%s", error)
         raise SystemExit(1)
diff --git a/mkdocs/utils/ghp_import.py b/mkdocs/utils/ghp_import.py
--- a/mkdocs/utils/ghp_import.py
+++ b/mkdocs/utils/ghp_import.py
@@ -158,7 +158,7 @@ def run_import(srcdir, branch, message, nojekyll):
         sys.stdout.write(enc("Failed to process commit.\n"))
 
 
-def ghp_import(directory, message, remote='origin', branch='gh-pages'):
+def ghp_import(directory, message, remote='origin', branch='gh-pages', force=False):
 
     if not try_rebase(remote, branch):
         log.error("Failed to rebase %s branch.", branch)
@@ -167,8 +167,12 @@ def ghp_import(directory, message, remote='origin', branch='gh-pages'):
 
     run_import(directory, branch, message, nojekyll)
 
-    proc = sp.Popen(['git', 'push', remote, branch],
-                    stdout=sp.PIPE, stderr=sp.PIPE)
+    cmd = ['git', 'push', remote, branch]
+
+    if force:
+        cmd.insert(2, '--force')
+
+    proc = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
 
     out, err = proc.communicate()
     result = proc.wait() == 0


Test patch:
diff --git a/mkdocs/tests/cli_tests.py b/mkdocs/tests/cli_tests.py
--- a/mkdocs/tests/cli_tests.py
+++ b/mkdocs/tests/cli_tests.py
@@ -349,6 +349,8 @@ def test_gh_deploy_defaults(self, mock_gh_deploy, mock_build, mock_load_config):
         g_args, g_kwargs = mock_gh_deploy.call_args
         self.assertTrue('message' in g_kwargs)
         self.assertEqual(g_kwargs['message'], None)
+        self.assertTrue('force' in g_kwargs)
+        self.assertEqual(g_kwargs['force'], False)
         self.assertEqual(mock_build.call_count, 1)
         b_args, b_kwargs = mock_build.call_args
         self.assertTrue('dirty' in b_kwargs)
@@ -456,3 +458,19 @@ def test_gh_deploy_remote_name(self, mock_gh_deploy, mock_build, mock_load_confi
             remote_branch=None,
             remote_name='foo'
         )
+
+    @mock.patch('mkdocs.config.load_config', autospec=True)
+    @mock.patch('mkdocs.commands.build.build', autospec=True)
+    @mock.patch('mkdocs.commands.gh_deploy.gh_deploy', autospec=True)
+    def test_gh_deploy_force(self, mock_gh_deploy, mock_build, mock_load_config):
+
+        result = self.runner.invoke(
+            cli.cli, ['gh-deploy', '--force'], catch_exceptions=False)
+
+        self.assertEqual(result.exit_code, 0)
+        self.assertEqual(mock_gh_deploy.call_count, 1)
+        g_args, g_kwargs = mock_gh_deploy.call_args
+        self.assertTrue('force' in g_kwargs)
+        self.assertEqual(g_kwargs['force'], True)
+        self.assertEqual(mock_build.call_count, 1)
+        self.assertEqual(mock_load_config.call_count, 1)