Repository: readthedocs/readthedocs.org

Issue:
After build is triggered, state is shown as failed
To reproduce, fire off a new build, make sure to catch the build list page while VCS operations are happening. Build will be in a failure state. This is a regression where we are setting the state of the build to failed without checking that the build has completed. This might be a byproduct of using multiple environments during the build process.


Discussion:
@davidfischer noted he also saw this in production, which means it might not even be related to the VCS changes
Mmm...

I'm not able to reproduce this in my local instance with `master` branch. I see the states: `Triggered`, `Clonning` and `Installing` without problem.
Same behaviour in .org production
@humitos I also experiment this behavior on my local instance:

- Build any project
- Enter to the build (failure is shown)
- Wait ~5 seconds, reload the page
- Build is continued and successful

![screenshot-2018-2-21 bookpy read the docs](https://user-images.githubusercontent.com/4975310/36497585-bcb8b084-1709-11e8-8b30-da6743bc8bf1.png)
![screenshot-2018-2-21 bookpy read the docs 1](https://user-images.githubusercontent.com/4975310/36497591-bff0aa2c-1709-11e8-9ffe-0896f0ec44a1.png)
![screenshot-2018-2-21 bookpy read the docs 2](https://user-images.githubusercontent.com/4975310/36497595-c3492faa-1709-11e8-87d0-f8310f40b488.png)

I haven't see this on production, yet.
@stsewd i get the same results as you once i click on the build, but the build list page is where i see a failure.

![image](https://user-images.githubusercontent.com/1140183/36554941-defc2234-17bd-11e8-81ac-65e2c4a99eba.png)

Another clue: I can reproduce on first build of the project, not afterwards though.
@humitos I think the problem is with a project without tags.

If I ran this command on a project without tags, the exit code is `1` (here the build is marked as failure).

![error](https://user-images.githubusercontent.com/4975310/36575270-1cf18bba-1818-11e8-9223-d0071a383827.png)

Then on another step the exit code is forced to be 0 (here is when the build goes back to the normality)
Another clue: if I build the same project, the second build doesn't initially show the build as failing in the build list, but if I quickly select the build, it's in a failed state by the end of the VCS commands:

![image](https://user-images.githubusercontent.com/1140183/36608417-01c1d074-1887-11e8-965d-b5b2006aafd8.png)

This also causes the page to stop loading the build commands from the API, as we halt this process with JS when a build has failed. Reloading the page, I see I failed on the next command.
I checked this again and I found something new:

1. I triggered a `stable` build
1. I kill my celery instance because it gets stucked
1. The build is in `Triggered`
1. I run celery again
1. I re-trigger the `stable` build
1. I get this error

Also, I have this crazy log in my `debug.log`

```
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING --- Logging error ---
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING Traceback (most recent call last):
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/doc_builder/backends/sphinx.py", line 148, in append_conf
    self.version.get_conf_py_path()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/builds/models.py", line 228, in get_conf_py_path
    conf_py_path = self.project.conf_dir(self.slug)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/models.py", line 557, in conf_dir
    conf_file = self.conf_file(version)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/models.py", line 553, in conf_file
    ProjectConfigurationError.NOT_FOUND
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING readthedocs.projects.exceptions.ProjectConfigurationError: A configuration file was not found. Make sure you have a conf.py file in your repository.
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING During handling of the above exception, another exception occurred:
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING Traceback (most recent call last):
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/tasks.py", line 399, in run_build
    outcomes = self.build_docs()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/tasks.py", line 583, in build_docs
    outcomes['html'] = self.build_docs_html()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/tasks.py", line 600, in build_docs_html
    html_builder.append_conf()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/doc_builder/backends/sphinx.py", line 151, in append_conf
    self._write_config(master_doc=master_doc)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/doc_builder/backends/sphinx.py", line 63, in _write_config
    safe_write(conf_file, conf_template)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/utils.py", line 125, in safe_write
    fh.write(contents.encode('utf-8', 'ignore'))
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING TypeError: write() argument must be str, not SafeBytes
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING During handling of the above exception, another exception occurred:
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING Traceback (most recent call last):
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/lib/python3.6/logging/__init__.py", line 995, in emit
    stream.write(self.terminator)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING OSError: [Errno 5] Input/output error
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING Call stack:
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/rtd3/bin/celery", line 11, in <module>
    sys.exit(main())
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/celery.py", line 480, in handle_argv
    return self.execute(command, argv)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/celery.py", line 412, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/worker.py", line 221, in run_from_argv
    return self(*args, **options)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/base.py", line 244, in __call__
    ret = self.run(*args, **kwargs)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bin/worker.py", line 256, in run
    worker.start()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/concurrency/base.py", line 131, in start
    self.on_start()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/concurrency/prefork.py", line 112, in on_start
    **self.options)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 422, in __init__
    super(AsynPool, self).__init__(processes, *args, **kwargs)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/pool.py", line 1007, in __init__
    self._create_worker_process(i)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 439, in _create_worker_process
    return super(AsynPool, self)._create_worker_process(i)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/pool.py", line 1116, in _create_worker_process
    w.start()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/process.py", line 124, in start
    self._popen = self._Popen(self)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/context.py", line 333, in _Popen
    return Popen(process_obj)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/popen_fork.py", line 79, in _launch
    code = process_obj._bootstrap()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/pool.py", line 289, in __call__
    sys.exit(self.workloop(pid=pid))
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/billiard/pool.py", line 358, in workloop
    result = (True, prepare_result(fun(*args, **kwargs)))
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/app/trace.py", line 537, in _fast_trace_task
    uuid, args, kwargs, request,
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/.pyenv/versions/3.6.4/envs/rtd3/lib/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/tasks.py", line 297, in run
    self.run_build(record=record, docker=docker)
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/projects/tasks.py", line 414, in run_build
    log.warning('No build ID, not syncing files')
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING File "/home/humitos/rtfd/code/readthedocs.org/readthedocs/doc_builder/environments.py", line 720, in __exit__
    msg='Build finished'))
[06/Mar/2018 15:39:29] celery.redirected:235[20794]: WARNING Message: '(Build) [discoin:stable] Build finished'
Arguments: ()
```
I have more clues...

First, the issue happen when RTD doesn't find a `conf.py` and raises `ProjectConfigurationError`. That exception is being handled by our code by creating a `conf.py` based in a template, so we call `render_to_string` that returns a `django.utils.safestring.SafeText` the one that we pass to `safe_write` which encodes it to `utf8` and call `fh.write` with a `django.utils.safestring.SafeBytes` which fails.

So, at that time, I think the Build is marked as FAILED but since it's an intermediate step that "it's OK to failed" (`ProjectConfigurationError` is not a error to abort) RTD continue building the docs.

I'm not 100% sure yet, but I wanted to write this down before I forget :)
I just noticed that even after closing Celery, there were some celery process around and that probably made things goes that crazy.
OK, I have two suppositions here:

1. go crazy with celery processes
1. this happen when running django and celery with Py3 and the project doesn't have a proper `index.rst` (or raise `ProjectConfigurationError` for some other reason that is not that terrible to abort the build).

I created a PR at #3745 that should fix number 2). I've been trying to replicate the error on that branch and I wasn't able to. I'd like you to take a look and try to test it.

I'm not sure if 1) worth to test it, since it's an edge case and we shouldn't be running multiple celery processes. I will give it a try, though.

Resolving patch:
diff --git a/readthedocs/doc_builder/environments.py b/readthedocs/doc_builder/environments.py
--- a/readthedocs/doc_builder/environments.py
+++ b/readthedocs/doc_builder/environments.py
@@ -183,21 +183,19 @@ def get_command(self):
 
     def save(self):
         """Save this command and result via the API."""
-        exit_code = self.exit_code
-
         # Force record this command as success to avoid Build reporting errors
         # on commands that are just for checking purposes and do not interferes
         # in the Build
         if self.record_as_success:
             log.warning('Recording command exit_code as success')
-            exit_code = 0
+            self.exit_code = 0
 
         data = {
             'build': self.build_env.build.get('id'),
             'command': self.get_command(),
             'description': self.description,
             'output': self.output,
-            'exit_code': exit_code,
+            'exit_code': self.exit_code,
             'start_time': self.start_time,
             'end_time': self.end_time,
         }
@@ -345,7 +343,6 @@ def run_command_class(
         # ``build_env`` is passed as ``kwargs`` when it's called from a
         # ``*BuildEnvironment``
         build_cmd = cls(cmd, **kwargs)
-        self.commands.append(build_cmd)
         build_cmd.run()
 
         if record:
@@ -354,6 +351,12 @@ def run_command_class(
             # only ones that can be saved/recorded)
             self.record_command(build_cmd)
 
+            # We want append this command to the list of commands only if it has
+            # to be recorded in the database (to keep consistency) and also, it
+            # has to be added after ``self.record_command`` since its
+            # ``exit_code`` can be altered because of ``record_as_success``
+            self.commands.append(build_cmd)
+
         if build_cmd.failed:
             msg = u'Command {cmd} failed'.format(cmd=build_cmd.get_command())
 


Test patch: