Repository: translate/pootle

Issue:
Running update_tmserver with --dry-run but no existing LocalTM will fail
If we try to build the LocalTM but don't specify `--rebuild` our initial import will fail because the revision counter does not exist.

``` pytb
$ pootle update_tmserver -v 3 --dry-run 
2015-09-30 10:55:33,485 INFO Loading custom settings from '/root/.pootle/pootle.conf'...
2015-09-30 13:55:33,704 INFO Using Python PO
System check identified some issues:

WARNINGS:
?: (pootle.W017) There are user accounts with duplicate emails. This will not be allowed in Pootle 2.8.
        HINT: Try using 'pootle find_duplicate_emails', and then update user emails with 'pootle update_user_email username email'. You might also want to consider using pootle merge_user or purge_user commands
Traceback (most recent call last):
  File "/var/www/pootle/env/bin/pootle", line 11, in <module>
    sys.exit(main())
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/pootle/runner.py", line 309, in main
    django_settings_module='pootle.settings')
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/pootle/runner.py", line 289, in run_app
    management.execute_from_command_line(command)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/pootle/apps/pootle_app/management/commands/update_tmserver.py", line 152, in handle
    'field': 'revision'
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 506, in search
    params=params, body=body)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 89, in perform_request
    self._raise_error(response.status, raw_data)
  File "/var/www/pootle/env/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[aQHk0CPtT1K_ZZ2YJG8rjQ][translations][0]: SearchParseException[[translations][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"match_all": {}}, "facets": {"stat1": {"statistical": {"field": "revision"}}}}]]]; nested: FacetPhaseExecutionException[Facet [stat1]: No mapping found for field [revision]]; }{[aQHk0CPtT1K_ZZ2YJG8rjQ][translations][1]: SearchParseException[[translations][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"match_all": {}}, "facets": {"stat1": {"statistical": {"field": "revision"}}}}]]]; nested: FacetPhaseExecutionException[Facet [stat1]: No mapping found for field [revision]]; }{[aQHk0CPtT1K_ZZ2YJG8rjQ][translations][2]: SearchParseException[[translations][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"match_all": {}}, "facets": {"stat1": {"statistical": {"field": "revision"}}}}]]]; nested: FacetPhaseExecutionException[Facet [stat1]: No mapping found for field [revision]]; }{[aQHk0CPtT1K_ZZ2YJG8rjQ][translations][3]: SearchParseException[[translations][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"match_all": {}}, "facets": {"stat1": {"statistical": {"field": "revision"}}}}]]]; nested: FacetPhaseExecutionException[Facet [stat1]: No mapping found for field [revision]]; }{[aQHk0CPtT1K_ZZ2YJG8rjQ][translations][4]: SearchParseException[[translations][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"match_all": {}}, "facets": {"stat1": {"statistical": {"field": "revision"}}}}]]]; nested: FacetPhaseExecutionException[Facet [stat1]: No mapping found for field [revision]]; }]')
```



Discussion:
Possible duplicate of #3948

Yes, I think this is the same. I hit this same problem when setting up the local TM. The command needs to be adjusted to ensure the local TM actually exists, and abort telling the user to try again using `--rebuild`. The docs also need to be expanded to inform readers about this scenario.

@unho I think the command options and default options need a little thought or at least need to be robust when in this situation of initial population.

OK this isn't a documentation issue at all.  From my reading this [if statement](https://github.com/translate/pootle/blob/master/pootle/apps/pootle_app/management/commands/update_tmserver.py#L140-L142),

``` py
        if (not options['rebuild'] and
            not options['overwrite'] and
            es.indices.exists(self.INDEX_NAME)):
```

covers the case where there is no `--rebuild` and no existing LocalTM.

So either `es.indices.exists(self.INDEX_NAME)` already existed in this case with no revision stored.  Or it isn't returning the results we needed.  I guess to bullet proof it we should simply bail if we can't get a revision and the index exists.

I think the documentation side has been mostly addressed by 2c5f5412, but I still find the documentation for `update_tmserver` a bit unclear. Will PR this.

To my understanding we need to raise a `CommandError` in case the LocalTM doesn't exist and `--rebuild` hasn't been passed. And this should be done as soon as possible. Will PR this as well.

@unho no I disagree, the issues is not addressed by the docs fix because you shouldn't need to supply `--rebuild` at when initialising the TM, as that if statement validates.

I don't think the current operation is wrong which is this.
1. With no options it should build if the TM is missing or update if the revision is older in LocalTM then in Pootle.  So normal operation is create or update.
2. With `--rebuild` it should dump the current TM and rebuild from scratch.  And you don't want that as default.
3. With `--overwrite` it should update the TM without actually first dumping it.   Thus the TM is still live while we update it.

I think the docs pretty much say that.  Its the non-obvious requirement to use `--rebuild` which is actualy a bug, it shouldn't be required unless specifically wanted.  Lets fix that correctly before trying to make the docs match a bug in the command.

I agree with everything except `--overwrite`. I actually don't understand its purpose. I mean you have the following scenarios:
1. Run with no options and no Local TM:
   - TM is created and
   - All units are pushed to the TM.
2. Run with no options and Local TM:
   - Units with revision newer than last revision present on TM are pushed to TM.
3. Run with `--rebuild` and no Local TM:
   - The same as scenario 1.
4. Run with `--rebuild` and Local TM:
   - TM is deleted and
   - TM is recreated from scratch and
   - All units are pushed to the TM.
5. Run any of the previous ones with `--dry-run`:
   - Tells how many units would be affected, but nothing else happens.

So `--overwrite` is needed for what?

`--overwrite` does not delete the TM and updates it instead of deleting it.  The advantage is that the TM remains live while you perform any update.

So `--overwrite` only makes sense along with `--rebuild`.

Sorry for hijacking the conversation, but Elasticsearch can hot-swap indexes, see https://groups.google.com/forum/#!topic/elasticsearch/tlmU_7kSrfg and https://www.elastic.co/blog/changing-mapping-with-zero-downtime

Ideally, users shouldn't be thinking whether they need to use `--overwrite` or `--rebuild`, rebuilding the index can be done with 100% uptime.

@unho, yes and @iafan hot swapping would be the other way to do it.


Resolving patch:
diff --git a/pootle/apps/pootle_app/management/commands/update_tmserver.py b/pootle/apps/pootle_app/management/commands/update_tmserver.py
--- a/pootle/apps/pootle_app/management/commands/update_tmserver.py
+++ b/pootle/apps/pootle_app/management/commands/update_tmserver.py
@@ -137,6 +137,9 @@ def handle(self, *args, **options):
             if es.indices.exists(self.INDEX_NAME):
                 es.indices.delete(index=self.INDEX_NAME)
 
+        if not options['dry_run'] and not es.indices.exists(self.INDEX_NAME):
+            es.indices.create(index=self.INDEX_NAME)
+
         if (not options['rebuild'] and
             not options['overwrite'] and
             es.indices.exists(self.INDEX_NAME)):


Test patch: