Repository: ckan/ckan

Issue:
search-index rebuild is destructive by default
**CKAN version**
all

**Describe the bug**
If you run `search-index rebuild` without the `-r` flag, the solr index is first cleared so is empty while being reloaded. On a site with 10k datasets data will be inaccessible for ~30 minutes.

**Steps to reproduce**
run `search-index rebuild` without the `-r` flag

**Expected behavior**
The whole update could happen as a single transaction so that you only see the new records after the reindex is complete, or if the `-e` flag is used then the default should be to not clear the index, a new `--clear` flag could be added for people that want the current behavior. 



Discussion:
@wardi 
I want to know if i understand the task to be done here:

we want the `search-index rebuild` to do the same thing as when `search-index rebuild -r` is called. i.e it refreshes the current index w/o  the `-r` flag.

Also, when `-e` flag is used, the current index should just be refreshed and committed.

And the search index should only be cleared and rebuild when `--clear` flag is used.

did I get it?
not exactly, ideally:

`search-index rebuild` can clear the index but it should be done in the same solr commit so that it doesn't look like all the datasets have disappeared while the rebuild takes place.

`search-index rebuild -e` should behave like `search-index rebuild -e -r`

`search-index rebuild -e --clear` should behave like the current `search-index rebuild -e` (clear the index then commit each record)
ok. i will start working on this
@wardi 

Base on your comment:  `search-index rebuild -e` should behave like `search-index rebuild -e -r`.

While looking at the current code, both commands end up to this line of code which update and commit:

```py
try:
  package_index.update_dict(
      logic.get_action('package_show')(context,
          {'id': pkg_id}
      ),
      defer_commit
  )
. . . . . . . . . .
```

To me, I think both commands are doing the same thing. What do you think?
> @wardi
> 
> Base on your comment: `search-index rebuild -e` should behave like `search-index rebuild -e -r`.
> 
> While looking at the current code, both commands end up to this line of code which update and commit:
> 
> ```python
> try:
>   package_index.update_dict(
>       logic.get_action('package_show')(context,
>           {'id': pkg_id}
>       ),
>       defer_commit
>   )
> . . . . . . . . . .
> ```
> 
> To me, I think both commands are doing the same thing. What do you think?

@wardi 
@wardi 
This command is used to update the search index, not the database. Start by setting up a site to test and understand the `search-index rebuild` command and the behavior of the options described, then trace the code.
ok.
Instead of setting up a site, can't I test this behavior with the test code in: https://github.com/ckan/ckan/blob/master/ckan/tests/cli/test_search_index.py

@wardi 
I just re-read your comment again, here is how I track the code;

I'm using the test for `cli/test_search_index.py`

```py
class TestSearchIndex(object):
    def test_search_index_rebuild(self, cli):
        """Direct update on package won't be reflected in search results till
        re-index.

        """
        dataset = factories.Dataset(title=u"Before rebuild")
        model.Session.query(model.Package).filter_by(id=dataset[u'id']).update(
            {u'title': u'After update'})
        model.Session.commit()

        search_result = helpers.call_action(u'package_search', q=u"After")
        assert search_result[u'count'] == 0
        print("IN TEST NOW")
        result = cli.invoke(ckan, [u'search-index', u'rebuild', '-e', '-r'])
```

Then I trace the functionality to this file `cli/search_index.py`:
```py
@click.argument(u'package_id', required=False)
def rebuild(
        verbose, force, refresh, only_missing, quiet, commit_each, package_id
):
    u''' Rebuild search index '''
    from ckan.lib.search import rebuild, commit
    try:
        rebuild(package_id,
                only_missing=only_missing,
                force=force,
                refresh=refresh,
                defer_commit=(not commit_each),
                quiet=quiet)
    except Exception as e:
        tk.error_shout(e)
    if not commit_each:
        commit()
```

The preceding code has all things to do with `search-index rebuild` and all its following commands.

From the preceding code, I trace the `rebuild` functionality to `ckan/lib/search/__init__.py`:

```py
def rebuild(package_id=None, only_missing=False, force=False, refresh=False,
            defer_commit=False, package_ids=None, quiet=False):
    '''
        Rebuilds the search index.

        If a dataset id is provided, only this dataset will be reindexed.
        When reindexing all datasets, if only_missing is True, only the
        datasets not already indexed will be processed. If force equals
        True, if an exception is found, the exception will be logged, but
        the process will carry on.
    '''
    log.info("Rebuilding search index...")
```

Then from the search-index test, I tried each command and trace them down to each line in `rebuild` function responsible for the cli command functionality. Hence the line of code in my previous comment was from this `rebuild` function.

Like you said, `this command is used to update the search-index and not the database.` I will be glad if you could provide more explanation on this since the `rebuild` function is used to `rebuilds the search index`

@wardi 

cc: @anuveyatsu

You're in the right place. The code that handles reindexing all datasets is https://github.com/ckan/ckan/blob/237644f8436be748b0cb2b74b52e80763083d4a5/ckan/lib/search/__init__.py#L171

and the package indexing logic is https://github.com/ckan/ckan/blob/237644f8436be748b0cb2b74b52e80763083d4a5/ckan/lib/search/index.py#L101

In `rebuild` the `defer_commit` and `refresh` parameters control whether a solr commit will happen after each dataset and whether the index will be cleared first. Clearing the index doesn't currently support a `defer_commit` parameter so that would need to be added.

Next the cli parameters would need to be changed as described above and the documentation updated: https://github.com/ckan/ckan/blob/237644f8436be748b0cb2b74b52e80763083d4a5/ckan/cli/search_index.py#L17


Thanks for the clearification

Resolving patch:
diff --git a/ckan/cli/search_index.py b/ckan/cli/search_index.py
--- a/ckan/cli/search_index.py
+++ b/ckan/cli/search_index.py
@@ -4,7 +4,6 @@
 
 import click
 import sqlalchemy as sa
-
 import ckan.plugins.toolkit as tk
 
 
@@ -18,7 +17,6 @@ def search_index():
 @click.option(u'-v', u'--verbose', is_flag=True)
 @click.option(u'-i', u'--force', is_flag=True,
               help=u'Ignore exceptions when rebuilding the index')
-@click.option(u'-r', u'--refresh', help=u'Refresh current index', is_flag=True)
 @click.option(u'-o', u'--only-missing',
               help=u'Index non indexed datasets only', is_flag=True)
 @click.option(u'-q', u'--quiet', help=u'Do not output index rebuild progress',
@@ -28,9 +26,11 @@ def search_index():
                    u'ensures that changes are immediately available on the'
                    u'search, but slows significantly the process. Default'
                    u'is false.')
+@click.option('-c', '--clear', help='Clear the index before reindexing',
+              is_flag=True)
 @click.argument(u'package_id', required=False)
 def rebuild(
-        verbose, force, refresh, only_missing, quiet, commit_each, package_id
+        verbose, force, only_missing, quiet, commit_each, package_id, clear
 ):
     u''' Rebuild search index '''
     from ckan.lib.search import rebuild, commit
@@ -39,9 +39,9 @@ def rebuild(
         rebuild(package_id,
                 only_missing=only_missing,
                 force=force,
-                refresh=refresh,
                 defer_commit=(not commit_each),
-                quiet=quiet)
+                quiet=quiet,
+                clear=clear)
     except Exception as e:
         tk.error_shout(e)
     if not commit_each:
diff --git a/ckan/lib/search/__init__.py b/ckan/lib/search/__init__.py
--- a/ckan/lib/search/__init__.py
+++ b/ckan/lib/search/__init__.py
@@ -137,8 +137,8 @@ def notify(self, entity, operation):
             log.warn("Discarded Sync. indexing for: %s" % entity)
 
 
-def rebuild(package_id=None, only_missing=False, force=False, refresh=False,
-            defer_commit=False, package_ids=None, quiet=False):
+def rebuild(package_id=None, only_missing=False, force=False, defer_commit=False,
+            package_ids=None, quiet=False, clear=False):
     '''
         Rebuilds the search index.
 
@@ -183,7 +183,7 @@ def rebuild(package_id=None, only_missing=False, force=False, refresh=False,
         else:
             log.info('Rebuilding the whole index...')
             # When refreshing, the index is not previously cleared
-            if not refresh:
+            if clear:
                 package_index.clear()
 
         total_packages = len(package_ids)


Test patch:
diff --git a/ckan/tests/cli/test_search_index.py b/ckan/tests/cli/test_search_index.py
--- a/ckan/tests/cli/test_search_index.py
+++ b/ckan/tests/cli/test_search_index.py
@@ -28,6 +28,13 @@ def test_search_index_rebuild(self, cli):
         search_result = helpers.call_action(u'package_search', q=u"After")
         assert search_result[u'count'] == 1
 
+        # Clear and defer commit
+        result = cli.invoke(ckan, [u'search-index', u'rebuild', '-e', '-c'])
+        print(result.output)
+        assert not result.exit_code
+        search_result = helpers.call_action(u'package_search', q=u"After")
+        assert search_result[u'count'] == 1
+
     def test_test_main_operations(self, cli):
         """Create few datasets, clear index, rebuild it - make sure search results
         are always reflect correct state of index.