Repository: ckan/ckan

Issue:
Default view is re-added when removed before DataStore upload is complete
I'm doing the following via the API on a CKAN 2.5.2 installation that has DataStore + DataPusher enabled:
1. Create a CSV resource
2. Remove the default `recline_view`
3. Add a new `recline_grid_view` view

I'd expect the resource to end up with a single view (the one I added in step 3). However, it ends up with _two_ views: the one I created and a _new_ default view (ID is different from the original default view).

It seems that the default view is added again once the data has been uploaded to the DataStore by the DataPusher. If steps 1-3 are done quickly enough (before the upload to the DataStore has completed) then the resource has only the intended `recline_grid_view`, which at that point shows an error (because the data is not yet available). As soon as the data is available (i.e. as soon as the error is gone after a reload) the resource has gotten an additional `recline_view`.

Here's a small script to demonstrate the problem:

``` python
ckan = ckanapi.RemoteCKAN(CKAN_URL, apikey=API_KEY)

res_dict = ckan.action.resource_create(package_id=PKG_ID, url=CSV_URL,
                                       name='Test', format='CSV')
view_dicts = ckan.action.resource_view_list(id=res_dict['id'])
for view_dict in view_dicts:
    ckan.action.resource_view_delete(id=view_dict['id'])
ckan.action.resource_view_create(resource_id=res_dict['id'], title='My View',
                                 view_type='recline_grid_view')

# Uncomment to make assertion fail
#import time
#time.sleep(10)

assert len(ckan.action.resource_view_list(id=res_dict['id'])) == 1
```

If you uncomment the `sleep` lines (to wait until the data is in the DataStore) then the assertion fails.



Discussion:
This is another side-effect of #2234 (which I'm very much starting to regret to ever have implemented). Essentially we added a call to `resource_patch` on `datastore_create` to set the `datastore_active` extra on the resource.

So what happens is:
- Resource is created, default views are created
- You delete the default view you don't want
- In the meantime, DataPusher is doing its thing
- When the data is uploaded to the DataStore, `datastore_create` calls `resource_patch` (to set up the `datastore_active` extra) which calls `resource_update` which calls `package_update` which calls `package_create_default_resource_views` and recreates the view.

@wardi I know you hate it but the only thing I can think of is passing a `create_views = False` entry on the context that ends up in `package_update` and avoids the default views creation.

@torfsen I know it's not ideal but on the meantime you can delete the extra view via the API if you are doing it as part of a script

So datapusher is deleting and recreating the datastore table. That shows up as a new table so the default views are created, or are we calling package_create_default_resource_views on every package_update?

> are we calling package_create_default_resource_views on every package_update?

This. It has no relation with the DataPusher creating or dropping tables, other than it triggers an update when it imports data to the DataStore 

So.. anyone editing a dataset will cause all the default resource views to reappear too, right?

We need something like a white-out for resource views that have been removed so they don't get re-added, or we just need to me more selective about when that code is run, not on every package_update

You are right.
Neither of the two options are easy as things stand right now. Views are not marked as deleted on the DB, they are just removed entirely so the white-out option is probably the hardest.

Thinking on when we want default views to be created, I think the most common cases are:
- Creating a package (if resources are provided, ie creating it via the API or a custom combined form)
- Creating resource on an existing package (like on the default UI)
- When we finish pushing data to the DataStore (DataPusher does that now)

So just on `package_create`, `resource_create` and `datapusher_hook`.

If we only cover these cases, and not any updates we will miss situations like people changing the format of a resource and new views for that format not appearing, but that might be preferable to the current behaviour.

:+1: That sounds like a simpler solution.

@torfsen Do you have time to work on patch along those lines?

@k-nut this is also a good one to get an introduction about how DataPusher / DataStore / Resource views work

@amercader Sorry, I'm currently busy with other stuff.

As a workaround I'm currently polling [`datastore_info`](http://docs.ckan.org/en/latest/maintaining/datastore.html#ckanext.datastore.logic.action.datastore_info) until the resource has been uploaded to the datastore before deleting the default views.

@amercader I am taking a look at this now.


Resolving patch:
diff --git a/ckan/logic/action/create.py b/ckan/logic/action/create.py
--- a/ckan/logic/action/create.py
+++ b/ckan/logic/action/create.py
@@ -316,6 +316,16 @@ def resource_create(context, data_dict):
     updated_pkg_dict = _get_action('package_show')(context, {'id': package_id})
     resource = updated_pkg_dict['resources'][-1]
 
+    ##  Add the default views to the new resource
+    logic.get_action('resource_create_default_resource_views')(
+        {'model': context['model'],
+         'user': context['user'],
+         'ignore_auth': True
+         },
+        {'resource': resource,
+         'package': updated_pkg_dict
+         })
+
     for plugin in plugins.PluginImplementations(plugins.IResourceController):
         plugin.after_create(context, resource)
 
diff --git a/ckan/logic/action/update.py b/ckan/logic/action/update.py
--- a/ckan/logic/action/update.py
+++ b/ckan/logic/action/update.py
@@ -307,13 +307,6 @@ def package_update(context, data_dict):
 
         item.after_update(context, data)
 
-    # Create default views for resources if necessary
-    if data.get('resources'):
-        logic.get_action('package_create_default_resource_views')(
-            {'model': context['model'], 'user': context['user'],
-             'ignore_auth': True},
-            {'package': data})
-
     if not context.get('defer_commit'):
         model.repo.commit()
 
diff --git a/ckanext/reclineview/plugin.py b/ckanext/reclineview/plugin.py
--- a/ckanext/reclineview/plugin.py
+++ b/ckanext/reclineview/plugin.py
@@ -62,7 +62,7 @@ def update_config(self, config):
     def can_view(self, data_dict):
         resource = data_dict['resource']
         return (resource.get('datastore_active') or
-                resource.get('url') == '_datastore_only_resource')
+                '_datastore_only_resource' in resource.get('url', ''))
 
     def setup_template_variables(self, context, data_dict):
         return {'resource_json': json.dumps(data_dict['resource']),
@@ -90,7 +90,7 @@ def can_view(self, data_dict):
         resource = data_dict['resource']
 
         if (resource.get('datastore_active') or
-                resource.get('url') == '_datastore_only_resource'):
+                '_datastore_only_resource' in resource.get('url', '')):
             return True
         resource_format = resource.get('format', None)
         if resource_format:


Test patch:
diff --git a/ckan/tests/lib/test_datapreview.py b/ckan/tests/lib/test_datapreview.py
--- a/ckan/tests/lib/test_datapreview.py
+++ b/ckan/tests/lib/test_datapreview.py
@@ -271,50 +271,6 @@ def test_default_views_created_on_package_create(self):
             eq_(len(views_list), 1)
             eq_(views_list[0]['view_type'], 'image_view')
 
-    def test_default_views_created_on_package_update(self):
-
-        dataset_dict = factories.Dataset(
-            resources=[{
-                'url': 'http://not.for.viewing',
-                'format': 'xxx',
-            }]
-        )
-
-        resource_id = dataset_dict['resources'][0]['id']
-
-        views_list = helpers.call_action('resource_view_list', id=resource_id)
-
-        eq_(len(views_list), 0)
-
-        updated_data_dict = {
-            'id': dataset_dict['id'],
-            'resources': [
-                {
-                    'url': 'http://not.for.viewing',
-                    'format': 'xxx',
-                },
-                {
-                    'url': 'http://some.image.png',
-                    'format': 'png',
-                },
-
-
-            ]
-        }
-
-        dataset_dict = helpers.call_action('package_update', **updated_data_dict)
-
-        for resource in dataset_dict['resources']:
-            resource_id = resource['id'] if resource['format'] == 'PNG' else None
-
-        assert resource_id
-
-        updated_views_list = helpers.call_action('resource_view_list', id=resource_id)
-        eq_(len(updated_views_list), 1)
-        eq_(updated_views_list[0]['view_type'], 'image_view')
-
-        pass
-
     def test_default_views_created_on_resource_create(self):
 
         dataset_dict = factories.Dataset(
@@ -336,32 +292,3 @@ def test_default_views_created_on_resource_create(self):
 
         eq_(len(views_list), 1)
         eq_(views_list[0]['view_type'], 'image_view')
-
-    def test_default_views_created_on_resource_update(self):
-
-        dataset_dict = factories.Dataset(
-            resources=[{
-                'url': 'http://not.for.viewing',
-                'format': 'xxx',
-            }]
-        )
-
-        resource_id = dataset_dict['resources'][0]['id']
-
-        views_list = helpers.call_action('resource_view_list', id=resource_id)
-
-        eq_(len(views_list), 0)
-
-        resource_dict = {
-            'id': resource_id,
-            'package_id': dataset_dict['id'],
-            'url': 'http://some.image.png',
-            'format': 'png',
-        }
-
-        updated_resource_dict = helpers.call_action('resource_update', **resource_dict)
-
-        views_list = helpers.call_action('resource_view_list', id=updated_resource_dict['id'])
-
-        eq_(len(views_list), 1)
-        eq_(views_list[0]['view_type'], 'image_view')