Repository: OCHA-DAP/hdx-ckan

Issue:
Search Page (Amok): Section "Feature Pages"
https://docs.google.com/drawings/d/1_zKuikRyl4Kufmxqho_VWdGPE-DRsvX2pkrZnnSjynA/edit
Implement the section called "Feature Pages"

This will populate in controller the feature pages collection.
1. In ALL tab
- it should display one organization, one topic and one country. But for the moment we will display the current results returned from SOLR.
- maximum displayed items = 3
- if there are no results it will display "0 Results"
- no pagination (remove it)
  1. In Feature Pages tab
- add pagination (25 items per page)
- the list with result will be a mix based on SOLR result: organizations, topics and pagination



Discussion:
Okay, so here's the status right now.

Since tags are not an entities, the controller loops through the datasets and isolates all the tags. Then it selects the three closest to the original query (or if there is no query it selects three at random). It generates a dictionary with three parameters: name, display_name, and url. If the feature tab is selected it returns all the tags.

How do we want to do the full feature list? Are we getting rid of this along with Feature Count?

Pinging @mbellotti @cjhendrix @alexandru-m-g 
This is what I remember we discussed in our call. Please correct me if I am wrong.
For every search that we perform there is a result with a list of datasets AND/OR indicators. Basically we iterate through this list and display the related features pages (this means: organizations, countries, topics assigned to datasets/indicators).
If we are on the "ALL" tab this means we we will display feature pages that are related to (max) 1 indicator and (max) 5 datasets that are displayed.
When we switch to "All feature pages" - we perform a full search for all datasets (based on keyword and filters/facets) and based on the list of results we iterate and retrieve the feature pages related to those datasets and display the feature pages in a pagination way (25 items per page).

On the ALL tab, I think the selection of Feature Pages needs to come from more than 5 Datasets + 1 indicator.  Could we use the first 25 instead?  

On our call yesterday, Marianne said she would play around with it, but that's where we left it.

@mbellotti We agreed to keep the feature count for now. 

Yes, but here's where the problem is: Pagination

So I'm on the ALL tab everything is fine, it pulls whatever number of datasets+indicators. Iterates through them and selects the three most relevant items to display (this now includes orgs and countries although Arti hasn't set a unique icon for countries yet). This is fine.

But now I click on the Features tab. CKAN pulls up twenty-five datasets, iterates through and isolates their features then PAGINATES that list.... the first page displays as expected, but when I click the second page I don't get the next twenty-five features, I get tags from the next twenty-five datasets which may be duplicates of the first twenty-five.

If we want to keep the "show all features" functionality I will need to find a solution to this.

I see the problem. I would suggest to try to paginate the list of feature pages only based on filtered datasets. How diff it is to do it?
This list with feature pages will reset in case we use the filters...
@cjhendrix , any feedback?

So if you click on the Featured tab there is no pagination, but if you add some filters there is?

Completely removing pagination from the Featured list and just displaying however many items there are in the first twenty-five datasets is an easy solution. Pending @cjhendrix approval

Ok.  I had to read it a couple of times to get it, but I get it.  CJ RTFI.  

I need to think about it a bit.  Will get back to you after the board call.

I understand Marianne's suggestion, but it has the drawback of potentially removing a lot of orgs and countries from the list. Clarification question: would filtering behave as expected in this scenario?

I'm not sure I understand Dan's suggestion and it's potential drawback. 

If we can't reach a conclusion this afternoon, then we'll talk when Marianne is online tomorrow morning (assuming you're on HDX tomorrow morning).

I will setup a meeting


Resolving patch:
diff --git a/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py b/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py
--- a/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py
+++ b/ckanext-hdx_search/ckanext/hdx_search/controllers/search_controller.py
@@ -2,34 +2,33 @@
 import logging
 import copy
 from urllib import urlencode
-# import datetime
-# import cgi
-
-# from ckanext.hdx_package.helpers import helpers
 
 from pylons import config
-# from genshi.template import MarkupTemplate
-# from genshi.template.text import NewTextTemplate
 from paste.deploy.converters import asbool
 
 import ckan.logic as logic
 import ckan.lib.base as base
 import ckan.lib.maintain as maintain
-# import ckan.lib.package_saver as package_saver
-# import ckan.lib.i18n as i18n
-# import ckan.lib.navl.dictization_functions as dict_fns
-# import ckan.lib.accept as accept
+import ckan.lib.navl.dictization_functions as dict_fns
 import ckan.lib.helpers as h
 import ckan.model as model
-# import ckan.lib.datapreview as datapreview
-# import ckan.lib.plugins
-# import ckan.new_authz as new_authz
+import ckan.lib.search as search
+import sqlalchemy
 import ckan.plugins as p
-
+_validate = dict_fns.validate
+_check_access = logic.check_access
+
+_select = sqlalchemy.sql.select
+_aliased = sqlalchemy.orm.aliased
+_or_ = sqlalchemy.or_
+_and_ = sqlalchemy.and_
+_func = sqlalchemy.func
+_desc = sqlalchemy.desc
+_case = sqlalchemy.case
+_text = sqlalchemy.text
 
 from ckan.common import OrderedDict, _, json, request, c, g, response
 from pydoc_data.topics import topics
-# from ckan.controllers.home import CACHE_PARAMETERS
 
 log = logging.getLogger(__name__)
 
@@ -42,18 +41,6 @@
 ValidationError = logic.ValidationError
 check_access = logic.check_access
 get_action = logic.get_action
-# tuplize_dict = logic.tuplize_dict
-# clean_dict = logic.clean_dict
-# parse_params = logic.parse_params
-# flatten_to_string_key = logic.flatten_to_string_key
-
-# CONTENT_TYPES = {
-#     'text': 'text/plain;charset=utf-8',
-#     'html': 'text/html;charset=utf-8',
-#     'json': 'application/json;charset=utf-8',
-# }
-
-# lookup_package_plugin = ckan.lib.plugins.lookup_package_plugin
 
 from ckan.controllers.package import PackageController
 
@@ -88,39 +75,188 @@ def count_types(context, data_dict, tab):
         indicator = None
     return (datasets['count'], indicators['count'], indicator)
 
+def package_search(context, data_dict):
+    '''
+    EDITTED VERSION GETS MORE INFO FROM FACETS
+    '''
+    # sometimes context['schema'] is None
+    schema = (context.get('schema') or
+              logic.schema.default_package_search_schema())
+    data_dict, errors = _validate(data_dict, schema, context)
+    # put the extras back into the data_dict so that the search can
+    # report needless parameters
+    data_dict.update(data_dict.get('__extras', {}))
+    data_dict.pop('__extras', None)
+    if errors:
+        raise ValidationError(errors)
+
+    model = context['model']
+    session = context['session']
+
+    _check_access('package_search', context, data_dict)
+
+    # Move ext_ params to extras and remove them from the root of the search
+    # params, so they don't cause and error
+    data_dict['extras'] = data_dict.get('extras', {})
+    for key in [key for key in data_dict.keys() if key.startswith('ext_')]:
+        data_dict['extras'][key] = data_dict.pop(key)
+
+    # check if some extension needs to modify the search params
+    for item in p.PluginImplementations(p.IPackageController):
+        data_dict = item.before_search(data_dict)
+
+    # the extension may have decided that it is not necessary to perform
+    # the query
+    abort = data_dict.get('abort_search', False)
+
+    if data_dict.get('sort') in (None, 'rank'):
+        data_dict['sort'] = 'score desc, metadata_modified desc'
+
+    results = []
+    if not abort:
+        data_source = 'data_dict' if data_dict.get('use_default_schema',
+            False) else 'validated_data_dict'
+        # return a list of package ids
+        data_dict['fl'] = 'id {0}'.format(data_source)
+
+        # If this query hasn't come from a controller that has set this flag
+        # then we should remove any mention of capacity from the fq and
+        # instead set it to only retrieve public datasets
+        fq = data_dict.get('fq', '')
+        if not context.get('ignore_capacity_check', False):
+            fq = ' '.join(p for p in fq.split(' ')
+                            if not 'capacity:' in p)
+            data_dict['fq'] = fq + ' capacity:"public"'
+
+        # Pop these ones as Solr does not need them
+        extras = data_dict.pop('extras', None)
+
+        query = search.query_for(model.Package)
+        query.run(data_dict)
+
+        # Add them back so extensions can use them on after_search
+        data_dict['extras'] = extras
+
+        for package in query.results:
+            # get the package object
+            package, package_dict = package['id'], package.get(data_source)
+            pkg_query = session.query(model.PackageRevision)\
+                .filter(model.PackageRevision.id == package)\
+                .filter(_and_(
+                    model.PackageRevision.state == u'active',
+                    model.PackageRevision.current == True
+                ))
+            pkg = pkg_query.first()
+
+            ## if the index has got a package that is not in ckan then
+            ## ignore it.
+            if not pkg:
+                log.warning('package %s in index but not in database' % package)
+                continue
+            ## use data in search index if there
+            if package_dict:
+                ## the package_dict still needs translating when being viewed
+                package_dict = json.loads(package_dict)
+                if context.get('for_view'):
+                    for item in p.PluginImplementations( p.IPackageController):
+                        package_dict = item.before_view(package_dict)
+                results.append(package_dict)
+            else:
+                results.append(model_dictize.package_dictize(pkg,context))
+
+        count = query.count
+        facets = query.facets
+    else:
+        count = 0
+        facets = {}
+        results = []
+
+    search_results = {
+        'count': count,
+        'facets': facets,
+        'results': results,
+        'sort': data_dict['sort']
+    }
+
+    # Transform facets into a more useful data structure.
+    restructured_facets = {}
+    for key, value in facets.items():
+        restructured_facets[key] = {
+                'title': key,
+                'items': []
+                }
+        for key_, value_ in value.items():
+            new_facet_dict = {}
+            new_facet_dict['name'] = key_
+            if key in ('groups', 'organization'):
+                group = model.Group.get(key_)
+                if group:
+                    new_facet_dict['display_name'] = group.display_name
+                    new_facet_dict['description'] = group.description
+                    if hasattr(group,'revision_timestamp'):
+                        new_facet_dict['last_update'] = group.revision_timestamp
+                else:
+                    new_facet_dict['display_name'] = key_
+                    new_facet_dict['description'] = ''
+                    new_facet_dict['last_update'] = ''
+            elif key == 'license_id':
+                license = model.Package.get_license_register().get(key_)
+                if license:
+                    new_facet_dict['display_name'] = license.title
+                else:
+                    new_facet_dict['display_name'] = key_
+            else:
+                new_facet_dict['display_name'] = key_
+            new_facet_dict['count'] = value_
+            restructured_facets[key]['items'].append(new_facet_dict)
+    search_results['search_facets'] = restructured_facets
+
+    # check if some extension needs to modify the search results
+    for item in p.PluginImplementations(p.IPackageController):
+        search_results = item.after_search(search_results,data_dict)
+
+    # After extensions have had a chance to modify the facets, sort them by
+    # display name.
+    for facet in search_results['search_facets']:
+        search_results['search_facets'][facet]['items'] = sorted(
+                search_results['search_facets'][facet]['items'],
+                key=lambda facet: facet['display_name'], reverse=True)
+
+    return search_results
+
+def sort_features(q, features):
+    return sorted(features, key=lambda x: x['count'])
 
-def isolate_features(context, q, packages, tab):
+def isolate_features(context, facets, q, tab, skip=0,limit=25):
     import difflib
     import random
-    #try:
-    #    all_topics = get_action('tag_list')(
-    #        context, {'vocabulary_id': 'Topics'})
-    #except NotFound, e:
-    #    all_topics = []
-    #    log.error('ERROR getting vocabulary named Topics: %r' %
-    #              str(e.extra_msg))
+
+    try:
+        all_topics = get_action('tag_list')(
+            context, {'vocabulary_id': 'Topics'})
+    except NotFound, e:
+        all_topics = []
+        log.error('ERROR getting vocabulary named Topics: %r' %
+                  str(e.extra_msg))
 
     extract = dict()
     tags = list()
     features = list()
-    for i in packages:
-        if 'organization' in i and i['organization'] and i['organization']['name'] not in tags:
-            tags.append(i['organization']['name'])
-            extract[i['organization']['name']] = {'name': i['organization']['name'], 'display_name': i['organization']['title'], 'url': h.url_for(controller='organization',
-                        action='read', id=i['organization']['id']), 'description': i['organization']['description'], 'last_update': i['organization']['revision_timestamp'], 'is_org':True}
+    for o in facets['organization']['items']:
+        tags.append(o['name'])
+        extract[o['name']] = {'name': o['name'], 'display_name': o['display_name'], 'url': h.url_for(controller='organization',
+                        action='read', id=o['name']), 'description': o['description'], 'last_update': '', 'is_org':True, 'count':o['count']}
         
-        for p in i['tags']:
-            if p['vocabulary_id'] and p['name'] not in tags:
-                tags.append(p['name'])
-                extract[p['name']] = {'name': p['name'], 'display_name': p['name'], 'url': h.url_for(controller='ckanext.hdx_search.controllers.search_controller:HDXSearchController',
-                        action='search', vocab_Topics=p['name']), 'description': '', 'last_update': ''}
+    for p in facets['tags']['items']:
+        if p['name'] in all_topics:
+              tags.append(p['name'])
+              extract[p['name']] = {'name': p['name'], 'display_name': p['name'], 'url': h.url_for(controller='ckanext.hdx_search.controllers.search_controller:HDXSearchController',
+                        action='search', vocab_Topics=p['name']), 'description': '', 'last_update': '', 'count':p['count']}
         
-        for g in i['groups']:
-            if g['name'] not in tags:
-                tags.append(g['name'])
-                extract[g['name']] = {'name': g['name'], 'display_name': g['display_name'], 'url': h.url_for( controller='group',
-                        action='read', id=g['id']), 'description': g['description'], 'last_update': '', 'is_org':False}
-
+    for g in facets['groups']['items']:
+        tags.append(g['name'])
+        extract[g['name']] = {'name': g['name'], 'display_name': g['display_name'], 'url': h.url_for( controller='group',
+                action='read', id=g['name']), 'description': g['description'], 'last_update': '', 'is_org':False, 'count':g['count']}
 
     if tab == 'all' and len(tags) > 3:
          if q:
@@ -130,7 +266,10 @@ def isolate_features(context, q, packages, tab):
     else:
         selected = tags
     for s in selected:
-        features.append(extract[s])
+         features.append(extract[s])
+    if tab == 'features':
+        feature_list = sort_features(q, features)
+        return (feature_list[skip:skip+limit], len(features))
     return features
 
 
@@ -304,23 +443,26 @@ def pager_url(q=None, page=None):
                 # For all tab, only paginate datasets
                 data_dict['extras']['ext_indicator'] = 0
 
-            query = get_action('package_search')(context, data_dict)
+            query = package_search(context, data_dict)
             c.dataset_counts, c.indicator_counts, c.indicator = count_types(
                 context, data_dict, c.tab)
-            if c.tab == "all" or c.tab == 'features':
-                ind_and_datasets = list(query['results'])
-                if c.indicator and c.indicator[0]:
-                    ind_and_datasets.append(c.indicator[0])
-                c.features = isolate_features(
-                    context, q, ind_and_datasets, c.tab)
+            c.count = c.dataset_counts+c.indicator_counts
+            if c.tab == "all":
+                c.features = isolate_features(context, query['search_facets'], q, c.tab)
+            
+            if c.tab == 'features':
+                c.features,c.count = isolate_features(context, query['search_facets'], q, c.tab, ((page - 1) * limit), limit)
+
             c.sort_by_selected = query['sort']
 
+            print pager_url
+
             if c.tab == 'features':
                 c.page = h.Page(
                     collection=c.features,
                     page=page,
                     url=pager_url,
-                    item_count=len(c.features),
+                    item_count=c.count,
                     items_per_page=limit
                 )
                 c.facets = query['facets']


Test patch: