internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: OCHA-DAP/hdx-ckan

Issue:
Dataset Create:  convert methodology selection to work like license selection
Not sure why I haven't noticed this before, but methodology selection would be much nicer if it worked like license selection, including the description box that appears if "Other" is chosen. 

Also, if a user types a methodology in the other box, but doesn't select "other" her choice is not saved.



Discussion:
Bumping the priority on this one as I think it was one part of a users problem with creating a dataset.  Other is a common methodology choice, and it has the bug mentioned above.

Assigned to @mbellotti as she worked on this previously and Arty is busy with Sprint 39a.

I think #1649 is related to this one.
Methodology is somehow not saving markdown for lists and when editing Other it will not display what we saved.


Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
@@ -153,6 +153,15 @@ def markdown_extract_strip(text, extract_length=190):
         '\n', ' ').replace('\r', '').replace('"', "&quot;")
     return result
 
+def render_markdown_strip(text, extract_length=190):
+    ''' return the plain text representation of markdown encoded text.  That
+    is the texted without any html tags.  If extract_length is 0 then it
+    will not be truncated.'''
+    result_text = h.render_markdown(text, extract_length)
+    result = result_text.rstrip('\n').replace(
+        '\n', ' ').replace('\r', '').replace('"', "&quot;")
+    return result
+
 
 def render_date_from_concat_str(str, separator='-'):
     result = ''
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py b/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
@@ -83,6 +83,7 @@ def get_helpers(self):
             'get_group_followers': hdx_helpers.get_group_followers,
             'get_group_members': hdx_helpers.get_group_members,
             'markdown_extract_strip': hdx_helpers.markdown_extract_strip,
+            'render_markdown_strip': hdx_helpers.render_markdown_strip,
             'render_date_from_concat_str': hdx_helpers.render_date_from_concat_str,
             'hdx_version': hdx_helpers.hdx_version,
             'hdx_build_nav_icon_with_message': hdx_helpers.hdx_build_nav_icon_with_message,


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Dataset Create:  convert methodology selection to work like license selection
Not sure why I haven't noticed this before, but methodology selection would be much nicer if it worked like license selection, including the description box that appears if "Other" is chosen. 

Also, if a user types a methodology in the other box, but doesn't select "other" her choice is not saved.

Ebola Page>Map: legend misbehaves on page zoon in chrome
Zoomed to 90%:

![image](https://cloud.githubusercontent.com/assets/1654485/5031073/c12139e8-6b54-11e4-9285-eaac8f3cbc41.png)



Discussion:
Bumping the priority on this one as I think it was one part of a users problem with creating a dataset.  Other is a common methodology choice, and it has the bug mentioned above.

Assigned to @mbellotti as she worked on this previously and Arty is busy with Sprint 39a.

I think #1649 is related to this one.
Methodology is somehow not saving markdown for lists and when editing Other it will not display what we saved.



Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
@@ -153,6 +153,15 @@ def markdown_extract_strip(text, extract_length=190):
         '\n', ' ').replace('\r', '').replace('"', "&quot;")
     return result
 
+def render_markdown_strip(text, extract_length=190):
+    ''' return the plain text representation of markdown encoded text.  That
+    is the texted without any html tags.  If extract_length is 0 then it
+    will not be truncated.'''
+    result_text = h.render_markdown(text, extract_length)
+    result = result_text.rstrip('\n').replace(
+        '\n', ' ').replace('\r', '').replace('"', "&quot;")
+    return result
+
 
 def render_date_from_concat_str(str, separator='-'):
     result = ''
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py b/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
@@ -83,6 +83,7 @@ def get_helpers(self):
             'get_group_followers': hdx_helpers.get_group_followers,
             'get_group_members': hdx_helpers.get_group_members,
             'markdown_extract_strip': hdx_helpers.markdown_extract_strip,
+            'render_markdown_strip': hdx_helpers.render_markdown_strip,
             'render_date_from_concat_str': hdx_helpers.render_date_from_concat_str,
             'hdx_version': hdx_helpers.hdx_version,
             'hdx_build_nav_icon_with_message': hdx_helpers.hdx_build_nav_icon_with_message,
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.5.0'
+hdx_version = 'v0.5.1'


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Ebola Page>Map: disable scroll wheel zoom
CJ - The specific property is here: https://github.com/OCHA-DAP/hdx-design/blob/gh-pages/js/country.js

line 111: map.scrollWheelZoom.disable();



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.5.1'
+hdx_version = 'v0.5.2'


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Dataset Create:  convert methodology selection to work like license selection
Not sure why I haven't noticed this before, but methodology selection would be much nicer if it worked like license selection, including the description box that appears if "Other" is chosen. 

Also, if a user types a methodology in the other box, but doesn't select "other" her choice is not saved.

Ebola Page>Map: disable scroll wheel zoom
CJ - The specific property is here: https://github.com/OCHA-DAP/hdx-design/blob/gh-pages/js/country.js

line 111: map.scrollWheelZoom.disable();



Discussion:
Bumping the priority on this one as I think it was one part of a users problem with creating a dataset.  Other is a common methodology choice, and it has the bug mentioned above.

Assigned to @mbellotti as she worked on this previously and Arty is busy with Sprint 39a.

I think #1649 is related to this one.
Methodology is somehow not saving markdown for lists and when editing Other it will not display what we saved.



Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
@@ -153,6 +153,15 @@ def markdown_extract_strip(text, extract_length=190):
         '\n', ' ').replace('\r', '').replace('"', "&quot;")
     return result
 
+def render_markdown_strip(text, extract_length=190):
+    ''' return the plain text representation of markdown encoded text.  That
+    is the texted without any html tags.  If extract_length is 0 then it
+    will not be truncated.'''
+    result_text = h.render_markdown(text, extract_length)
+    result = result_text.rstrip('\n').replace(
+        '\n', ' ').replace('\r', '').replace('"', "&quot;")
+    return result
+
 
 def render_date_from_concat_str(str, separator='-'):
     result = ''
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py b/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/plugin.py
@@ -83,6 +83,7 @@ def get_helpers(self):
             'get_group_followers': hdx_helpers.get_group_followers,
             'get_group_members': hdx_helpers.get_group_members,
             'markdown_extract_strip': hdx_helpers.markdown_extract_strip,
+            'render_markdown_strip': hdx_helpers.render_markdown_strip,
             'render_date_from_concat_str': hdx_helpers.render_date_from_concat_str,
             'hdx_version': hdx_helpers.hdx_version,
             'hdx_build_nav_icon_with_message': hdx_helpers.hdx_build_nav_icon_with_message,
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/version.py b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/version.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/version.py
@@ -1 +1 @@
-hdx_version = 'v0.5.0'
+hdx_version = 'v0.5.2'


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Ebola page: loading second page of datasets reloads to top of page
Would it be easy to have it load the page at the `Datasets [41]` line?



Discussion:
@cjhendrix - do we want to load only the part with datasets or we want to load the whole page and scroll down to datasets?
I know the first option is nicer, but not very easy to be done.

I don't think we want to only load a partial page.  How much effort do you think to do it by reloading the page and scrolling to the datasets?  If more than half a day, let's not do it. 
 

I think we can make it with scrolling...


Resolving patch:
diff --git a/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py b/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py
--- a/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py
+++ b/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py
@@ -54,7 +54,8 @@ def show(self):
         query = get_action("package_search")(context, data_dict)
 
         def pager_url(q=None, page=None):
-            return h.url_for('show_crisis', page=page)
+            url = h.url_for('show_crisis', page=page) + '#datasets-section'
+            return url
 
         c.page = h.Page(
             collection=query['results'],


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Search: search doesn't appear to use the organization
Example: the MapAction org has two public datasets, but searching "mapaction" or MapAction returns 0 results.  

Other org searches will return results, but this is probably because the name of the org is mentioned in other metadata. 

To do: 
1. confirm that search queries from the homepage or main search bar are not using organizations
2. if that is the source of the problem, add org to the search queries



Discussion:
@mbellotti  it might be related to https://github.com/OCHA-DAP/hdx-ckan/issues/283


Resolving patch:
diff --git a/ckanext-hdx_search/ckanext/hdx_search/plugin.py b/ckanext-hdx_search/ckanext/hdx_search/plugin.py
--- a/ckanext-hdx_search/ckanext/hdx_search/plugin.py
+++ b/ckanext-hdx_search/ckanext/hdx_search/plugin.py
@@ -1,8 +1,13 @@
-import logging
+import logging, re
 import ckan.plugins as plugins
 import ckan.plugins.toolkit as tk
 import ckan.lib.plugins as lib_plugins
 
+def convert_country(q):
+    for c in tk.get_action('group_list')({'user':'127.0.0.1'},{'all_fields': True}):
+        if re.findall(c['display_name'].lower(),q.lower()):
+            q += ' '+c['name']
+    return q
 
 class HDXSearchPlugin(plugins.SingletonPlugin):
     plugins.implements(plugins.IConfigurer, inherit=False)
@@ -31,6 +36,7 @@ def after_map(self, map):
         return map
 
     def before_search(self, search_params):
+        search_params['q'] = convert_country(search_params['q'])
         if 'facet.field' in search_params and 'vocab_Topics' not in search_params['facet.field']:
             search_params['facet.field'].append('vocab_Topics')
 


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Organization view pages result in 500 error
Only on stag.  I tested several different orgs. 

![image](https://cloud.githubusercontent.com/assets/1654485/5165739/e9046902-73e8-11e4-9358-19cd8652c0c8.png)



Discussion:

Resolving patch:
diff --git a/ckanext-hdx_search/ckanext/hdx_search/plugin.py b/ckanext-hdx_search/ckanext/hdx_search/plugin.py
--- a/ckanext-hdx_search/ckanext/hdx_search/plugin.py
+++ b/ckanext-hdx_search/ckanext/hdx_search/plugin.py
@@ -36,7 +36,7 @@ def after_map(self, map):
         return map
 
     def before_search(self, search_params):
-        search_params['q'] = convert_country(search_params['q'])
+        #search_params['q'] = convert_country(search_params['q'])
         if 'facet.field' in search_params and 'vocab_Topics' not in search_params['facet.field']:
             search_params['facet.field'].append('vocab_Topics')
 


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Datastore plugin fails to startup sometimes
We are running ckan using gunicorn.
Gunicorn run multiple threads.

At start time, datastore plugin checks the existence of db write permission by creating a table named _foo and removes it immediately afterwards. 
However, since we are starting multiple threads, it just happens that a second thread will try to create the table while it has not been removed yet (race condition? :) ).
A solution would be to change the table name with something random / depending on the time.

Please find solutions, as this is another "untouchable" piece from ckan.



Discussion:
Still not fixed in 2.2.2.
I filed an issue on ckan repo (https://github.com/ckan/ckan/issues/2082)

I submitted a pull request to ckan repo.

Requesting permission to make the changes here as well.

tagging @alexandru-m-g 


Resolving patch:
diff --git a/ckanext/datastore/plugin.py b/ckanext/datastore/plugin.py
--- a/ckanext/datastore/plugin.py
+++ b/ckanext/datastore/plugin.py
@@ -1,6 +1,9 @@
 import sys
 import logging
 
+import string
+import random
+
 import ckan.plugins as p
 import ckanext.datastore.logic.action as action
 import ckanext.datastore.logic.auth as auth
@@ -155,15 +158,19 @@ def _read_connection_has_correct_privileges(self):
         read_connection = db._get_engine(
             {'connection_url': self.read_url}).connect()
 
-        drop_foo_sql = u'DROP TABLE IF EXISTS _foo'
+        # create a nice random table name
+        chars = string.uppercase + string.lowercase
+        table_name = '_'.join(random.choice(chars) for _ in range(9))
+        
+        drop_foo_sql = u'DROP TABLE IF EXISTS ' + table_name
 
         write_connection.execute(drop_foo_sql)
 
         try:
             try:
-                write_connection.execute(u'CREATE TABLE _foo ()')
+                write_connection.execute(u'CREATE TEMP TABLE ' + table_name + ' ()')
                 for privilege in ['INSERT', 'UPDATE', 'DELETE']:
-                    test_privilege_sql = u"SELECT has_table_privilege('_foo', '{privilege}')"
+                    test_privilege_sql = u"SELECT has_table_privilege('" + table_name + "', '{privilege}')"
                     sql = test_privilege_sql.format(privilege=privilege)
                     have_privilege = read_connection.execute(sql).first()[0]
                     if have_privilege:


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
The 'Tags' field from the 'Submit Data' page should be moved up. 
Now the tags field is places within "Methodology" and "Caveats" when submitting data. I feel that this is a field that is much easier to fill than those other two. It would be probably a good idea to move it to right under visibility, as the first non-mandatory field. 

![screen shot 2014-09-18 at 1 01 20 pm](https://cloud.githubusercontent.com/assets/953118/4323926/6f08061c-3f55-11e4-8939-d030f4ea9717.png)



Discussion:
+1  @ochadataproject ?

Sure fine 


Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
@@ -168,20 +168,20 @@ def methodology_bk_compat(meth, other, render=True):
     standard_meths = ["Census","Sample Survey","Direct Observational Data/Anecdotal Data","Registry","Other"]
     if meth in standard_meths and meth != "Other":
         if render:
-            return (h.render_markdown(meth), None)
+            return (meth, None)
         else:
-            return (h.markdown_extract(meth), None)
+            return (meth, None)
     elif other:
         if render:
             return ("Other", h.render_markdown(other))
         else:
-            return ("Other", h.markdown_extract(other))
+            return ("Other", other)
     else:
         meth =  meth.split('Other - ')
         if render:
             return ("Other", h.render_markdown(meth[0]))
         else:
-            return ("Other", h.markdown_extract(meth[0]))
+            return ("Other", meth[0])
 
 
 def render_date_from_concat_str(str, separator='-'):


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
The 'Tags' field from the 'Submit Data' page should be moved up. 
Now the tags field is places within "Methodology" and "Caveats" when submitting data. I feel that this is a field that is much easier to fill than those other two. It would be probably a good idea to move it to right under visibility, as the first non-mandatory field. 

![screen shot 2014-09-18 at 1 01 20 pm](https://cloud.githubusercontent.com/assets/953118/4323926/6f08061c-3f55-11e4-8939-d030f4ea9717.png)

Investigating behaviour of crisis page on smaller browser windows
On browser windows that are smaller than the container size the map legend is cut off. We should investigate to see if there's a way to keep it fully on the screen. I think the same goes for the layer selector on the site.

I think it would be important to at least be able to scroll horizontally to the map legend. The layer selector can be reached by scrolling

Pinging @cjhendrix and @aalecs 



Discussion:
+1  @ochadataproject ?

Sure fine 



Resolving patch:
diff --git a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
--- a/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
+++ b/ckanext-hdx_theme/ckanext/hdx_theme/helpers/helpers.py
@@ -168,20 +168,20 @@ def methodology_bk_compat(meth, other, render=True):
     standard_meths = ["Census","Sample Survey","Direct Observational Data/Anecdotal Data","Registry","Other"]
     if meth in standard_meths and meth != "Other":
         if render:
-            return (h.render_markdown(meth), None)
+            return (meth, None)
         else:
-            return (h.markdown_extract(meth), None)
+            return (meth, None)
     elif other:
         if render:
             return ("Other", h.render_markdown(other))
         else:
-            return ("Other", h.markdown_extract(other))
+            return ("Other", other)
     else:
         meth =  meth.split('Other - ')
         if render:
             return ("Other", h.render_markdown(meth[0]))
         else:
-            return ("Other", h.markdown_extract(meth[0]))
+            return ("Other", meth[0])
 
 
 def render_date_from_concat_str(str, separator='-'):


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Please allow markdown to the organization description field
Right now markdown is not allowed in that field. I believe that this is preventing me from adding paragraphs and other particular styles to the text in question. 

![screen shot 2014-10-17 at 12 27 34 pm](https://cloud.githubusercontent.com/assets/953118/4682223/b9992900-561a-11e4-9cec-7f450c6909dc.png)



Discussion:
@luiscape @cjhendrix 
Does this need to be just in the edit/create form or also in the request org form ?

@alexandru-m-g I would think only in the edit / create. The request form should contain a snippet of text that describes the reasons why an organization should be created rather than a description of what the organization is. 

agreed.


Resolving patch:
diff --git a/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py b/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py
--- a/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py
+++ b/ckanext-hdx_crisis/ckanext/hdx_crisis/controllers/crisis_controller.py
@@ -46,7 +46,7 @@ def show(self):
 
         self._generate_other_links(search_term)
 
-        return render('crisis/crisis.html')
+        return render('crisis/crisis-ebola.html')
 
     def _generate_dataset_results(self, context, search_term):
         limit = 25


Test patch:
Repository: OCHA-DAP/hdx-ckan

Issue:
Datastore plugin fails to startup sometimes
We are running ckan using gunicorn.
Gunicorn run multiple threads.

At start time, datastore plugin checks the existence of db write permission by creating a table named _foo and removes it immediately afterwards. 
However, since we are starting multiple threads, it just happens that a second thread will try to create the table while it has not been removed yet (race condition? :) ).
A solution would be to change the table name with something random / depending on the time.

Please find solutions, as this is another "untouchable" piece from ckan.



Discussion:
Still not fixed in 2.2.2.
I filed an issue on ckan repo (https://github.com/ckan/ckan/issues/2082)

I submitted a pull request to ckan repo.

Requesting permission to make the changes here as well.

tagging @alexandru-m-g 

I talked with @alexandru-m-g and we settled that we will wait 1-2 weeks for #1814 and if it is not complete, we will proceed to change only the table type created by the test (temp table).


Resolving patch:
diff --git a/ckanext/datastore/plugin.py b/ckanext/datastore/plugin.py
--- a/ckanext/datastore/plugin.py
+++ b/ckanext/datastore/plugin.py
@@ -1,9 +1,6 @@
 import sys
 import logging
 
-import string
-import random
-
 import ckan.plugins as p
 import ckanext.datastore.logic.action as action
 import ckanext.datastore.logic.auth as auth
@@ -158,19 +155,15 @@ def _read_connection_has_correct_privileges(self):
         read_connection = db._get_engine(
             {'connection_url': self.read_url}).connect()
 
-        # create a nice random table name
-        chars = string.uppercase + string.lowercase
-        table_name = '_'.join(random.choice(chars) for _ in range(9))
-        
-        drop_foo_sql = u'DROP TABLE IF EXISTS ' + table_name
+        drop_foo_sql = u'DROP TABLE IF EXISTS _foo'
 
         write_connection.execute(drop_foo_sql)
 
         try:
             try:
-                write_connection.execute(u'CREATE TEMP TABLE ' + table_name + ' ()')
+                write_connection.execute(u'CREATE TABLE _foo ()')
                 for privilege in ['INSERT', 'UPDATE', 'DELETE']:
-                    test_privilege_sql = u"SELECT has_table_privilege('" + table_name + "', '{privilege}')"
+                    test_privilege_sql = u"SELECT has_table_privilege('_foo', '{privilege}')"
                     sql = test_privilege_sql.format(privilege=privilege)
                     have_privilege = read_connection.execute(sql).first()[0]
                     if have_privilege:


Test patch: