Repository: ckan/ckan

Issue:
Solr user/password not honoured for basic authentication
When using Solr with username/password basic authentication, a connection cannot be made to check if Solr is available, even when the `solr_user` and `solr_password` are set in `ckan.ini`. 

### CKAN version

Tested with 2.7.2. The culprit code looks unchanged in master.

### To reproduce

1. Create a Solr instance with basic authentication.

2. Add the following to `ckan.ini`:
```
solr_user=solr
solr_password=my_password
```

3. Restart CKAN

### Expected behaviour

Everything works ;)

### Actual behaviour

CKAN outputs the following error on startup:

```
2018-03-08 15:27:53,460 ERROR [pysolr] Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]
2018-03-08 15:27:53,460 ERROR [ckan.lib.search.common] Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]
Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/common.py", line 57, in is_available
    conn.search(q="*:*", rows=1)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 720, in search
    response = self._select(params, handler=search_handler)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 418, in _select
    return self._send_request('get', path)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pysolr.py", line 393, in _send_request
    raise SolrError(error_message % (resp.status_code, solr_message))
SolrError: Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]
2018-03-08 15:27:53,461 WARNI [ckan.lib.search] Problems were found while connecting to the SOLR server
```

Subsequent searches yield a server error:

```
2018-03-08 15:35:24,487 ERROR [pysolr] Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]
2018-03-08 15:35:24,487 ERROR [ckan.controllers.package] Dataset search error: ('SOLR returned an error running query: {\'sort\': \'score desc, metadata_modified desc\', \'fq\': [u\'+dataset_type:dataset\', u\'+site_id:"default"\', \'+state:active\'], \'facet.mincount\': 1, \'rows\': 21, \'facet.field\': [u\'organization\', u\'groups\', u\'tags\', u\'res_format\', u\'license_id\'], \'facet.limit\': \'50\', \'mm\': \'2<-1 5<80%\', \'facet\': \'true\', \'q\': u\'test\', \'start\': 0, \'wt\': \'json\', \'qf\': \'name^4 title^4 tags^2 groups^2 text\', \'tie\': \'0.1\', \'fl\': \'id validated_data_dict\', \'defType\': \'dismax\'} Error: SolrError(u\'Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]\',)',)
2018-03-08 15:35:24,492 ERROR [pysolr] Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]
Error - <class 'ckan.lib.search.common.SearchError'>: SOLR returned an error running query: {'fq': [u'+site_id:"default"', '+state:active'], 'facet.mincount': 1, 'rows': 2, 'facet.field': ['groups', 'owner_org'], 'facet': 'true', 'q': '+capacity:public', 'facet.limit': -1, 'wt': 'json', 'fl': 'groups'} Error: SolrError(u'Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]',)
URL: http://docker.for.mac.localhost:5000/dataset?q=test
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
  app_iter = self.application(environ, sr_checker)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
  return request.get_response(self.app)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
  response = request.get_response(self.app)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/pylons_app.py', line 250 in inner
  result = application(environ, start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/beaker/middleware.py', line 156 in __call__
  return self.wrap_app(environ, session_start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/common_middleware.py', line 80 in __call__
  return self.app(environ, start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 212 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/usr/lib/ckan/default/src/ckan/ckan/controllers/package.py', line 310 in search
  package_type=package_type)
File '/usr/lib/ckan/default/src/ckan/ckan/controllers/package.py', line 72 in _setup_template_variables
  setup_template_variables(context, data_dict)
File '/usr/lib/ckan/default/src/ckan/ckan/lib/plugins.py', line 286 in setup_template_variables
  c.groups_authz = authz_fn(context, data_dict)
File '/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py', line 457 in wrapped
  result = _action(context, data_dict, **kw)
File '/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py', line 592 in group_list_authz
  group_list = model_dictize.group_list_dictize(groups, context)
File '/usr/lib/ckan/default/src/ckan/ckan/lib/dictization/model_dictize.py', line 51 in group_list_dictize
  group_dictize_context['dataset_counts'] = get_group_dataset_counts()
File '/usr/lib/ckan/default/src/ckan/ckan/lib/dictization/model_dictize.py', line 337 in get_group_dataset_counts
  query.run(q)
File '/usr/lib/ckan/default/src/ckan/ckan/lib/search/query.py', line 372 in run
  (query, e))
SearchError: SOLR returned an error running query: {'fq': [u'+site_id:"default"', '+state:active'], 'facet.mincount': 1, 'rows': 2, 'facet.field': ['groups', 'owner_org'], 'facet': 'true', 'q': '+capacity:public', 'facet.limit': -1, 'wt': 'json', 'fl': 'groups'} Error: SolrError(u'Solr responded with an error (HTTP 401): [Reason: Error 401 require authentication]',)
```

Both of these errors appear to be caused by `make_connection()` in [lib/search/common.py](https://github.com/ckan/ckan/blob/master/ckan/lib/search/common.py#L66), where the `solr_user` and `solr_password` settings are not honoured when the connection is established.

### Attempted workaround

Because I love a dirty hack, I tried dropping the username & password into the solr_url (e.g. https://user:pass@solr:8983/solr/ckan). This yielded a much more obscure error, because `urllib2` appears not to handle user/pass in the URL:

```
Traceback (most recent call last):
  File "/usr/local/bin/ckan-paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 333, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 306, in _load_config
    self.site_user = load_config(self.options.config, load_site_user)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 221, in load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 99, in load_environment
    p.load_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 129, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 182, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 210, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 207, in update_config
    search.check_solr_schema_version()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 291, in check_solr_schema_version
    res = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
```

### Proposed solution

Updating `make_connection` to create the connection with the username/password resolves this issue.



Discussion:

Resolving patch:
diff --git a/ckan/lib/search/common.py b/ckan/lib/search/common.py
--- a/ckan/lib/search/common.py
+++ b/ckan/lib/search/common.py
@@ -6,6 +6,7 @@
 import pysolr
 import simplejson
 from six import string_types
+import urllib
 
 log = logging.getLogger(__name__)
 
@@ -65,6 +66,16 @@ def is_available():
 
 def make_connection(decode_dates=True):
     solr_url, solr_user, solr_password = SolrSettings.get()
+
+    if solr_url and solr_user:
+        # Rebuild the URL with the username/password
+        protocol = re.search('http(?:s)?://', solr_url).group()
+        solr_url = re.sub(protocol, '', solr_url)
+        solr_url = "{}{}:{}@{}".format(protocol,
+                                       urllib.quote_plus(solr_user),
+                                       urllib.quote_plus(solr_password),
+                                       solr_url)
+
     if decode_dates:
         decoder = simplejson.JSONDecoder(object_hook=solr_datetime_decoder)
         return pysolr.Solr(solr_url, decoder=decoder)


Test patch: