Repository: ckan/ckan

Issue:
AttributeError: '_Globals' object has no attribute '__timer'
### CKAN Version if known (or site URL)

2.8.1

### What steps can be taken to reproduce the issue? 

It happens from time to time.

```
[2018-12-06 11:26:09,833] ERROR in app: Exception on /user/ [GET]
Traceback (most recent call last):
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1615, in full_dispatch_request
    return self.finalize_request(rv)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1632, in finalize_request
    response = self.process_response(response)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1856, in process_response
    response = handler(response)
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/flask_app.py", line 324, in ckan_after_request
    r_time = time.time() - g.__timer
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/flask_app.py", line 368, in __getattr__
    return getattr(app_globals.app_globals, name)
AttributeError: '_Globals' object has no attribute '__timer'
[2018-12-06 11:26:09,846] ERROR in app: Request finalizing failed with an error while handling an error
Traceback (most recent call last):
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1632, in finalize_request
    response = self.process_response(response)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1856, in process_response
    response = handler(response)
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/flask_app.py", line 324, in ckan_after_request
    r_time = time.time() - g.__timer
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/flask_app.py", line 368, in __getattr__
    return getattr(app_globals.app_globals, name)
AttributeError: '_Globals' object has no attribute '__timer'
```


Discussion:
The attributes error comes from this line, inside `ckan_after_request`:
https://github.com/ckan/ckan/blob/51e413087652a05f94d384a0f174a655c5d5b04d/ckan/config/middleware/flask_app.py#L330

The value of `g.__timer` is set in another function, `ckan_before_request`, but it is not set because there is an error before it:  `_check_uptodate` is called, which uses `get_system_info`, which tries to fetch the first value of the query, but _I suspect_ it could fail because the table is empty:
https://github.com/ckan/ckan/blob/51e413087652a05f94d384a0f174a655c5d5b04d/ckan/model/system_info.py#L55-L64

```
Traceback (most recent call last):
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1610, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1831, in preprocess_request
    rv = func()
  File "/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/flask_app.py", line 305, in ckan_before_request
    app_globals.app_globals._check_uptodate()
  File "/usr/lib/ckan/venv/src/ckan/ckan/lib/app_globals.py", line 201, in _check_uptodate
    value = model.get_system_info('ckan.config_update')
  File "/usr/lib/ckan/venv/src/ckan/ckan/model/system_info.py", line 59, in get_system_info
    obj = meta.Session.query(SystemInfo).filter_by(key=key).first()
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3222, in first
    ret = list(self[0:1])
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3012, in __getitem__
    return list(res)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3324, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3349, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: SELECT system_info.id AS system_info_id, system_info.key AS system_info_key, system_info.value AS system_info_value, system_info.state AS system_info_state, system_info.revision_id AS ystem_info_revision_id 
FROM system_info 
WHERE system_info.key = %(key_1)s 
 LIMIT %(param_1)s]
[parameters: {'key_1': 'ckan.config_update', 'param_1': 1}]
(Background on this error at: http://sqlalche.me/e/e3q8)
```

It should have been setup by `config_option_update` probably:
https://github.com/ckan/ckan/blob/51e413087652a05f94d384a0f174a655c5d5b04d/ckan/logic/action/update.py#L1234

...or somewhere using the `set_system_info`.
https://github.com/ckan/ckan/blob/51e413087652a05f94d384a0f174a655c5d5b04d/ckan/model/system_info.py#L76-L77

...and/or by using a default value (but in this case an `OperationError` seems raised, which should be handled or, even better, avoided).

Inserting the entry manually on the database does not seem to be a valid workaround:
```sql
INSERT INTO system_info (id, key, value) VALUES (0, 'ckan.config_update', 1);
```

I still do not understand why I have an `OperationError` and if it is somehow related to that specific query and why everything else seems to work as usual, especially after https://github.com/ckan/ckan/pull/4853 (I wonder if the new option as been properly applied).
I am not able to replicate that issue on my PC, only on some servers with Docker. I start to think it could be a Docker networking bug.
@frafra Have you found a solution to this? We are having the same issue on a Docker server.
@hardingalexh No, still happening. No idea why. I spent much time with some internal server error raised by CKAN and I do not know if they are related. One of them is mitigated by fetching a page from time to time actually...
Hello everyone, could they find the solution to the error?
@gasti10 I am also getting this error.

This appears to be driven by `ckan/config/middleware/flask_app.py`. I think that, as indicated by the docstring for `ckan_before_request`, `g.__timer = time.time()` is not being reached. This is probably due to `response = identify_user()`, although it is hard to confirm without any logging of that function. As such, `ckan_after_request` is always doomed to fail where `r_time = time.time() - g.__timer` tries to reference the unset `g.__timer`

```python
def ckan_before_request():
    u'''
    Common handler executed before all Flask requests

    If a response is returned by any of the functions called (
    currently ``identify_user()` only) any further processing of the
    request will be stopped and that response will be returned.

    '''
    response = None

    # Update app_globals
    app_globals.app_globals._check_uptodate()

    # Identify the user from the repoze cookie or the API header
    # Sets g.user and g.userobj
    response = identify_user()

    # Provide g.controller and g.action for backward compatibility
    # with extensions
    set_controller_and_action()

    set_ckan_current_url(request.environ)
    g.__timer = time.time()

    return response


def ckan_after_request(response):
    u'''Common handler executed after all Flask requests'''

    # Dispose of the SQLALchemy session
    model.Session.remove()

    # Check session cookie
    response = check_session_cookie(response)

    # Set CORS headers if necessary
    response = set_cors_headers_for_response(response)

    # Set Cache Control headers
    response = set_cache_control_headers_for_response(response)

    r_time = time.time() - g.__timer
    url = request.environ['PATH_INFO']

    log.info(' %s render time %.3f seconds' % (url, r_time))

    return response
```

I will try to modify the assignment of `g.__timer = time.time()` to be above `response = identify_user()`, and make a pull request if successful
can confirm that moving assignment of `g.__timer = time.time()` fixed the issue my my deployment:

```python
def ckan_before_request():
    u'''
    Common handler executed before all Flask requests

    If a response is returned by any of the functions called (
    currently ``identify_user()` only) any further processing of the
    request will be stopped and that response will be returned.

    '''
    response = None

    g.__timer = time.time()

    # Update app_globals
    app_globals.app_globals._check_uptodate()

    # Identify the user from the repoze cookie or the API header
    # Sets g.user and g.userobj
    response = identify_user()

    # Provide g.controller and g.action for backward compatibility
    # with extensions
    set_controller_and_action()

    set_ckan_current_url(request.environ)

    return response
```

I will leave this for someone with a better knowledge of this repository to action, as I am simply running from `2.9` branch. ideally this would be resolved in `master` also

Resolving patch:
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -361,6 +361,8 @@ def ckan_before_request():
     '''
     response = None
 
+    g.__timer = time.time()
+
     # Update app_globals
     app_globals.app_globals._check_uptodate()
 
@@ -373,7 +375,6 @@ def ckan_before_request():
     set_controller_and_action()
 
     set_ckan_current_url(request.environ)
-    g.__timer = time.time()
 
     return response
 


Test patch: