Repository: ckan/ckan

Issue:
Support for pdb and debuggers
### Proposed fixes:

It is now possible to debug ckan with pdb/ipdb/PyCharm debugger and others, both outside Docker and inside Docker.
I just exposed a `werkzeug` option to the CKAN CLI, called `passthrough_errors`. Enabling that, together with `--disable-reloader` (which should be the default in my opinion, like it was in the past), allow to run pdb without making other changes to the source code.
`threads` should not be enabled and `processes` should be set to 1. These are the defaults already.

> passthrough_errors (bool) – set this to True to disable the error catching. This means that the server will die on errors but it can be useful to hook debuggers in (pdb etc.)
-- https://werkzeug.palletsprojects.com/en/2.0.x/serving/

Example:
```
$ cd contrib/docker
$ docker-compose up --build -d
$ # wait...
$ docker-compose exec ckan bash
root@f6a71d0b7686:/# python3 -m pdb /usr/lib/ckan/venv/bin/ckan -c /etc/ckan/production.ini run --host 0.0.0.0 -E --disable-reloader
> /usr/lib/ckan/venv/bin/ckan(3)<module>()
-> import re
(Pdb) b ckan/views/api.py:215
Breakpoint 1 at /usr/lib/ckan/venv/src/ckan/ckan/views/api.py:215
(Pdb) c
2021-11-01 17:00:50,832 INFO  [ckan.cli] Using configuration file /etc/ckan/production.ini
2021-11-01 17:00:50,832 INFO  [ckan.config.environment] Loading static files from public
2021-11-01 17:00:50,954 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/venv/src/ckan/ckan/templates
2021-11-01 17:00:51,552 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/venv/src/ckan/ckan/templates
2021-11-01 17:00:52,173 INFO  [ckan.cli.server] Running CKAN on http://0.0.0.0:5000
2021-11-01 17:00:52,174 WARNI [werkzeug]  * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
```

...then `http://localhost:5000/api/3/action/package_search` can be opened in the web browser to trigger the breakpoint:

```
> /usr/lib/ckan/venv/src/ckan/ckan/views/api.py(215)action()
-> try:
(Pdb)
```

### Features:

- [ ] includes tests covering changes
- [x] includes updated documentation
- [X] includes user-visible changes
- [ ] includes API changes
- [ ] includes bugfix for possible backport


Discussion:
It could be improved by having a `--debug` option which is mutually exclusive with everything else, but it would require a bigger change, and it would clash with the current nomenclature (debugging on CKAN today is mostly done on the web server, which is very limiting).
Any comment on this PR?
@frafra can you give an example on how this new option is used (`passthrough_errors`) or why it is useful?
> @frafra can you give an example on how this new option is used (`passthrough_errors`) or why it is useful?

Sure. I thought that it was that werkzeug documentation was good enough to explain what such option does.

It is not possible to debug CKAN with regular Python tools and IDE if the server is executed with certain options: automatic reload, multithread/multiprocessing, and error catching. This last behaviour cannot be controlled from CKAN CLI, so I exposed that via the `--passthrough_error` option. Enabling such options disables werkzeug error-catching, which hides all the errors from the user, by catching them, and allows it to just fail.

As reported initially, from the werkzeug documentation:

> passthrough_errors (bool) – set this to True to disable the error catching. This means that the server will die on errors but it can be useful to hook debuggers in (pdb etc.)

If CKAN CLI is executed using `--disable-reloader --passthrough_error`, and if `ckan.devserver.threaded` is set to `False` (default) and `ckan.devserver.multiprocess` is set to 1 (default), then CKAN can be debugged with normal tools, like pdb, ipdb, IDEs and such.

To make things even easier to set up (especially for IDEs), there is a draft PR open which I will propose as soon as this one gets merged: https://github.com/ckan/ckan/pull/6458
It could be interesting to have it as environmental variable, like the other parameters: http://docs.ckan.org/en/2.9/maintaining/configuration.html
Any feedback on that?
I have been using this patch regularly for the last month combined with Docker (I run CKAN with these special options) and it made the development of CKAN extensions so much easier. I catch all the exceptions, I am able to run step by step, set breakpoints, list the source code, skip a line, print a value, execute a statement just once, and such.

Is there anything else I should do to see this option merged? Write some documentation, maybe?
Meeting notes 10/02/2020: the new option should override the other options which it depends on; documentation should be also added.

Resolving patch:
diff --git a/ckan/cli/server.py b/ckan/cli/server.py
--- a/ckan/cli/server.py
+++ b/ckan/cli/server.py
@@ -19,6 +19,8 @@
 @click.option(u"-p", u"--port", help=u"Port number")
 @click.option(u"-r", u"--disable-reloader", is_flag=True,
               help=u"Disable reloader")
+@click.option(u"-E", u"--passthrough-errors", is_flag=True,
+              help=u"Disable error caching (useful to hook debuggers)")
 @click.option(
     u"-t", u"--threaded", is_flag=True,
     help=u"Handle each request in a separate thread"
@@ -40,10 +42,16 @@
     help=u"Key file to use to enable SSL. Passing 'adhoc' will "
     " automatically generate a new one (on each server reload).")
 @click.pass_context
-def run(ctx, host, port, disable_reloader, threaded, extra_files, processes,
-        ssl_cert, ssl_key):
+def run(ctx, host, port, disable_reloader, passthrough_errors, threaded,
+        extra_files, processes, ssl_cert, ssl_key):
     u"""Runs the Werkzeug development server"""
 
+    # passthrough_errors overrides conflicting options
+    if passthrough_errors:
+        disable_reloader = True
+        threaded = False
+        processes = 1
+
     # Reloading
     use_reloader = not disable_reloader
     config_extra_files = tk.aslist(
@@ -95,4 +103,5 @@ def run(ctx, host, port, disable_reloader, threaded, extra_files, processes,
         processes=processes,
         extra_files=extra_files,
         ssl_context=ssl_context,
+        passthrough_errors=passthrough_errors,
     )


Test patch: