opened
jhamrick
username_0
This is very much a wishlist item, but it would be awesome to somehow be able to autograde markdown cells that have simple and clear solutions.
Autograding markdown cells
issue
created
oarriaga
username_1
Hello @username_0 is there any updates on this issue?
comment
created
jhamrick
username_0
@username_1 No, unfortunately -- as mentioned this is a wishlist item, which effectively means it's something we're not planning to address anytime in the near future (but if someone else wanted to implement it, it's a feature we'd be happy to have in nbgrader).
comment
created
psychemedia
username_2
Picking up on this issue, we're exploring the possibility of autograding free text answers, provided via markdown cells.

One possible architecture would be to support grading of free text markdown cells, if we could find a way to introspect on a notebook's markdown cells.

The text could either be handled by loading in a package and using it to autograde the free text, or by passing the text to a third party service API and getting a grade back: `response =  markFreeText(ANSWER)`. 

Markdown answer cells could be tagged as `studentTextAnswer` cells to simplify discovery and retrieval.

But I'm not sure a code in a notebook can easily introspect the content of markdown cells in the same notebook (at least, not directly).

A second approach might be to parse the notebook JSON, run tests on particular (markdown) cells and inject the marks back in to the notebook.

This also suggests a third, more general approach, of a marking service. My understanding is that `nbgrader` uses `nbconvert` to run the notebooks, but what if a grading service instead loaded the JSON and then processed either all on only gradeable cells  according to some sort of policy.

This requires stepping back from thinking about the notebook as a linear notebook, and instead thinking about it as a container for assessment answers.

For example, you might want to execute some cells in their own shell so there is no confusion about state. On the other hand, you might want some cells to run in the same context (perhaps identifying those cells using cell tags). Or you might want to run all the cells in the notebook in the same context (i.e. run the notebook *as a notebook*. (We could achieve this manually, for example by adding a `%%python` shell magic at the top of a cell in the first case, or using something like the [`execution_dependencies`](https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/execution_dependencies) extension in the second case.

So rather than preprocess a notebook and then execute it using `nbconvert` tools, the idea would be to load in the notebook as a JSON files then work through it, deciding in turn for each cell what to do with it.

This approach perhaps also suggests a new way of looking at the `nbgrader` marking process, perhaps in terms of a process that:

1. accepts one or more notebooks for one or more students.
2. passes them to a marking service.

The marking service then marks the notebooks and returns them as marked notebooks.

With suitable protocols defined, the marking service could be a local marking service or it could be a remote, batch execution service. So for example, I might run a central grading service, and separate `nbgrader` "clients" could then submit notebooks to that service for grading, and retrieve marked scripts from the grading service.

This would separate concerns, seeing `nbgrader` more as a co-ordinator running in a JupyterHub instance, managing release and collection of scripts, *then passing them to a separate, external service for grading*, retrieving the graded scripts and managing them locally again.

I'm not sure if the way `nbgrader` is architected already supports this sort of operation? But what it would do would open the ability to separate student notebook management from the actual assessment / grading service.
comment