Repository: inducer/relate

Issue:
More accurate categorization of error type in code question?
As the python code is lazily evaluated, sometimes a problematic code from user will pass ``exec``, and the error won't be raised until it is runned in ``test_code``. That bring me tons of email. I know we can define some try-exception to prevent that, and use some code to generate the error messages, but we need to put that code in every ``test_code`` which might encounter the problem. Is it possible to create some ``Exception`` types, when which is raised, the runpy will categorize the exception to ``user_error`` instead of ``test_error`` so that won't trigger the error email?


Discussion:
I don't think that can be done in general, i.e. I think it's impossible to figure out based on the type of exception whose fault it is, or whether it's indicative of a problem in the student's or the grading code.

The policy I pursue for the code we write for our courses is that an exception in the grading code is our fault, full stop. That leads to lots of code that very carefully checks its assumptions, like [this](https://github.com/inducer/relate/blob/master/course/page/code_feedback.py#L48-L160). I think it's more promising to increase that library of "careful comparisons with good feedback" than take the approach you suggest. If you post your grading code, maybe we can figure out how to keep it short and readable while making it bulletproof.
I agree with you that it cannot be done in general. But maybe there's some condition that will help. As I am teaching some student to learn basic Python, not advanced Python which require numpy scipy or so. One of the code question is to write a simple Python function, the code follows:


```
        type: PythonCodeQuestion
        id: python_func
        access_rules:
            add_permissions:
                - change_answer
        show_setup_code: False
        value: 5
        timeout: 10
        prompt: |

            # Calculate the BMI Index

            Mr. Xiaoming's height is y meter, weight is x kg. The BMI index can be calculated by weight devided by the square of height. Please calculate his BMI index.

            Your are required to write a funtion named  ``get_bmi(height, weight)``, which returns the BMI index (a float type).

        setup_code: |
            import random
            height = random.uniform(1.5, 2)
            weight = random.uniform(0, 500)

        names_from_user: [get_bmi]

        initial_code: |

            def get_bmi(y, x):

        test_code: |

            if not callable(get_bmi):
                feedback.finish(0, "get_bmi is not a function.")

            if not isinstance(get_bmi(height, weight), float):
                feedback.finish(0, "the return value of get_bmi(height, weight) is not a float.")

            def correct_func(height, weight):
                return weight/(height**2)

            correct_bmi = correct_func(height, weight)
            user_bmi = get_bmi(height, weight)
            rel_err = abs(correct_bmi-user_bmi)/abs(correct_bmi)

            if rel_err < 1e-7:
                feedback.finish(1, The answer is correct)
            else:
                feedback.finish(1, The answer is incorrect)

```

Then some students submitted code like this:

```
def get_bmi(height, weight):
    return (weight/height^2)

```
That code passed exec, and when in the test_code part, it raised:

```
Traceback (most recent call last):
  File "/opt/runpy/code_runpy_backend.py", line 282, in run_code
    exec(test_code, maint_ctx)
  File "<test code>", line 4, in <module>
  File "<user code>", line 2, in get_bmi
TypeError: unsupported operand type(s) for ^: 'float' and 'int'
```

Then I changed the upper part of the test_code to


```
        test_code: |
            def feedback_with_traceback():
                import sys
                import traceback
                type_, value_, traceback_ = sys.exc_info()
                traceback_list = traceback.format_tb(traceback_)
                if len(traceback_list) >1:
                    traceback_list.pop(0)
                error_messages = "Error: %s" % ", ".join(["".join(traceback_list), repr(value_)])
                feedback.finish(0, error_messages)

            if not callable(get_bmi):
                feedback.finish(0, "get_bmi is not a function.")

            try:
                if not isinstance(get_bmi(height, weight), float):
                    feedback.finish(0, "the return value of get_bmi(height, weight) is not a float.")
            except:
                feedback_with_traceback()

            def correct_func(height, weight):
                return weight/(height**2)
           
           ...

```

So that the error is not recognized as a ``test_code_error``. Originally, I'm think of adding some RelateStudentCodeException, and when it is raised in the test_code part, runpy will categorize those error to ``user_error``, and I don't need to write that ``feedback_with_traceback`` in each simple python code questions
How about a

```
feedback.call_user_code(f, *args, **kwargs)
```
function that acts as a wrapper to safely call student code and that takes care of proper "assignment of blame"?
That sounds good, can you make a commit for that? Thanks.
Try abc4616. (Totally untested, needs rebuilt container.)
Great thanks for that!
The html message generated are displaying plain text instead of html as result of [this line](https://github.com/inducer/relate/blob/master/course/page/code.py#L779). I don't know if removing ``escape`` will cause other issues.

Another thing is it can be used to test functions, but also other callables, like a class. So I think the description [here](https://github.com/inducer/relate/commit/abc4616e1fb379d4c24477412ac7d885fc26da58#diff-434176dd89318c160467ff17bb3975c1R169) can be changed to ``The '%s' supplied in your code failed with``
Well, code feedback comes from an untrusted context. (I.e. it's not that hard for a student to completely take over the grading container.) That means any HTML that comes out of a context like that must be regarded as untrustworthy. (Imagine such HTML, say, with injected Javascript, being run in the grading view as part of an instructor's session--at that point, you might as well give students instructor access). `escape` is one way of making sure of that, [this](https://github.com/inducer/relate/blob/master/course/page/code.py#L845-L881) is another. I'm happy to generally allow HTML feedback.

Could you submit a PR to address these two issues?
Thanks for the detailed explanation. I'll submit a PR.

Resolving patch:
diff --git a/course/page/code.py b/course/page/code.py
--- a/course/page/code.py
+++ b/course/page/code.py
@@ -770,13 +770,16 @@ def transfer_attr(name):
             raise RuntimeError("invalid runpy result: %s" % response.result)
 
         if hasattr(response, "feedback") and response.feedback:
+            def sanitize(s):
+                import bleach
+                return bleach.clean(s, tags=["p", "pre"])
             feedback_bits.append("".join([
                 "<p>",
                 _("Here is some feedback on your code"),
                 ":"
                 "<ul>%s</ul></p>"]) %
                         "".join(
-                            "<li>%s</li>" % escape(fb_item)
+                            "<li>%s</li>" % sanitize(fb_item)
                             for fb_item in response.feedback))
         if hasattr(response, "traceback") and response.traceback:
             feedback_bits.append("".join([
diff --git a/course/page/code_feedback.py b/course/page/code_feedback.py
--- a/course/page/code_feedback.py
+++ b/course/page/code_feedback.py
@@ -166,7 +166,7 @@ def call_user(self, f, *args, **kwargs):
             from traceback import format_exc
             self.add_feedback(
                     "<p>"
-                    "The function '%s' supplied in your code failed with "
+                    "The callable '%s' supplied in your code failed with "
                     "an exception while it was being called by the grading "
                     "code:"
                     "</p>"
diff --git a/course/page/code_runpy_backend.py b/course/page/code_runpy_backend.py
--- a/course/page/code_runpy_backend.py
+++ b/course/page/code_runpy_backend.py
@@ -175,7 +175,7 @@ def run_code(result, run_req):
     if getattr(run_req, "setup_code", None):
         try:
             setup_code = compile(
-                    run_req.setup_code, "<setup code>", 'exec')
+                    run_req.setup_code, "[setup code]", 'exec')
         except:
             package_exception(result, "setup_compile_error")
             return
@@ -184,7 +184,7 @@ def run_code(result, run_req):
 
     try:
         user_code = compile(
-                run_req.user_code, "<user code>", 'exec')
+                run_req.user_code, "[user code]", 'exec')
     except:
         package_exception(result, "user_compile_error")
         return
@@ -192,7 +192,7 @@ def run_code(result, run_req):
     if getattr(run_req, "test_code", None):
         try:
             test_code = compile(
-                    run_req.test_code, "<test code>", 'exec')
+                    run_req.test_code, "[test code]", 'exec')
         except:
             package_exception(result, "test_compile_error")
             return


Test patch: