Repository: hedyorg/hedy

Issue:
Exception found in server error log
Needs to be investigated:

```
2021-05-05T13:35:15.574265+00:00 app[web.1]: [2021-05-05 13:35:15,573] ERROR   : Exception on /parse [POST]
2021-05-05T13:35:15.574277+00:00 app[web.1]: Traceback (most recent call last):
2021-05-05T13:35:15.574278+00:00 app[web.1]:   File "/app/hedy.py", line 763, in transpile
2021-05-05T13:35:15.574278+00:00 app[web.1]:     result = transpile_inner(input_string, new_level)
2021-05-05T13:35:15.574279+00:00 app[web.1]:   File "/app/hedy.py", line 906, in transpile_inner
2021-05-05T13:35:15.574279+00:00 app[web.1]:     raise HedyException('Parse', level=level, location=["?", "?"], characters_expected="?")
2021-05-05T13:35:15.574281+00:00 app[web.1]: hedy.HedyException: Parse
2021-05-05T13:35:15.574281+00:00 app[web.1]: 
2021-05-05T13:35:15.574282+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2021-05-05T13:35:15.574282+00:00 app[web.1]: 
2021-05-05T13:35:15.574282+00:00 app[web.1]: Traceback (most recent call last):
2021-05-05T13:35:15.574283+00:00 app[web.1]:   File "/app/app.py", line 228, in parse
2021-05-05T13:35:15.574283+00:00 app[web.1]:     result = hedy.transpile(code, level)
2021-05-05T13:35:15.574283+00:00 app[web.1]:   File "/app/hedy.py", line 766, in transpile
2021-05-05T13:35:15.574284+00:00 app[web.1]:     raise E
2021-05-05T13:35:15.574285+00:00 app[web.1]:   File "/app/hedy.py", line 753, in transpile
2021-05-05T13:35:15.574285+00:00 app[web.1]:     return transpile_inner(input_string, level)
2021-05-05T13:35:15.574285+00:00 app[web.1]:   File "/app/hedy.py", line 906, in transpile_inner
2021-05-05T13:35:15.574286+00:00 app[web.1]:     raise HedyException('Parse', level=level, location=["?", "?"], characters_expected="?")
2021-05-05T13:35:15.574286+00:00 app[web.1]: hedy.HedyException: Parse
2021-05-05T13:35:15.574286+00:00 app[web.1]: 
2021-05-05T13:35:15.574287+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2021-05-05T13:35:15.574287+00:00 app[web.1]: 
2021-05-05T13:35:15.574287+00:00 app[web.1]: Traceback (most recent call last):
2021-05-05T13:35:15.574288+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 2446, in wsgi_app
2021-05-05T13:35:15.574288+00:00 app[web.1]:     response = self.full_dispatch_request()
2021-05-05T13:35:15.574289+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1951, in full_dispatch_request
2021-05-05T13:35:15.574289+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2021-05-05T13:35:15.574289+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1820, in handle_user_exception
2021-05-05T13:35:15.574290+00:00 app[web.1]:     reraise(exc_type, exc_value, tb)
2021-05-05T13:35:15.574290+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
2021-05-05T13:35:15.574291+00:00 app[web.1]:     raise value
2021-05-05T13:35:15.574291+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1949, in full_dispatch_request
2021-05-05T13:35:15.574292+00:00 app[web.1]:     rv = self.dispatch_request()
2021-05-05T13:35:15.574292+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1935, in dispatch_request
2021-05-05T13:35:15.574292+00:00 app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2021-05-05T13:35:15.574293+00:00 app[web.1]:   File "/app/app.py", line 242, in parse
2021-05-05T13:35:15.574293+00:00 app[web.1]:     response["Error"] = error_template.format(**E.arguments)
2021-05-05T13:35:15.574293+00:00 app[web.1]: KeyError: 'character_found'
2021-05-05T13:35:15.574846+00:00 app[web.1]: Traceback (most recent call last):
2021-05-05T13:35:15.574847+00:00 app[web.1]:   File "/app/hedy.py", line 763, in transpile
2021-05-05T13:35:15.574848+00:00 app[web.1]:     result = transpile_inner(input_string, new_level)
2021-05-05T13:35:15.574848+00:00 app[web.1]:   File "/app/hedy.py", line 906, in transpile_inner
2021-05-05T13:35:15.574848+00:00 app[web.1]:     raise HedyException('Parse', level=level, location=["?", "?"], characters_expected="?")
2021-05-05T13:35:15.574849+00:00 app[web.1]: hedy.HedyException: Parse
2021-05-05T13:35:15.574849+00:00 app[web.1]: 
2021-05-05T13:35:15.574850+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2021-05-05T13:35:15.574850+00:00 app[web.1]: 
2021-05-05T13:35:15.574850+00:00 app[web.1]: Traceback (most recent call last):
2021-05-05T13:35:15.574851+00:00 app[web.1]:   File "/app/app.py", line 228, in parse
2021-05-05T13:35:15.574851+00:00 app[web.1]:     result = hedy.transpile(code, level)
2021-05-05T13:35:15.574851+00:00 app[web.1]:   File "/app/hedy.py", line 766, in transpile
2021-05-05T13:35:15.574852+00:00 app[web.1]:     raise E
2021-05-05T13:35:15.574852+00:00 app[web.1]:   File "/app/hedy.py", line 753, in transpile
2021-05-05T13:35:15.574853+00:00 app[web.1]:     return transpile_inner(input_string, level)
2021-05-05T13:35:15.574853+00:00 app[web.1]:   File "/app/hedy.py", line 906, in transpile_inner
2021-05-05T13:35:15.574854+00:00 app[web.1]:     raise HedyException('Parse', level=level, location=["?", "?"], characters_expected="?")
2021-05-05T13:35:15.574854+00:00 app[web.1]: hedy.HedyException: Parse
2021-05-05T13:35:15.574854+00:00 app[web.1]: 
2021-05-05T13:35:15.574855+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2021-05-05T13:35:15.574855+00:00 app[web.1]: 
2021-05-05T13:35:15.574855+00:00 app[web.1]: Traceback (most recent call last):
2021-05-05T13:35:15.574856+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 2446, in wsgi_app
2021-05-05T13:35:15.574856+00:00 app[web.1]:     response = self.full_dispatch_request()
2021-05-05T13:35:15.574857+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1951, in full_dispatch_request
2021-05-05T13:35:15.574857+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2021-05-05T13:35:15.574857+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1820, in handle_user_exception
2021-05-05T13:35:15.574858+00:00 app[web.1]:     reraise(exc_type, exc_value, tb)
2021-05-05T13:35:15.574858+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
2021-05-05T13:35:15.574858+00:00 app[web.1]:     raise value
2021-05-05T13:35:15.574859+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1949, in full_dispatch_request
2021-05-05T13:35:15.574859+00:00 app[web.1]:     rv = self.dispatch_request()
2021-05-05T13:35:15.574860+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/flask/app.py", line 1935, in dispatch_request
2021-05-05T13:35:15.574860+00:00 app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2021-05-05T13:35:15.574861+00:00 app[web.1]:   File "/app/app.py", line 242, in parse
2021-05-05T13:35:15.574861+00:00 app[web.1]:     response["Error"] = error_template.format(**E.arguments)
2021-05-05T13:35:15.574861+00:00 app[web.1]: KeyError: 'character_found'
```


Discussion:
@rix0rrr good catch. It looks like the last exception is not a Hedy exception (which is a normal event if the code is invalid) but an actual Python uncaught exception, right?

I see that line 240-1 of `app.py` is:
```
                if 'character_found' in E.arguments:
                    E.arguments['character_found'] = hedy_errors[E.arguments['character_found']]
```

We're checking for the existence of `character_found`, so I don't know what could be the issue here.

Also, for reference, I think the current prod/beta is: a9981696553c5322aa8dbab8d4cb99bc2fc69ec6
There are 2 dictionary lookups in that line:

```
E.arguments['character_found'] 
```

Which we check for, as you said.

Then there is:

```
hedy_errors[something]

WHERE

something = E.arguments['character_found']
```

If the value of `E.arguments['character_found'] == 'character_found'` (for whatever reason), we would see the error we're seeing above.

Since the first one is unlikely to impossible, I think it has to be that?
Oh but the traceback says the error is actually in:

```
response["Error"] = error_template.format(**E.arguments)
```
That tracks, it would throw the same error:

```
>>> 'x {banaan} y'.format()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'banaan'
```

Looks like `error_template` looks like `"...something... {character_found}... more"`, but `E.arguments` does not contain a `character_found` field.
Did you happen to save the logs at the time @rix0rrr? Because it would be helpful if we saw the Hedy code that caused this. I guess it was my doing when I prettified the error messages with {character_found}



Resolving patch:
diff --git a/app.py b/app.py
--- a/app.py
+++ b/app.py
@@ -285,16 +285,16 @@ def parse():
                 response["Code"] = "# coding=utf8\n" + E.arguments['fixed_code']
                 response["Warning"] = error_template.format(**E.arguments)
             elif E.args[0] == "Parse":
-                try:
-                    error_template = hedy_errors[E.error_code]
-                    # Localize the names of characters
-                    if 'character_found' in E.arguments:
-                        E.arguments['character_found'] = hedy_errors[E.arguments['character_found']]
-                    response["Error"] = error_template.format(**E.arguments)
-                except:
-                    print('DEBUG ISSUE 375', code) #TEMP!! We need to find the programs that cause weird issue #375
-                    raise E
+                error_template = hedy_errors[E.error_code]
+                # Localize the names of characters. If we can't do that, just show the original
+                # character.
+                if 'character_found' in E.arguments.keys():
+                    E.arguments['character_found'] = hedy_errors.get(E.arguments['character_found'], E.arguments['character_found'])
+                elif 'keyword_found' in E.arguments.keys():
+                    #if we find an invalid keyword, place it in the same location in the error message but without translating
+                    E.arguments['character_found'] = E.arguments['keyword_found']
 
+                response["Error"] = error_template.format(**E.arguments)
             elif E.args[0] == "Unquoted Text":
                 error_template = hedy_errors[E.error_code]
                 response["Error"] = error_template.format(**E.arguments)
diff --git a/hedy.py b/hedy.py
--- a/hedy.py
+++ b/hedy.py
@@ -1223,11 +1223,11 @@ def transpile_inner(input_string, level, sub = 0):
     except UnexpectedCharacters as e:
         try:
             location = e.line, e.column
-            characters_expected = str(e.allowed)
+            characters_expected = str(e.allowed) #not yet in use, could be used in the future (when our parser rules are better organize, now it says ANON*__12 etc way too often!)
             character_found  = beautify_parse_error(e.args[0])
             # print(e.args[0])
             # print(location, character_found, characters_expected)
-            raise HedyException('Parse', level=level, location=location, character_found=character_found, characters_expected=characters_expected) from e
+            raise HedyException('Parse', level=level, location=location, character_found=character_found) from e
         except UnexpectedEOF:
             # this one can't be beautified (for now), so give up :)
             raise e
@@ -1257,7 +1257,7 @@ def transpile_inner(input_string, level, sub = 0):
             closest = closest_command(invalid_command, commands_per_level[level])
             if closest == None: #we couldn't find a suggestion because the command itself was found
                 # clearly the error message here should be better or it should be a different one!
-                raise HedyException('Parse', level=level, location=["?", "?"], characters_expected="?")
+                raise HedyException('Parse', level=level, location=["?", "?"], keyword_found=invalid_command)
             raise HedyException('Invalid', invalid_command=invalid_command, level=level, guessed_command=closest)
 
     is_complete = IsComplete().transform(program_root)


Test patch:
diff --git a/tests/tests_level_03.py b/tests/tests_level_03.py
--- a/tests/tests_level_03.py
+++ b/tests/tests_level_03.py
@@ -214,6 +214,20 @@ def test_transpile_missing_opening_quote(self):
 
     self.assertEqual('Unquoted Text', context.exception.args[0])  # hier moet nog we een andere foutmelding komen!
 
+  def test_transpile_issue_375(self):
+    code = textwrap.dedent("""
+      is Foobar
+      print welcome""")
+
+    with self.assertRaises(Exception) as context:
+      result = hedy.transpile(code, 3)
+
+    self.assertEqual('Parse', context.exception.args[0])  # hier moet nog we een andere foutmelding komen!
+
+
+
+
+