Repository: hedyorg/hedy

Issue:
Allow unicode characters as letters
I've added a new UNICODE terminal to lark's common terminals which is any character outside of ASCII, and defined it to be an option for the LETTER terminal. This isn't strictly correct (unicode covers many wierd characters!) but allows variable names etc. to contain accents (see #20 ).

I've copied the common terminals file from lark to the 'grammars' directory, and now exclusively import from there - the other option was a mix of importing number-terminals from the old one and string-terminals from the new one and that sounds a terrible idea down the line. It's possible the relative import will break in some circumstances, so worth keeping an eye out for.

I've updated all the grammar files to the new path.
Correct grammar in demo code for Level 2 Ask, en/es 
["color is ask" is correct grammar, not "ask color is"]


Discussion:
Wow, thanks a lot for that extensive fix!! I will have a look at it tomorrow.
Hi! 

I have checked out your branch and ran the tests locally (still in the process of running them here on GitHub automatically) but the tests fail for me, see Stack Trace below. Have you tried running the tests? Is this something on my side? There seems to be an issue in creating the grammar from the text files.


---------
Failure
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
    method()
  File "/Users/Felienne/Code/hedy/tests.py", line 34, in test_transpile_print
    result = hedy.transpile("print Hallo welkom bij Hedy!", 1)
  File "/Users/Felienne/Code/hedy/hedy.py", line 294, in transpile
    parser = create_parser(level)
  File "/Users/Felienne/Code/hedy/hedy.py", line 81, in create_parser
    return Lark(grammar)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lark/lark.py", line 204, in __init__
    self.grammar = load_grammar(grammar, self.source)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lark/load_grammar.py", line 813, in load_grammar
    g = import_grammar(grammar_path, base_paths=base_paths)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lark/load_grammar.py", line 590, in import_grammar
    assert False
AssertionError
I have added an Action to rune the tests now on GitHub. I think it requires a fresh commit to run the tests so you might want to try that?
Superseded by https://github.com/Felienne/hedy/commit/2d4993e6b8b83de44d5504e76de286bf34ff74a9
The English version is fixed (in the meantime) by commit 026fb34.
The Spanish version is not fixed yet.

Resolving patch:
diff --git a/app.py b/app.py
--- a/app.py
+++ b/app.py
@@ -231,7 +231,6 @@ def parse():
                 response['prev_feedback_level'] = session['error_level']
                 response['prev_similar_code'] = session['similar_code']
                 session['error_level'] = 0  # Code is correct: reset error_level back to 0
-                print ('DEBUG SESSION SET GFM', session)
         except hedy.HedyException as E:
             # some 'errors' can be fixed, for these we throw an exception, but also
             # return fixed code, so it can be ran
@@ -277,7 +276,6 @@ def parse():
     return jsonify(response)
 
 def gradual_feedback_model(code, level, gradual_feedback, E, hedy_exception):
-    print ('DEBUG SESSION GFM', session)
     response = {}
     response['prev_feedback_level'] = session['error_level']
     response['prev_similar_code'] = session['similar_code']
@@ -295,7 +293,10 @@ def gradual_feedback_model(code, level, gradual_feedback, E, hedy_exception):
             else:
                 response["Feedback"] = gradual_feedback["Expanded_Unknown"]
         elif session['error_level'] == 3:  # Give a reminder what is new in this specific level
-            response["Feedback"] = gradual_feedback["New_level" + str(level)]
+            try:
+                response["Feedback"] = gradual_feedback["New_level" + str(level)]
+            except:
+                response["Feedback"] = gradual_feedback["Expanded_Uknown"]
         elif session['error_level'] == 4:
             similar_code = get_similar_code(preprocess_code_similarity_measure(code), level)
             if similar_code is None:  # No similar code is found against a, to be defined, threshold
@@ -345,19 +346,21 @@ def get_similar_code(processed_code, level):
     filename = "coursedata/level" + str(level) + ".csv"
     shortest_distance = 10 # This is the threshold: when differ more than this value it's no longer similar code
     similar_code = None
-
-    with open(filename, mode='r') as file:
-        csvFile = csv.reader(file)
-        for lines in csvFile:
-            distance = lev(processed_code, lines[2])
-            if distance == 0:  # The code is identical, no need to search any further
-                similar_code = lines[0]
-                break
-            else:
-                if distance < shortest_distance:
-                    shortest_distance = distance
+    try:
+        with open(filename, mode='r', encoding='utf-8') as file:
+            csvFile = csv.reader(file)
+            for lines in csvFile:
+                distance = lev(processed_code, lines[2])
+                if distance == 0:  # The code is identical, no need to search any further
                     similar_code = lines[0]
-        return similar_code
+                    break
+                else:
+                    if distance < shortest_distance:
+                        shortest_distance = distance
+                        similar_code = lines[0]
+    except:
+        similar_code = None
+    return similar_code
 
 @app.route('/report_error', methods=['POST'])
 def report_error():


Test patch: