Repository: snowblink14/smatch

Issue:
interpretation of TOP property
when comparing SMATCH results with the scorer from the [2019 CoNLL Shared Task](http://mrp.nlpl.eu) on Cross-Framework Meaning Representation Parsing (MRP), we discovered that SMATCH will only consider the TOP property correct if the node labels also match.  this appears to double-penalize for label mismatches and is maybe not the intended behavior?  for more technical detail and a minimal test case, please see the MRP [`mtool` issue](https://github.com/cfmrp/mtool/issues/12#issue-456527664).

Comments:

Patch:
diff --git a/.travis.yml b/.travis.yml
index 50980a8..1aeed33 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -9,7 +9,7 @@ deploy:
   provider: pypi
   user: __token__
   password:
-    secure: CVA8RIEx1IdOHUBMjF/MPd6FFE3wu0sAzJkowu0PzZ4VwbOlqSYXuYWjkNUV8plCVm8mgovMXBDjTC8q9AGYTJi8B5f92AY6YRfLVjJCpdMd8EH6VNlymxJTYg0t5W5RKpzdAyOLj5GyAPhVqY805TIE+ao2XKQ+UwlwRUq/SUhxi6gcvLsamuabfKg2OZxijKp6dPk+tqw33K+DYJVV3WOUqyI169Z3iXRhljghmLWiV6xdIs1/V34XsbdgRJDRs/kOsrhE19khOIUqZ/+to++qRFwLpgAsF71n23vGs/FwvP+ab1oMYtUNC8DHI2gmiGHO/ipE/FXqsYdYuhULnzs7nfx04YQoaZliD7Hbvze0zEczQZxHQpuHbykNND2WU20NOnckBzDgqhqzPIVjHwQTGFKMZqg7nZ8w81NrGSeGTXagtoBPtq6/RPU1alppFE4EU7/fZrAN/fXgiemcxmsK5Nl9Ps6zWoB2eFhCaLDeUI9BktIrp5nLplLQJmLfF+RfLaIYULD5hdwFzgcUTjG/nPr41XNMvwYvsM+rCGxlxBtL2Gc2xf7Kfv9T6vUr0eb0Yxp+oKNv7XN3j1rZ+0PIqMbi1dpMJu7OqxIR5lQB6bYwDcCeKxrFpnP/9drtXzjqGZnzCt/rdGnMUd9PrkZTHFXOrsq0ZtmFSNK5/m4=
+    secure: jXgeZ3fjIODIltATnt487VOBf6TIOp9bpd79r/wUG373pa6GVrWKEbX1LBAprmSW7dI+pYccQpeNF5OIsUJFFNjy59iLlKtfY+9N8ugD7rvuQL4pP+jwiVvuKB9SLyLwzJWrArsmVOju9JQUogtKXPJiflY1PE4fGV/Zjx+FNl6816HO8i2uHaLOBzDbNXtOCnAmioaM72cQ/qvtbKdkxF7u7xtAwqA8TEBeaf6PYR2b1mn8Vcgo7pG/uJGZtICPVINxv914jDpu0CDcWl3rmtVM1Q44XxKLV3c+6QUeonTy5s+35plDiWnI0F0HJiGfS51NYvUUaZZQKuNbIKg27dJcfhuzZDarD4C/P6fjN8NSdmB/B4P058pBkUTmmaeaLBDENBOlJAwRwHupRvsnWZ9xazw1edEVo2tnFPhAZJ+BBDv9lISiNg2kiQx6+5/BbnecjS4lyz3Kje+QljocDXHwshsSl8FxnmJ6TWI21yJJauKM4LGxHGyCrz0W1AsYw+S/cwppC1/BzS9ZLW5DdUeVzULGKzY1MqZJ7ABt70MLBMmrV5VjpUjtTS9dqVBMSvCxshR2hOM47x4wEzG9yQec4aDvHImEPOMXKHkzk6fVhm5qO9IxJTPldYrLcR/UFpcFdHKelIReqTQsjzaEI7CMCicux/DDpRt9msrCeQY=
   on:
     tags: true
   skip_existing: true
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 96c99a1..24cd58a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,9 @@
 * Add CI/CD configuration (see PR [#23][])
 * Better handle deinversion of special roles ([#10][])
 * Get `smatch-table.py` working again (part of PR [#27][])
+* Add `tests/` subdirectory and `test_top.py` ([#25])
+* Make TOP relation's value a constant string and not the top node's
+  concept to avoid double-penalizing different top concepts ([#25])
 
 ## [1.0.2][]
 
@@ -166,5 +169,6 @@ The following are taken from an old `update_log` file:
 [#19]: https://github.com/snowblink14/smatch/issues/19
 [#22]: https://github.com/snowblink14/smatch/issues/22
 [#23]: https://github.com/snowblink14/smatch/pull/23
+[#25]: https://github.com/snowblink14/smatch/pull/25
 [#27]: https://github.com/snowblink14/smatch/pull/27
 
diff --git a/amr.py b/amr.py
index 020f52e..182b6b5 100755
--- a/amr.py
+++ b/amr.py
@@ -420,8 +420,8 @@ class AMR(object):
             # each node has a relation list and attribute list
             relation_list.append(node_rel_list)
             attribute_list.append(node_attr_list)
-        # add TOP as an attribute. The attribute value is the top node value
-        attribute_list[0].append(["TOP", node_value_list[0]])
+        # add TOP as an attribute. The attribute value just needs to be constant
+        attribute_list[0].append(["TOP", 'top'])
         result_amr = AMR(node_name_list, node_value_list, relation_list, attribute_list)
         return result_amr
 


Test patch:
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 0000000..f8c1899
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,18 @@
+
+# smatch tests
+
+This directory contains test cases to verify the correct behavior of
+smatch. Run the tests with [pytest](https://pytest.org/). This will
+require you to install both pytest and smatch:
+
+```console
+$ pip install pytest
+$ pip install -e .  # current directory is smatch
+$ pytest
+```
+
+**Note:** As smatch is inherently non-deterministic due to its
+hill-climbing implementation, the tests can be "flaky" (i.e.,
+sometimes pass, sometimes fail). To mitigate the possibility of flaky
+tests, test cases should use *minimal* AMRs so it becomes trivial for
+smatch to get the optimal solution.
diff --git a/tests/test_top.py b/tests/test_top.py
new file mode 100644
index 0000000..a9bf4c4
--- /dev/null
+++ b/tests/test_top.py
@@ -0,0 +1,57 @@
+
+"""
+The top node in an AMR determines the focus of the encoded
+meaning. For example, the following have the same dependencies but a
+different node at the top:
+
+    (w / white-03
+       :ARG1 (m / marble))
+
+The above AMR means "the marble is white" or "the whiteness of the
+marble".
+
+    (m / marble
+       :ARG1-of (w / white-03))
+
+This AMR means "the marble that is white" or "the white marble".
+
+For this reason, AMRs that differ only in which node is the top will
+get smatch scores less than 1.0.
+
+For more information see:
+
+    https://github.com/amrisi/amr-guidelines/blob/master/amr.md#focus
+"""
+
+import smatch
+
+a = '(a / alpha :ARG0 (b / beta))'
+b = '(a / alternative :ARG0 (b / beta))'
+c = '(a / alpha :ARG0 (b / b-side))'
+d = '(b / beta :ARG0-of (a / alpha))'
+
+
+def get_amr_match(amr1, amr2):
+    vals = smatch.get_amr_match(amr1, amr2)
+    smatch.match_triple_dict.clear()
+    return vals
+
+
+def test_same():
+    assert get_amr_match(a, a) == (4, 4, 4)
+    smatch.match_triple_dict.clear()
+
+
+def test_same_top_different_top_concept():
+    assert get_amr_match(a, b) == (3, 4, 4)
+    smatch.match_triple_dict.clear()
+
+
+def test_same_top_different_dependent_concept():
+    assert get_amr_match(a, c) == (3, 4, 4)
+    smatch.match_triple_dict.clear()
+
+
+def test_same_different_top():
+    assert get_amr_match(a, d) == (3, 4, 4)
+    smatch.match_triple_dict.clear()