Repository: open-mmlab/mmocr

Issue:
Bad Performance using PANet pre-trained models
Hi there,

I downloaded the PANet checkpoint for CTW and used them to carry out inference on CTW test set. The results that I got are significantly lower than what's reported on [this page](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/panet/README.md). I have been using all default configurations without any changes. 

The results that I got are:
CTW1500
thr 0.3, recall：0.638, precision: 0.636, hmean:0.637
thr 0.4, recall：0.638, precision: 0.636, hmean:0.637
thr 0.5, recall：0.638, precision: 0.636, hmean:0.637
thr 0.6, recall：0.638, precision: 0.636, hmean:0.637
thr 0.7, recall：0.638, precision: 0.636, hmean:0.637
thr 0.8, recall：0.638, precision: 0.636, hmean:0.637
thr 0.9, recall：0.618, precision: 0.671, hmean:0.643
{'hmean-iou:recall': 0.6182113821138211, 'hmean-iou:precision': 0.6705467372134039, 'hmean-iou:hmean': 0.64331641285956}

I suspect that there is something wrong with the released checkpoints, it seems like it is not trained properly.
Based on this [CTW training log file](https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm_sbn_600e_ctw1500_20210219-3b3a9aa3.log.json), the validation set's H-mean has already reached 0.66035 at 10th epoch, which is higher than what I get through the released checkpoint.
Could you please help me to verify this?


Discussion:
Thanks for the feedback! We have located the buggy PR (https://github.com/open-mmlab/mmocr/pull/448), which had replaced `Polygon3` with `shapely` and affected hmean-iou compuation. We are now fixing this issue. You may reset the codebase to commit `7c1bf45c63e3962bae3ed88ce0fdab967172c07b` to avoid this bug. Sorry for causing inconvenience.
Thanks for the prompt reply, I have confirmed that switching to this commit works. However, I tested PANet -IC15, FCENet -IC15 & CTW, and all of the results look normal to me except PANet -CTW, if the root cause is the library difference, then shouldn't this affects other models/datasets as well?

Also, does mmocr supports the output of results to ICDAR format (in text files)? because I can't seem to find this implementation.
We've tested the new implementation thoroughly and found that some models, including PANet, would output some "invalid" polygons that are self-touching or self-crossing, tho they make sense from appearances. These polygons are ignored by the new implementation but were not by the old one. We will release a patch to fix this discrepancy soon.

> Also, does mmocr supports the output of results to ICDAR format (in text files)? because I can't seem to find this implementation.

Currently not, but any contribution would be appreciated :)
Okay, I will see what can I do to output them in ICDAR format, thank you once again for your replies, appreciate it very much!

Resolving patch:
diff --git a/mmocr/core/evaluation/utils.py b/mmocr/core/evaluation/utils.py
--- a/mmocr/core/evaluation/utils.py
+++ b/mmocr/core/evaluation/utils.py
@@ -136,72 +136,88 @@ def points2polygon(points):
     return plg(point_mat)
 
 
-def poly_intersection(poly_det, poly_gt, invalid_ret=0, return_poly=False):
+def poly_make_valid(poly):
+    """Convert a potentially invalid polygon to a valid one by eliminating
+    self-crossing or self-touching parts.
+
+    Args:
+        poly (Polygon): A polygon needed to be converted.
+
+    Returns:
+        A valid polygon.
+    """
+    return poly if poly.is_valid else poly.buffer(0)
+
+
+def poly_intersection(poly_det, poly_gt, invalid_ret=None, return_poly=False):
     """Calculate the intersection area between two polygon.
 
     Args:
         poly_det (Polygon): A polygon predicted by detector.
         poly_gt (Polygon): A gt polygon.
-        invalid_ret (int|float): The return value when invalid polygon exists.
+        invalid_ret (None|float|int): The return value when the invalid polygon
+            exists. If it is not specified, the function allows the computation
+            to proceed with invalid polygons by cleaning the their
+            self-touching or self-crossing parts.
         return_poly (bool): Whether to return the polygon of the intersection
-                            area.
+            area.
+
     Returns:
         intersection_area (float): The intersection area between two polygons.
         poly_obj (Polygon, optional): The Polygon object of the intersection
-                                    area. Set as `None` if the input is
-                                    invalid.
+            area. Set as `None` if the input is invalid.
     """
     assert isinstance(poly_det, plg)
     assert isinstance(poly_gt, plg)
+    assert invalid_ret is None or isinstance(invalid_ret, float) or \
+        isinstance(invalid_ret, int)
+
+    if invalid_ret is None:
+        poly_det = poly_make_valid(poly_det)
+        poly_gt = poly_make_valid(poly_gt)
 
+    poly_obj = None
+    area = invalid_ret
     if poly_det.is_valid and poly_gt.is_valid:
         poly_obj = poly_det.intersection(poly_gt)
-        if return_poly:
-            return poly_obj.area, poly_obj
-        else:
-            return poly_obj.area
-    else:
-        if return_poly:
-            return invalid_ret, None
-        else:
-            return invalid_ret
+        area = poly_obj.area
+    return (area, poly_obj) if return_poly else area
 
 
-def poly_union(poly_det, poly_gt, invalid_ret=0, return_poly=False):
+def poly_union(poly_det, poly_gt, invalid_ret=None, return_poly=False):
     """Calculate the union area between two polygon.
-
     Args:
         poly_det (Polygon): A polygon predicted by detector.
         poly_gt (Polygon): A gt polygon.
-        invalid_ret (int|float): The return value when invalid polygon exists.
+        invalid_ret (None|float|int): The return value when the invalid polygon
+            exists. If it is not specified, the function allows the computation
+            to proceed with invalid polygons by cleaning the their
+            self-touching or self-crossing parts.
         return_poly (bool): Whether to return the polygon of the intersection
-                            area.
+            area.
 
     Returns:
         union_area (float): The union area between two polygons.
-        poly_obj (Polygon, optional): The polygon object of the union
-                                        area between two polygons. Set as
-                                        `None` if the input is invalid.
         poly_obj (Polygon|MultiPolygon, optional): The Polygon or MultiPolygon
-                                    object of the union of the inputs. The type
-                                    of object depends on whether they intersect
-                                    or not. Set as `None` if the input is
-                                    invalid.
+            object of the union of the inputs. The type of object depends on
+            whether they intersect or not. Set as `None` if the input is
+            invalid.
     """
     assert isinstance(poly_det, plg)
     assert isinstance(poly_gt, plg)
+    assert invalid_ret is None or isinstance(invalid_ret, float) or \
+        isinstance(invalid_ret, int)
 
+    if invalid_ret is None:
+        poly_det = poly_make_valid(poly_det)
+        poly_gt = poly_make_valid(poly_gt)
+
+    poly_obj = None
+    area = invalid_ret
     if poly_det.is_valid and poly_gt.is_valid:
         poly_obj = poly_det.union(poly_gt)
-        if return_poly:
-            return poly_obj.area, poly_obj
-        else:
-            return poly_obj.area
-    else:
-        if return_poly:
-            return invalid_ret, None
-        else:
-            return invalid_ret
+        area = poly_obj.area
+    return (area, poly_obj) if return_poly else area
 
 
 def boundary_iou(src, target, zero_division=0):


Test patch:
diff --git a/tests/test_metrics/test_eval_utils.py b/tests/test_metrics/test_eval_utils.py
--- a/tests/test_metrics/test_eval_utils.py
+++ b/tests/test_metrics/test_eval_utils.py
@@ -133,11 +133,17 @@ def test_poly_intersection():
     # test invalid polygons
     assert utils.poly_intersection(poly2, poly2) == 0
     assert utils.poly_intersection(poly3, poly3, invalid_ret=1) == 1
+    # The return value depends on the implementation of the package
+    assert utils.poly_intersection(poly3, poly3, invalid_ret=None) == 0.25
 
     # test poly return
     _, poly = utils.poly_intersection(poly, poly4, return_poly=True)
     assert isinstance(poly, Polygon)
-    _, poly = utils.poly_intersection(poly2, poly3, return_poly=True)
+    _, poly = utils.poly_intersection(
+        poly3, poly3, invalid_ret=None, return_poly=True)
+    assert isinstance(poly, Polygon)
+    _, poly = utils.poly_intersection(
+        poly2, poly3, invalid_ret=1, return_poly=True)
     assert poly is None
 
 
@@ -153,10 +159,12 @@ def test_poly_union():
     points1 = [2, 2, 2, 3, 3, 3, 3, 2]
     points2 = [0, 0, 0, 0, 0, 0, 0, 0]  # Invalid polygon
     points3 = [0, 0, 0, 1, 1, 0, 1, 1]  # Self-intersected polygon
+    points4 = [0.5, 0.5, 1, 0, 1, 1, 0.5, 0.5]
     poly = utils.points2polygon(points)
     poly1 = utils.points2polygon(points1)
     poly2 = utils.points2polygon(points2)
     poly3 = utils.points2polygon(points3)
+    poly4 = utils.points2polygon(points4)
 
     assert utils.poly_union(poly, poly1) == 2
 
@@ -167,10 +175,17 @@ def test_poly_union():
     assert utils.poly_union(poly2, poly2) == 0
     assert utils.poly_union(poly3, poly3, invalid_ret=1) == 1
 
+    # The return value depends on the implementation of the package
+    assert utils.poly_union(poly3, poly3, invalid_ret=None) == 0.25
+    assert utils.poly_union(poly2, poly3) == 0.25
+    assert utils.poly_union(poly3, poly4) == 0.5
+
     # test poly return
     _, poly = utils.poly_union(poly, poly1, return_poly=True)
     assert isinstance(poly, MultiPolygon)
-    _, poly = utils.poly_union(poly2, poly3, return_poly=True)
+    _, poly = utils.poly_union(poly3, poly3, return_poly=True)
+    assert isinstance(poly, Polygon)
+    _, poly = utils.poly_union(poly2, poly3, invalid_ret=0, return_poly=True)
     assert poly is None