Repository: cvat-ai/cvat
Pull request: Fixed uploading track annotations for multi-segment tasks
<!---
Copyright (C) 2020 Intel Corporation

SPDX-License-Identifier: MIT
-->

## Fixed uploading track annotations for multi-segment tasks

resolve #1313 

### Motivation and context
Uploading track annotation in case of multi-segment task works incorrectly because imported tracked shapes is not filtered by frame and it leads to artifacts in the old UI and wrong dump file.
Also fixed bug with dump when track in the dump file may have no `closing` shape with `outside=True`.

### How has this been tested?
tested manually

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes that apply.
If an item isn't applicable by a reason then ~~explicitly strikethrough~~ the whole
line. If you don't do that github will show incorrect process for the pull request.
If you're unsure about any of these, don't hesitate to ask. We're here to help! -->

- [x] I have raised an issue to propose this change ([required](https://github.com/opencv/cvat/issues))
- [x] My issue has received approval from the maintainers
- [x] I've read the [CONTRIBUTION](https://github.com/opencv/cvat/blob/develop/CONTRIBUTING.md) guide
- [x] I have added description of my changes into [CHANGELOG](https://github.com/opencv/cvat/blob/develop/CHANGELOG.md) file
- [x] I have updated the [documentation](
  https://github.com/opencv/cvat/blob/develop/README.md#documentation) accordingly
- [ ] I have added tests to cover my changes
- [x] I have linked related issues (#1313 )

### License

- [x] I submit _my code changes_ under the same [MIT License](
  https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the project.
  Feel free to contact the maintainers if that's a concern.
- [x] I have updated the license header for each file (see an example below)

```python
# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT
```


Problem statement:
annotation upload corrupt for overlapping segments
i've noticed that if one has a video in CVAT which is segmented into several jobs and then uploads an annotation (containing annotations for one segment) the annotation cannot be changed and saved or deleted and saved if the annotation is bigger than the specific segment range. also the same is if there is an overlap between segments.

an example:

job 1: from 0 to 200 
job 2: from 195 to 395
job 3: from 390 to 590

so if the annotation has data for frames 195 to 395, the problem described above occurs.
also there will be a problem if one uploads an annotation for frames in the range 201 to 396.
im only able to edit and save the uploaded annotation if it is in the range of 201 to 395.

Hints from issue discussion:
i think this issue doesnt happen for every kind of annotation.
i used a single point with interpolation. fyi.

what is also interesting to explore is that i see the annotation appearing in job 1 if the object appears mostly in job 2 but overlaps with job 1 a bit. however, the interesting part is that the object in job 1 is reverted in its movement, basically it is the mirror of the movement of the object in job 2. this cannot be right.

Commit message: fixed comments

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e620d960da67..02f6a059ca06 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -53,6 +53,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - AttributeError: 'tuple' object has no attribute 'read' in ReID algorithm (https://github.com/opencv/cvat/issues/1403)
 - Wrong semi-automatic segmentation near edges of an image (https://github.com/opencv/cvat/issues/1403)
 - Git repos paths (https://github.com/opencv/cvat/pull/1400)
+- Uploading annotations for tasks with multiple jobs (https://github.com/opencv/cvat/pull/1396)
 
 ### Security
 -
diff --git a/cvat/apps/annotation/annotation.py b/cvat/apps/annotation/annotation.py
index 49ababe9ffc1..4e4305b57d27 100644
--- a/cvat/apps/annotation/annotation.py
+++ b/cvat/apps/annotation/annotation.py
@@ -77,13 +77,57 @@ def serialize(self):
         if serializer.is_valid(raise_exception=True):
             return serializer.data
 
+    @staticmethod
+    def _is_shape_inside(shape, start, stop):
+        return start <= int(shape['frame']) <= stop
+
+    @staticmethod
+    def _is_track_inside(track, start, stop):
+        # a <= b
+        def has_overlap(a, b):
+            return 0 <= min(b, stop) - max(a, start)
+
+        prev_shape = None
+        for shape in track['shapes']:
+            if prev_shape and not prev_shape['outside'] and \
+                has_overlap(prev_shape['frame'], shape['frame']):
+                    return True
+            prev_shape = shape
+
+        if not prev_shape["outside"] and prev_shape['frame'] <= stop:
+            return True
+
+        return False
+
+    @staticmethod
+    def _slice_track(track_, start, stop):
+        track = copy.deepcopy(track_)
+        segment_shapes = [s for s in track['shapes'] if AnnotationIR._is_shape_inside(s, start, stop)]
+
+        if len(segment_shapes) < len(track['shapes']):
+            interpolated_shapes = TrackManager.get_interpolated_shapes(track, start, stop)
+
+            for shape in interpolated_shapes:
+                if shape['frame'] == start and \
+                    (not segment_shapes or segment_shapes[0]['frame'] > start):
+                    segment_shapes.insert(0, shape)
+                elif shape['frame'] == stop and \
+                    (not segment_shapes or segment_shapes[-1]['frame'] < stop):
+                    segment_shapes.append(shape)
+            del track['interpolated_shapes']
+            for shape in segment_shapes:
+                del shape['keyframe']
+
+        track['shapes'] = segment_shapes
+        track['frame'] = track['shapes'][0]['frame']
+        return track
+
     #makes a data copy from specified frame interval
     def slice(self, start, stop):
-        is_frame_inside = lambda x: (start <= int(x['frame']) <= stop)
         splitted_data = AnnotationIR()
-        splitted_data.tags = copy.deepcopy(list(filter(is_frame_inside, self.tags)))
-        splitted_data.shapes = copy.deepcopy(list(filter(is_frame_inside, self.shapes)))
-        splitted_data.tracks = copy.deepcopy(list(filter(lambda y: len(list(filter(is_frame_inside, y['shapes']))), self.tracks)))
+        splitted_data.tags = [copy.deepcopy(t) for t in self.tags if self._is_shape_inside(t, start, stop)]
+        splitted_data.shapes = [copy.deepcopy(s) for s in self.shapes if self._is_shape_inside(s, start, stop)]
+        splitted_data.tracks = [self._slice_track(t, start, stop) for t in self.tracks if self._is_track_inside(t, start, stop)]
 
         return splitted_data
 
diff --git a/cvat/apps/engine/annotation.py b/cvat/apps/engine/annotation.py
index bacfd0d7e5ad..ad005bd13ec6 100644
--- a/cvat/apps/engine/annotation.py
+++ b/cvat/apps/engine/annotation.py
@@ -6,7 +6,6 @@
 from enum import Enum
 from collections import OrderedDict
 from django.utils import timezone
-from PIL import Image
 
 from django.conf import settings
 from django.db import transaction
diff --git a/cvat/apps/engine/data_manager.py b/cvat/apps/engine/data_manager.py
index b39c6783f616..8964062da943 100644
--- a/cvat/apps/engine/data_manager.py
+++ b/cvat/apps/engine/data_manager.py
@@ -290,6 +290,15 @@ def _modify_unmached_object(obj, end_frame):
             shape["frame"] = end_frame
             shape["outside"] = True
             obj["shapes"].append(shape)
+            # Need to update cached interpolated shapes
+            # because key shapes were changed
+            if "interpolated_shapes" in obj and obj["interpolated_shapes"]:
+                last_interpolated_shape = obj["interpolated_shapes"][-1]
+                for frame in range(last_interpolated_shape["frame"] + 1, end_frame):
+                    last_interpolated_shape = copy.deepcopy(last_interpolated_shape)
+                    last_interpolated_shape["frame"] = frame
+                    obj["interpolated_shapes"].append(last_interpolated_shape)
+                obj["interpolated_shapes"].append(shape)
 
     @staticmethod
     def normalize_shape(shape):


Code review comments:
diff_hunk: @@ -290,6 +290,15 @@ def _modify_unmached_object(obj, end_frame):
             shape["frame"] = end_frame
             shape["outside"] = True
             obj["shapes"].append(shape)
+            # Need to update cached interpolated shapes
+            # because key shapes were changed
+            if "interpolated_shapes" in obj and obj["interpolated_shapes"]:
path: cvat/apps/engine/data_manager.py
text: @user1:
```suggestion
            if obj.get("interpolated_shapes"):
```

diff_hunk: @@ -77,13 +77,57 @@ def serialize(self):
         if serializer.is_valid(raise_exception=True):
             return serializer.data
 
+    @staticmethod
+    def _is_shape_inside(shape, start, stop):
+        return start <= int(shape['frame']) <= stop
+
+    @staticmethod
+    def _is_track_inside(track, start, stop):
+        # a <= b
+        def has_overlap(a, b):
+            return 0 <= min(b, stop) - max(a, start)
+
+        prev_shape = None
+        for shape in track['shapes']:
+            if prev_shape and not prev_shape['outside'] and \
+                has_overlap(prev_shape['frame'], shape['frame']):
+                    return True
+            prev_shape = shape
+
+        if not prev_shape["outside"] and prev_shape['frame'] <= stop:
+            return True
+
+        return False
+
+    @staticmethod
+    def _slice_track(track_, start, stop):
+        track = copy.deepcopy(track_)
+        segment_shapes = [s for s in track['shapes'] if AnnotationIR._is_shape_inside(s, start, stop)]
+
+        if len(segment_shapes) < len(track['shapes']):
+            interpolated_shapes = TrackManager.get_interpolated_shapes(track, start, stop)
+
+            for shape in interpolated_shapes:
+                if shape['frame'] == start and \
+                    (not segment_shapes or segment_shapes[0]['frame'] > start):
path: cvat/apps/annotation/annotation.py
text: @user1:
Is it possible that here `shape` will be with outside == True? Thus we will insert the shape but it looks wrong.

@author:
added filter to remove starting shapes with outside==True after processing