Repository: openvinotoolkit/datumaro

Issue:
cvat_format/extractor type casting value to float, causing issues
We have a text type attribute for a bounding box label in CVAT. The values for this attribute are all digits some of which start with zeros. Ex: 0012345, 0123456, 1234567, etc.

We are using datumaro to automatically do some checks on the annotations imported from CVAT. It looks like the extractor for CVAT is trying to typecast all imported numeric [attribute values to float](https://github.com/openvinotoolkit/datumaro/blob/6a2022ed467c7557fb2055c143452bc4ca7f6e86/datumaro/plugins/cvat_format/extractor.py#L96). This is causing an issue for us since this removes the leading zeros automatically from the value, changing user input.

Is there any reason behind typecasting the the attribute value to float? Can this typecasting step be deleted. If required a user can always do the typecasting later. 


Discussion:
@kirill-sizov , could you please look at the issue?
> @kirill-sizov , could you please look at the issue?

@nmanovic Yes, sure

> Is there any reason behind typecasting the the attribute value to float? 

Datumaro does not have strong attribute data model yet, where value ranges and data types could be described, it should be resolved with #144. So the way is how it is implemented now is our best effort in preserving the attribute data types - logical values, numbers and strings. We tried to have everything as string values, but it didn't work very well.

> Can this typecasting step be deleted. If required a user can always do the typecasting later.

I think yes, especially in case of CVAT format. It has attribute type information, which can be utilized even without big changes.


BTW, could you share any details about your dataset checks? Are you using some custom code for this, which maybe can be shared? Is it possible that Datumaro does it for you with `merge`, `stats` and `validate` commands or any other?
> could you share any details about your dataset checks?

We are doing simple checks on user input like string length (we know this attribute should have 7 digits). Which is failing because because of leading zeros being removed. All failed checks are exported in CSV format, which we use as a reference to make changes in CVAT. 

CVAT has 2 attribute types: `number ` and `text` for undefined user inputs. However, while importing it looks like both the types are being typecast to float without checking the input_type in CVAT xml currently.  So, if a `text` type  has only digits it is also being typecast into a float. 

>  Is it possible that Datumaro does it for you with `merge`,` stats` and `validate` commands or any other?

We have not used `stats` or `validate` because we could not find examples for these commands. I think it would not matter if we did since, these commands will be executed after the dataset has already been extracted from CVAT xml, by which time data has already been typecast.

> Are you using some custom code for this, which maybe can be shared? 

We are iterating through each dataset_element->label->attribute to carry out checks. 

```
for item in dataset:
    for box in item.annotations:

        # being forced to typecast the attribute value back to string
        #to check length
        if (len(str(box.attributes['text_test'])) !=7):
            print ("Error: Length of user input not 7.")
```

The above snippet will print an error if the attribute is defined as:
```
<attribute>
   <name>text_test</name>
   <mutable>False</mutable>
   <input_type>text</input_type>
   <default_value></default_value>
   <values></values>
</attribute>
```
and value is defined as `0203450`  by  `<attribute name="text">0203450</attribute>`.  Datumaro is converting this `0203450` to `203450.0` .


@zhiltsov-max , Datumaro should not convert text to a number. 

Resolving patch:
diff --git a/datumaro/plugins/cvat_format/extractor.py b/datumaro/plugins/cvat_format/extractor.py
--- a/datumaro/plugins/cvat_format/extractor.py
+++ b/datumaro/plugins/cvat_format/extractor.py
@@ -41,7 +41,7 @@ def _parse(cls, path):
         context = ElementTree.iterparse(path, events=("start", "end"))
         context = iter(context)
 
-        categories, frame_size = cls._parse_meta(context)
+        categories, frame_size, attribute_types = cls._parse_meta(context)
 
         items = OrderedDict()
 
@@ -89,9 +89,10 @@ def _parse(cls, path):
             elif ev == 'end':
                 if el.tag == 'attribute' and attributes is not None:
                     attr_value = el.text or ''
+                    attr_type = attribute_types.get(el.attrib['name'])
                     if el.text in ['true', 'false']:
                         attr_value = attr_value == 'true'
-                    else:
+                    elif  attr_type is not None and attr_type != 'text':
                         try:
                             attr_value = float(attr_value)
                         except ValueError:
@@ -189,11 +190,12 @@ def consumed(expected_state, tag):
                 elif accepted('original_size', 'width', next_state='frame_width'): pass
                 elif accepted('task', 'labels'): pass
                 elif accepted('labels', 'label'):
-                    label = { 'name': None, 'attributes': set() }
+                    label = { 'name': None, 'attributes': [] }
                 elif accepted('label', 'name', next_state='label_name'): pass
                 elif accepted('label', 'attributes'): pass
                 elif accepted('attributes', 'attribute'): pass
                 elif accepted('attribute', 'name', next_state='attr_name'): pass
+                elif accepted('attribute', 'input_type', next_state='attr_type'): pass
                 elif accepted('annotations', 'image') or \
                      accepted('annotations', 'track') or \
                      accepted('annotations', 'tag'):
@@ -214,7 +216,9 @@ def consumed(expected_state, tag):
                 elif consumed('label_name', 'name'):
                     label['name'] = el.text
                 elif consumed('attr_name', 'name'):
-                    label['attributes'].add(el.text)
+                    label['attributes'].append({'name': el.text})
+                elif consumed('attr_type', 'input_type'):
+                    label['attributes'][-1]['input_type'] = el.text
                 elif consumed('attribute', 'attribute'): pass
                 elif consumed('attributes', 'attributes'): pass
                 elif consumed('label', 'label'):
@@ -234,12 +238,15 @@ def consumed(expected_state, tag):
             common_attrs.append('track_id')
 
         label_cat = LabelCategories(attributes=common_attrs)
+        attribute_types = {}
         for label, attrs in labels.items():
-            label_cat.add(label, attributes=attrs)
+            attr_names = {v['name'] for v in attrs}
+            label_cat.add(label, attributes=attr_names)
+            for attr in attrs:
+                attribute_types[attr['name']] = attr['input_type']
 
         categories[AnnotationType.label] = label_cat
-
-        return categories, frame_size
+        return categories, frame_size, attribute_types
 
     @classmethod
     def _parse_shape_ann(cls, ann, categories):


Test patch:
diff --git a/tests/assets/cvat_dataset/for_images/train.xml b/tests/assets/cvat_dataset/for_images/train.xml
--- a/tests/assets/cvat_dataset/for_images/train.xml
+++ b/tests/assets/cvat_dataset/for_images/train.xml
@@ -12,16 +12,25 @@
               <name>a1</name>
               <input_type>checkbox</input_type>
               <default_value>false</default_value>
-              <values>false
-true</values>
+              <values>false true</values>
             </attribute>
             <attribute>
               <name>a2</name>
               <input_type>radio</input_type>
               <default_value>v1</default_value>
-              <values>v1
-v2
-v3</values>
+              <values>v1 v2 v3</values>
+            </attribute>
+            <attribute>
+              <name>a3</name>
+              <input_type>text</input_type>
+              <default_value>0000</default_value>
+              <values>0001 0002 0003</values>
+            </attribute>
+            <attribute>
+              <name>a4</name>
+              <input_type>number</input_type>
+              <default_value>0</default_value>
+              <values>2.4 1 -5.6</values>
             </attribute>
           </attributes>
         </label>
@@ -35,6 +44,8 @@ v3</values>
     <box label="label1" occluded="1" xbr="4" xtl="0" ybr="4" ytl="2" z_order="1">
       <attribute name="a1">true</attribute>
       <attribute name="a2">v3</attribute>
+      <attribute name="a3">0003</attribute>
+      <attribute name="a4">2.4</attribute>
     </box>
     <polyline label="" points="1.0,2;3,4;5,6;7,8"/>
   </image>
diff --git a/tests/test_cvat_format.py b/tests/test_cvat_format.py
--- a/tests/test_cvat_format.py
+++ b/tests/test_cvat_format.py
@@ -43,7 +43,7 @@ def test_can_load_image(self):
                     Bbox(0, 2, 4, 2, label=0, z_order=1,
                         attributes={
                             'occluded': True,
-                            'a1': True, 'a2': 'v3'
+                            'a1': True, 'a2': 'v3', 'a3': '0003', 'a4': 2.4,
                         }),
                     PolyLine([1, 2, 3, 4, 5, 6, 7, 8],
                         attributes={'occluded': False}),
@@ -58,7 +58,7 @@ def test_can_load_image(self):
                 ], attributes={'frame': 1}),
         ], categories={
             AnnotationType.label: LabelCategories.from_iterable([
-                ['label1', '', {'a1', 'a2'}],
+                ['label1', '', {'a1', 'a2', 'a3', 'a4'}],
                 ['label2'],
             ])
         })
@@ -211,9 +211,9 @@ def test_can_save_and_load(self):
                     Points([1, 1, 3, 2, 2, 3],
                         label=2,
                         attributes={ 'occluded': False, 'empty': '',
-                            'a1': 'x', 'a2': 42 }),
+                            'a1': 'x', 'a2': '42' }),
                     Label(1),
-                    Label(2, attributes={ 'a1': 'y', 'a2': 44 }),
+                    Label(2, attributes={ 'a1': 'y', 'a2': '44' }),
                 ], attributes={'frame': 0}
             ),
             DatasetItem(id=1, subset='s1',
@@ -258,9 +258,9 @@ def test_can_allow_undeclared_attrs(self):
         target_label_cat.add('a', attributes={'x'})
         target_dataset = Dataset.from_iterable([
             DatasetItem(id=0, annotations=[
-                Label(0, attributes={ 'x': 4, 'y': 2 }),
+                Label(0, attributes={ 'x': '4', 'y': '2' }),
                 Bbox(1, 2, 3, 4, label=0,
-                    attributes={ 'x': 1, 'y': 1, 'occluded': False }),
+                    attributes={ 'x': '1', 'y': '1', 'occluded': False }),
             ], attributes={'frame': 0}),
         ], categories={ AnnotationType.label: target_label_cat })