Repository: lapix-ufsc/lapixdl (language: python)

Issue:
Converters module
As describe in the [milestone](https://github.com/lapix-ufsc/lapixdl/milestone/1), the converters will use a builtin format to standardize the data.

This issue is to document the converters that will be built, as well as to discuss the standard that we will adopt.


---
#### Shapely example
Personally I think it is overkill to use geopandas for this, and we can directly use shapely with pandas to handle our cases.

```python
import numpy as np
import pandas as pd
from shapely.geometry import Polygon

raw_polygons = [ [{"x":  0, "y": 0}, 
                  {"x":  10, "y": 0}, 
                  {"x":  10, "y": 10}, 
                  {"x":  5, "y": 5}],

                 [{"x":  10, "y": 10}, 
                  {"x":  110, "y": 10}, 
                  {"x":  110, "y": 110}, 
                  {"x":  15, "y": 15}]]
geometries = [Polygon(np.array([(p['x'], p['y']) for p in polygon])) 
              for polygon in raw_polygons]

df = pd.DataFrame()

df['geometry'] = geometries
df['category'] = [1, 2]
...
```


---
#### Converters list

##### From
- [x] label box
- [ ] labelme
- [ ] COCO
  - [ ] Object detection
  - [ ] Instance segmentation
  - [ ] Panoptic segmentation


#### To
- [ ] COCO
  - [x] Object detection
  - [ ] Instance segmentation
  - [ ] Panoptic segmentation
- [x] masks
- [ ] Pascal VOC 


---
#### Implementation idea

lapixdl
- convert
  - formats - To help in handle the various formats  
    - annotation - To handle and represent with annotations, for example it must contain geometry and category and can have a specific color (panoptic segmentation). [Example](https://github.com/johnnv1/CCAgT-utils/blob/840a20fca8defc7a398d5baaa0b4121e9a5c8795/CCAgT_utils/formats/annotation.py)
    - lapixdl - To load, validate, save, or manipulate the builtin format. As in this [case](https://github.com/johnnv1/CCAgT-utils/blob/840a20fca8defc7a398d5baaa0b4121e9a5c8795/CCAgT_utils/formats/CCAgT.py).
  - from_coco - code to convert the annotations/labels from coco format to builtin format
  - to_coco - code to convert the annotations/labels from builtin format to coco (OD, IS, PS)
  - ...
 

(I don't really like how this would look when used in code.)
```python
from lapixdl.converter import to_coco
from lapixdl.converter import formats

raw = formats.lapix.load("source/path/lapix.parquet")

out = to_coco.convert(raw, decimals=2)
...
```

#### Entrypoint

```shell
# From builtin format to COCO detection
lapixdl converter to_coco --target detection --source <PATH/TO/lapixdl.parquet> --destine <PATH/OUTPUT/>

# From builtin format to mask
lapixdl converter to_masks --source <PATH/TO/lapixdl.parquet> --destine <PATH/OUTPUT/>
```



---
cc: @Andrevmatias @rodrigopsr 


Discussion / hints:
The other way is as I planned for CCAgT-utils, a single module (file) for all converters, and a separate module for the formats.

```python
from lapixdl import converter
from lapixdl.formats import lapix

raw = lapix.load("source/path/lapix.parquet")

out = converter.to_coco(raw, decimals=2)
...
```
I believe the buildin format should be one of the most used standard format and work with that.
In my opinion we should start from the coco dataset format as is the most used and complete (and often derived from).
Then create a class for it contemplating all its keys-values pairs.
Doing so, we'll  need to just map any input type to coco type. (i'm doing this way in almost all my convertions)
And from coco type to any type. Avoiding i.e: yolo > pascal, pscal to yolo.   
any to coco
coco to any
I think this will make it easier to add new formats too.
And we can keep the COCO base format updated with oficial coco and in cases of major update, just make a clone of the class with the new modifications.


The problem is that COCO is not suitable for filtering or making any changes to the data. Doing this with a dataframe will be faster and simpler. And working with shapely, we can also easily do some transformations on the geometry, like point to polygon, translate (that allow to slice images and translate the geometry to the new 0), easily extract specific category image.

Fix patch:
diff --git a/lapixdl/__main__.py b/lapixdl/__main__.py
new file mode 100644
index 0000000..03c620d
--- /dev/null
+++ b/lapixdl/__main__.py
@@ -0,0 +1,7 @@
+from __future__ import annotations
+
+from lapixdl.commands.main import main
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())
diff --git a/lapixdl/commands/__init__.py b/lapixdl/commands/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lapixdl/commands/converter.py b/lapixdl/commands/converter.py
new file mode 100644
index 0000000..1a00bbf
--- /dev/null
+++ b/lapixdl/commands/converter.py
@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+import argparse
+import os
+from typing import Sequence
+
+from typing_extensions import TypeAlias
+
+from lapixdl.convert import save_lapixdf_as_masks
+from lapixdl.formats import lapix
+
+
+SUBPARSER_T: TypeAlias = 'argparse._SubParsersAction[argparse.ArgumentParser]'
+
+
+def converter_command_parser(subparsers: SUBPARSER_T | None = None) -> argparse.ArgumentParser:
+    if subparsers is not None:
+        parser = subparsers.add_parser('converter')
+    else:
+        parser = argparse.ArgumentParser(prog='lapixdl dataset converter command')
+
+    group_output = parser.add_argument_group('Define the output format desired')
+    mexg_output = group_output.add_mutually_exclusive_group(required=True)
+    mexg_output.add_argument(
+        '--to-masks',
+        action='store_true',
+        help='Convert the input data into semantic segmentation masks. This conversion is allowed from lapixdl format.',
+    )
+
+    converter_group = parser.add_argument_group('General parameters for the converters command')
+    converter_group.add_argument('-i', '--in-path', help='Path for the input file.', required=True, metavar='INPUT_PATH')
+    converter_group.add_argument('-o', '--out-path', help='Path for the output dir.', required=True, metavar='OUTPUT_DIR')
+    converter_group.add_argument('-j', '--jobs', help='Number of processes/jobs to be used.', default=1)
+
+    masks_group = parser.add_argument_group('Parameters for the conversion of masks')
+    masks_group.add_argument('--draw-order', nargs='+', type=int, default=None)
+    masks_group.add_argument('--mask-extension', type=str, default='.png')
+
+    if subparsers is not None:
+        parser.set_defaults(func=converter_command)
+
+    return parser
+
+
+def to_masks(
+    in_path: str,
+    out_path: str,
+    mask_extension: str,
+    draw_order: tuple[int, ...] | None,
+    processes: int,
+) -> int:
+
+    print(f'Loading annotations from {in_path}')
+    lapix_df = lapix.load(in_path)
+
+    print('Generating the masks...')
+    save_lapixdf_as_masks(lapix_df, out_path, mask_extension, draw_order, processes)
+
+    return 0
+
+
+def converter_command(args: argparse.Namespace | None = None) -> int:
+    if args is None:
+        return 1
+
+    # if args.to_masks:
+    draw_order = tuple(args.draw_order) if isinstance(args.draw_order, list) else None
+
+    return to_masks(
+        in_path=os.path.abspath(args.in_path),
+        out_path=os.path.abspath(args.out_path),
+        mask_extension=args.mask_extension,
+        draw_order=draw_order,
+        processes=args.jobs
+    )
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    parser = converter_command_parser()
+    args = parser.parse_args(argv)
+    return converter_command(args)
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())
diff --git a/lapixdl/commands/main.py b/lapixdl/commands/main.py
new file mode 100644
index 0000000..55b77cc
--- /dev/null
+++ b/lapixdl/commands/main.py
@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+import argparse
+import sys
+from typing import Sequence
+
+from lapixdl.commands.converter import converter_command_parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    argv = argv if argv is not None else sys.argv[1:]
+
+    parser = argparse.ArgumentParser(
+        prog='LapixDL CLI tools',
+        usage='lapixdl <command> [<args>]',
+        description='This tool will help you to convert datasets between different formats.'
+    )
+
+    subparsers = parser.add_subparsers(
+        dest='command',
+        help='lapixdl command helpers',
+    )
+
+    # Register commands
+    converter_command_parser(subparsers)
+
+    help = subparsers.add_parser(
+        'help',
+        help='Show help for a specific command.',
+    )
+    help.add_argument(
+        'help_cmd',
+        nargs='?',
+        help='Command to show help for.',
+    )
+
+    if len(argv) == 0:
+        argv = ['help']
+
+    args = parser.parse_args(argv)
+
+    if args.command == 'help' and args.help_cmd:
+        parser.parse_args([args.help_cmd, '--help'])
+    elif args.command == 'help':
+        parser.parse_args(['--help'])
+
+    if not hasattr(args, 'func'):  # pragma: no cover
+        parser.print_help()
+        return 1
+
+    return args.func(args)
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())
diff --git a/setup.cfg b/setup.cfg
index 827896d..db4b19a 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -25,9 +25,14 @@ install_requires =
     seaborn
     shapely
     tqdm
+    typing-extensions
 python_requires = >=3.7
 
 [options.packages.find]
 exclude =
     tests*
     testing*
+
+[options.entry_points]
+console_scripts =
+    lapixdl = lapixdl.commands.main:main


Test patch:
diff --git a/tests/commands/__init__.py b/tests/commands/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tests/commands/conftest.py b/tests/commands/conftest.py
new file mode 100644
index 0000000..8da82c2
--- /dev/null
+++ b/tests/commands/conftest.py
@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+import argparse
+
+import pytest
+
+
+@pytest.fixture
+def parser():
+    return argparse.ArgumentParser(prog='lapixdl tests')
+
+
+@pytest.fixture
+def subparser(parser):
+    return parser.add_subparsers(dest='command')
diff --git a/tests/commands/converter_test.py b/tests/commands/converter_test.py
new file mode 100644
index 0000000..3dc6c58
--- /dev/null
+++ b/tests/commands/converter_test.py
@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+from lapixdl.commands.converter import converter_command
+from lapixdl.commands.converter import converter_command_parser
+from lapixdl.commands.converter import main
+from lapixdl.commands.converter import to_masks
+
+
+def test_converter_command_parser(subparser):
+    parser_converter = converter_command_parser(subparser)
+    args = parser_converter.parse_args(['--to-masks', '-i', 'in-path', '-o', 'out-path'])
+
+    assert args.to_masks
+    assert len(args.in_path) > 0
+    assert len(args.out_path) > 0
+    assert hasattr(args, 'func')
+    assert parser_converter.prog == 'lapixdl tests converter'
+
+    parser_converter = converter_command_parser()
+    assert parser_converter.prog == 'lapixdl dataset converter command'
+
+
+def test_to_masks(lapix_filename, tmpdir):
+    outdir = tmpdir.mkdir('output_A/')
+    to_masks(lapix_filename, str(outdir), '.png', None, 1)
+    assert len(outdir.listdir()) == 2
+
+
+def test_converter_command(subparser, lapix_filename, tmpdir):
+    assert converter_command(None) == 1
+
+    parser = converter_command_parser(subparser)
+
+    outdir = tmpdir.mkdir('output_A/')
+    args = parser.parse_args(['--to-masks', '-i', lapix_filename, '-o', str(outdir)])
+    out = converter_command(args)
+    assert out == 0
+    assert len(outdir.listdir()) == 2
+
+
+def test_main(lapix_filename, tmpdir):
+    out = main(['--to-mask', '-i', lapix_filename, '-o', str(tmpdir)])
+    assert out == 0
diff --git a/tests/commands/main_test.py b/tests/commands/main_test.py
new file mode 100644
index 0000000..f093e6a
--- /dev/null
+++ b/tests/commands/main_test.py
@@ -0,0 +1,34 @@
+from __future__ import annotations
+
+import pytest
+
+from lapixdl.commands.main import main
+
+
+def test_main(lapix_filename, tmpdir):
+    out = main(['converter', '--to-masks', '-i', lapix_filename, '-o', str(tmpdir)])
+    assert out == 0
+
+
+def test_main_help():
+    with pytest.raises(SystemExit) as pytest_wrapped_e1:
+        main(['help'])
+    assert pytest_wrapped_e1.value.code == 0
+
+    with pytest.raises(SystemExit) as pytest_wrapped_e2:
+        main([])
+    assert pytest_wrapped_e2.value.code == 0
+
+
+def test_main_help_other_command():
+    with pytest.raises(SystemExit) as pytest_wrapped_e1:
+        main(['help', 'converter'])
+
+    assert pytest_wrapped_e1.value.code == 0
+
+
+def test_main_wrong_command():
+    with pytest.raises(SystemExit) as pytest_wrapped_e1:
+        main(['wrong-command'])
+
+    assert pytest_wrapped_e1.value.code == 2