Repository: dtrx-py/dtrx (language: python)

Issue:
Official list of supported file types
Hi,

we are using dtrx as part of larger scripts to unroll / extract huge bulks of files. As part of the script dtrx receives input from 'find'. However, this leads to the situation that we call dtrx on files that are obviously no archives. Hence, I would like to create a filter to only call dtrx on file that have actual supported extensions.

Is there an official list of all the file types supported by dtrx? The readme list a bunch in the README but this list is not exhaustive.

Thanks


Discussion / hints:
Hi @an-schall , thanks for filing this! I think it's something we can add- would a command-line flag to dump out the list make sense?

Something like:

```bash
$ dtrx --dump-extension-map
Z
bz2
bzip2
bzip2
compress
gz
gzip
gzip
lrz
lrz
lrzip
lz
lzma
lzma
lzma
tar.Z
tar.bz2
tar.gz
tar.lrz
tar.lz
tar.lzma
tar.xz
tar.zst
taz
tb2
tbz
tbz2
tgz
tlz
txz
xz
xz
xz
zstd
```

Note that `dtrx` also maps by mime-type and using the output of the `file` command, but I think extension matching is what would be the most useful to you?
Oh also note that `dtrx` relies on the matching decompression application for the actual decompression (eg you'd need to have `tar`, etc), so the list would indicate which extensions dtrx supports but might not match what the current system can decompress.
Hi,

indeed extension matching is what we are looking for. Also your last point is rather valid: We experienced that dtrx does allow for decompressing jar files, while this is not explicitly listed. Would there be a straight-forward war to get a comprehensive list of all the file types covered even including those supported by the 3rd party applications?

Cheers
I think we could provide a list of:
- built in extensions
- extensions from MIME type mapping

Let me take a crack at this, I'll post here when I have something.

Fix patch:
diff --git a/dtrx/dtrx.py b/dtrx/dtrx.py
index 16dfa1a..ff0f0eb 100755
--- a/dtrx/dtrx.py
+++ b/dtrx/dtrx.py
@@ -25,6 +25,7 @@
 
 import errno
 import fcntl
+import itertools
 import logging
 import mimetypes
 import optparse
@@ -1537,6 +1538,37 @@ def abort(self, signal_num, frame):
                 self.clean_destination(os.path.join(directory, basename))
         sys.exit(1)
 
+    @staticmethod
+    def get_supported_extensions():
+        """
+        return supported extensions
+        """
+        # get the lists of built-in extensions and combine them
+        ext_map_base = set(ExtractorBuilder.extension_map.keys())
+        ext_map = set(
+            itertools.chain(
+                *[
+                    x["extensions"]
+                    for x in ExtractorBuilder.extractor_map.values()
+                    if "extensions" in x
+                ]
+            )
+        )
+        ext_map = ext_map_base.union(ext_map)
+
+        # get the list of extensions supplied by mimetypes
+        mimetypes_encodings_map = set([x.lstrip(".") for x in mimetypes.encodings_map])
+        # dtrx only supports a subset of the total types_map set, filter it
+        mimetypes_exts = filter(
+            lambda x: mimetypes.types_map[x] in ExtractorBuilder.mimetype_map,
+            mimetypes.types_map,
+        )
+        mimetypes_exts = set([x.lstrip(".") for x in mimetypes_exts])
+        mimetypes_exts = mimetypes_encodings_map.union(mimetypes_exts)
+
+        # sort the output for consistent order
+        return sorted(ext_map.union(mimetypes_exts))
+
     def parse_options(self, arguments):
         parser = optparse.OptionParser(
             usage="%prog [options] archive [archive2 ...]",
@@ -1604,6 +1636,24 @@ def parse_options(self, arguments):
             default=False,
             help="extract everything to the current directory",
         )
+
+        def list_extensions(option, opt, value, parser, *args, **kwargs):
+            """callback for optparse to list supported extensions"""
+            print("\n".join(ExtractorApplication.get_supported_extensions()))
+            sys.exit()
+
+        parser.add_option(
+            "--list-extensions",
+            action="callback",
+            callback=list_extensions,
+            help=(
+                "list supported filetypes by extension. note that these are the"
+                " filetypes recognized by dtrx, but extraction still relies on the"
+                " appropriate tool to be installed. also note that this is not a"
+                " comprehensive list; dtrx will fall back on the 'file' command if the"
+                " extension is unknown"
+            ),
+        )
         parser.add_option(
             "-v",
             "--verbose",


Test patch:
diff --git a/tests/tests.yml b/tests/tests.yml
index 91911e9..879651a 100644
--- a/tests/tests.yml
+++ b/tests/tests.yml
@@ -860,3 +860,51 @@
   filenames: test-text.br
   baseline: |
     brotli --decompress $1 --output=test-text
+
+- name: list extensions
+  options: "--list-extensions"
+  # note: this pattern is directly passed to python's regex .search function
+  grep: |
+    7z
+    Z
+    arj
+    br
+    bz2
+    cab
+    cpio
+    deb
+    dmg
+    epub
+    gem
+    gz
+    hdr
+    jar
+    lha
+    lrz
+    lz
+    lzh
+    lzma
+    msi
+    rar
+    rpm
+    tar
+    tar.Z
+    tar.bz2
+    tar.gz
+    tar.lrz
+    tar.lz
+    tar.lzma
+    tar.xz
+    tar.zst
+    taz
+    tb2
+    tbz
+    tbz2
+    tgz
+    tlz
+    txz
+    xpi
+    xz
+    zip
+    zst
+    zstd