Repository: chrovis/cljam
Pull request #321 (2024-08-27 06:17:57)

Use XZ for LZMA compression in CRAM
Currently, CRAM files generated by the CRAM writer containing LZMA-compressed blocks cannot be read correctly by samtools, resulting in an error.

## Repro

```clojure
(require '[cljam.io.cram :as cram])

(with-open [r (cram/reader "test-resources/cram/medium.cram"
                           {:reference "hg19.fa"})
            w (cram/writer "medium_lzma.cram"
                           {:reference "hg19.fa"
                            :tag-compressor-overrides (constantly :lzma)})]
  (cram/write-header w (cram/read-header r))
  (cram/write-alignments w (cram/read-alignments r) (cram/read-header r)))
```

```console
$ samtools view -T hg19.fa medium_lzma.cram
[E::lzma_mem_inflate] LZMA decode failure (error 7)
[E::cram_next_slice] Failure to decode slice
samtools view: error reading file "medium_lzma.cram"
$
```

## Cause

According to the CRAM specification, the LZMA compression method used in CRAM is actually based on the XZ format:

> CRAM uses the xz Stream format to encapsulate this algorithm, as defined in `https://tukaani.org/xz/xz-file-format.txt`.
> https://github.com/samtools/hts-specs/blob/ebebbc8c2910ef2d4c5e7119c6f9ffac3bb6a0cb/CRAMv3.tex#L2503

In fact, htsjdk [uses `XZCompressorOutputStream`/`XZCompressorInputStream`](https://github.com/samtools/htsjdk/blob/204a0dbc132739c54387446793a948904fd30715/src/main/java/htsjdk/samtools/cram/compression/LZMAExternalCompressor.java#L44-L61) from commons-compress when the LZMA method is specified.

On the other hand, the current cljam CRAM writer uses `LZMACompressorOutputStream` for LZMA compression. This mismatch causes the generated CRAM files to be unreadable by samtools or htsjdk.

## Change

The PR replaces the use of `LZMACompressorOutputStream`/`LZMACompressorInputStream` with `XZCompressorOutputStream`/`XZCompressorInputStream` for LZMA compression and decompression.

I have confirmed that with this change, CRAM files generated by the CRAM writer containing LZMA-compressed blocks can be successfully read by samtools without errors.


Patch:
diff --git a/src/cljam/io/cram/decode/structure.clj b/src/cljam/io/cram/decode/structure.clj
index 078d0a2..d9d0a1f 100644
--- a/src/cljam/io/cram/decode/structure.clj
+++ b/src/cljam/io/cram/decode/structure.clj
@@ -115,7 +115,7 @@
                 compressor (case method
                              1 CompressorStreamFactory/GZIP
                              2 CompressorStreamFactory/BZIP2
-                             3 CompressorStreamFactory/LZMA
+                             3 CompressorStreamFactory/XZ
                              (throw
                               (ex-info (str "compression method " method
                                             " not supported")
diff --git a/src/cljam/io/cram/encode/compressor.clj b/src/cljam/io/cram/encode/compressor.clj
index d11dcf9..d36a1fd 100644
--- a/src/cljam/io/cram/encode/compressor.clj
+++ b/src/cljam/io/cram/encode/compressor.clj
@@ -2,7 +2,7 @@
   (:import [java.io ByteArrayOutputStream OutputStream]
            [org.apache.commons.compress.compressors.bzip2 BZip2CompressorOutputStream]
            [org.apache.commons.compress.compressors.gzip GzipCompressorOutputStream]
-           [org.apache.commons.compress.compressors.lzma LZMACompressorOutputStream]
+           [org.apache.commons.compress.compressors.xz XZCompressorOutputStream]
            [org.apache.commons.io.output CountingOutputStream]))
 
 (defprotocol ICompressor
@@ -32,7 +32,7 @@
     {:compressor :bzip, :data (.toByteArray baos)}))
 
 (deftype LZMACompressor
-         [^LZMACompressorOutputStream out ^ByteArrayOutputStream baos]
+         [^XZCompressorOutputStream out ^ByteArrayOutputStream baos]
   ICompressor
   (compressor-output-stream [_] out)
   (->compressed-result [_]
@@ -60,7 +60,7 @@
                                                           uncompressed)
                                      :bzip (compress-with #(BZip2CompressorOutputStream. %)
                                                           uncompressed)
-                                     :lzma (compress-with #(LZMACompressorOutputStream. %)
+                                     :lzma (compress-with #(XZCompressorOutputStream. %)
                                                           uncompressed)
                                      (throw
                                       (ex-info (str "compression method " method
@@ -78,7 +78,7 @@
                 :raw (->RawCompressor baos)
                 :gzip (->GzipCompressor (GzipCompressorOutputStream. baos) baos)
                 :bzip (->BZip2Compressor (BZip2CompressorOutputStream. baos) baos)
-                :lzma (->LZMACompressor (LZMACompressorOutputStream. baos) baos)
+                :lzma (->LZMACompressor (XZCompressorOutputStream. baos) baos)
                 :best (->SelectiveCompressor baos #{:raw :gzip :bzip :lzma})
                 (if (set? method-or-methods)
                   (->SelectiveCompressor baos method-or-methods)


Test patch:
diff --git a/test/cljam/io/cram/data_series_test.clj b/test/cljam/io/cram/data_series_test.clj
index 3b56b56..14bfced 100644
--- a/test/cljam/io/cram/data_series_test.clj
+++ b/test/cljam/io/cram/data_series_test.clj
@@ -10,7 +10,7 @@
   (:import [java.io ByteArrayInputStream ByteArrayOutputStream]
            [org.apache.commons.compress.compressors.bzip2 BZip2CompressorInputStream]
            [org.apache.commons.compress.compressors.gzip GzipCompressorInputStream]
-           [org.apache.commons.compress.compressors.lzma LZMACompressorInputStream]))
+           [org.apache.commons.compress.compressors.xz XZCompressorInputStream]))
 
 (deftest build-data-series-decoders-test
   (let [encodings {:BA {:codec :external, :content-id 1}
@@ -454,7 +454,7 @@
                      :raw bais
                      :gzip (GzipCompressorInputStream. bais)
                      :bzip (BZip2CompressorInputStream. bais)
-                     :lzma (LZMACompressorInputStream. bais))]
+                     :lzma (XZCompressorInputStream. bais))]
       (lsb/read-bytes in raw-size))))
 
 (deftest build-data-series-encoders-test