Repository: trifectatechfoundation/bzip2-rs (language: rust)

Issue:
Filesize mismatch when decompressing multi-stream with sizes greater than 2GB (2^31)
Here is a self contained reproducer of the problem: [bzip_bug.zip](https://github.com/alexcrichton/bzip2-rs/files/4919986/bzip_bug.zip)

To reproduce:
```
unzip bzip_bug.zip
cd bzip_bug
cargo run --release --bin bzip_bug
```

You will see the following output:

```
$ cargo run --release --bin bzip_bug
   Compiling pkg-config v0.3.18
   Compiling libc v0.2.72
   Compiling cc v1.0.58
   Compiling bzip2-sys v0.1.9+1.0.8
   Compiling bzip2 v0.4.1
   Compiling bzip_bug v0.1.0 (/private/tmp/foo/bzip_bug)
    Finished release [optimized] target(s) in 9.08s
     Running `target/release/bzip_bug`
Generating expected results
Decompressing stream made with bzip2
Decompressing stream made with pbzip2
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `405900000`,
 right: `3000000000`: decompressed length mismatch', src/main.rs:37:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```

The expected output is that the program complete's sucessfully

The test has two files:
1. raw.dat.bz2 is a 3GB generated file produced using the `bzip2` tool on a mac
2. raw.dat.pbz2 is the same 3GB generated file produced using the `pbzip2` tool on a mac

The data was generated using the generate.rs tool in the package, via the following commands:
```
$ cargo build --release --all
$ ./target/release/generate | nice bzip2 > raw.dat.bz2&
$ ./target/release/generate | nice pbzip2 > raw.dat.pbz2&
```

You can check the output byte counts using `bzcat`:
```
$ bzcat raw.dat.bz2 | wc -c
  3000000000
```

```
$ bzcat raw.dat.pbz2 | wc -c
 3000000000
```

The issue seems to affect files that are larger than 2^31 (which smells like a u32 overflow somewhere)



Discussion / hints:
Thanks for the report! Is there some code to peek at as well? That'd help narrow down where the issue is in this crate.
Hi @alexcrichton  -- yes sorry the code is in `src/main.rs` of the attached [bzip_bug.zip](https://github.com/alexcrichton/bzip2-rs/files/4919986/bzip_bug.zip). Copy / pasting below as well:

```rust
use std::fs::File;
use bzip2::read::{BzDecoder, MultiBzDecoder};
use std::io::{Read, Write};

fn main() -> Result<(), std::io::Error> {

    // decompress them with a bzip multi stream and compare the results (as well to the expected output)                                                                                                                        

    println!("Generating expected results");
    let mut expected = Vec::with_capacity(3_000_000_100);
    let data = "0123456789";
    for _ in 0..300_000_000 {
	expected.write(data.as_bytes())?;
    }
    let expected_len = 3_000_000_000;
    assert_eq!(expected.len(), expected_len);

    // This passes                                                                                                                                                                                                              
    println!("Decompressing stream made with bzip2");
    let mut decompressor = BzDecoder::new(File::open("raw.dat.bz2").expect("raw.dat.bz2 not found"));
    let mut contents = Vec::with_capacity(3_000_000_100);
    let num_read = decompressor.read_to_end(&mut contents).expect("error decompressing bz2 data");
    assert_eq!(num_read, expected_len, "decompressed length mismatch");
    assert_eq!(contents, expected, "data mismatch");

    // This fails with:                                                                                                                                                                                                         
    // ...                                                                                                                                                                                                                      
    // Decompressing stream made with pbzip2                                                                                                                                                                                    
    // thread 'main' panicked at 'assertion failed: `(left == right)`                                                                                                                                                           
    //   left: `405900000`,                                                                                                                                                                                                     
    //  right: `3000000000`: decompressed length mismatch', src/main.rs:30:5                                                                                                                                                    
    //note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                                                                                                                             
    println!("Decompressing stream made with pbzip2");
    let mut decompressor = MultiBzDecoder::new(File::open("raw.dat.pbz2").expect("raw.dat.pbz2 not found"));
    let mut contents = Vec::with_capacity(3_000_000_100);
    let num_read = decompressor.read_to_end(&mut contents).expect("error decompressing pdbz2 data");
    assert_eq!(num_read, expected_len, "decompressed length mismatch");
    assert_eq!(contents, expected, "data mismatch");

    println!("Done");
    Ok(())
}
```


Oh gah sorry I missed the link from before, my bad!
OK sorry I don't have a ton of time to look into this right now, but it may be a relatively easy bug to fix in `MultiBzDecoder`. For me though it will likely take quite a while for me to have time to get back to debugging this.
No worries @alexcrichton  -- if I get a chance I will look into it too. I figured the reproducer was the most important thing to prepare so I wanted to get that posted.

Fix patch:
diff --git a/src/bufread.rs b/src/bufread.rs
index 874c9bdd..a6994f2e 100644
--- a/src/bufread.rs
+++ b/src/bufread.rs
@@ -192,33 +192,44 @@ impl<R> BzDecoder<R> {
 
 impl<R: BufRead> Read for BzDecoder<R> {
     fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
-        if self.done {
-            return Ok(0);
-        }
         loop {
-            let (read, consumed, eof, ret);
+            if self.done && !self.multi {
+                return Ok(0);
+            }
+            let (read, consumed, remaining, ret);
             {
                 let input = self.obj.fill_buf()?;
-                eof = input.is_empty();
+                if self.done {
+                    assert!(self.multi);
+                    if input.is_empty() {
+                        // beyond last stream in multi-stream case
+                        return Ok(0);
+                    } else {
+                        // previous stream ended, more data follows => create new decompressor
+                        self.data = Decompress::new(false);
+                        self.done = false;
+                    }
+                }
                 let before_out = self.data.total_out();
                 let before_in = self.data.total_in();
                 ret = self.data.decompress(input, buf);
                 read = (self.data.total_out() - before_out) as usize;
                 consumed = (self.data.total_in() - before_in) as usize;
+                remaining = input.len() - consumed;
             }
             self.obj.consume(consumed);
 
             let ret = ret.map_err(|e| io::Error::new(io::ErrorKind::InvalidInput, e))?;
             if ret == Status::StreamEnd {
-                if !eof && self.multi {
-                    self.data = Decompress::new(false);
-                } else {
-                    self.done = true;
-                }
-
-                return Ok(read);
+                self.done = true;
+            } else if consumed == 0 && remaining == 0 && read == 0 {
+                return Err(io::Error::new(
+                    io::ErrorKind::UnexpectedEof,
+                    "decompression not finished but EOF reached",
+                ));
             }
-            if read > 0 || eof || buf.len() == 0 {
+
+            if read > 0 || buf.len() == 0 {
                 return Ok(read);
             }
         }
@@ -304,3 +315,22 @@ impl<R: AsyncWrite + BufRead> AsyncWrite for MultiBzDecoder<R> {
         self.get_mut().shutdown()
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::MultiBzDecoder;
+    use std::io::{BufReader, Read};
+
+    #[test]
+    fn bug_61() {
+        let compressed_bytes = include_bytes!("../tests/bug_61.bz2");
+        let uncompressed_bytes = include_bytes!("../tests/bug_61.raw");
+        let reader = BufReader::with_capacity(8192, compressed_bytes.as_ref());
+
+        let mut d = MultiBzDecoder::new(reader);
+        let mut data = Vec::new();
+
+        assert_eq!(d.read_to_end(&mut data).unwrap(), uncompressed_bytes.len());
+        assert_eq!(data, uncompressed_bytes);
+    }
+}


Test patch:
diff --git a/tests/bug_61.bz2 b/tests/bug_61.bz2
new file mode 100644
index 00000000..adb45e7f
Binary files /dev/null and b/tests/bug_61.bz2 differ
diff --git a/tests/bug_61.raw b/tests/bug_61.raw
new file mode 100644
index 00000000..7961b127
Binary files /dev/null and b/tests/bug_61.raw differ