Repository: OpenNMT/OpenNMT-tf

Issue:
Parallel text inputter fails with compressed text files
Hello,

When trying to train a NMT model using GZIP compressed text files (see [this doc](https://opennmt.net/OpenNMT-tf/data.html#compressed-data)) with **OpenNMT-tf 2.29.0**, the following exception is raised: `RuntimeError: Parallel datasets do not have the same size`

The issue seems to lie in the `count_line` function of https://github.com/OpenNMT/OpenNMT-tf/blob/master/opennmt/utils/misc.py#L222 which is not able to deal properly with gzipped files.

Here is how to reproduce the issue with the toy-ende data from https://opennmt.net/OpenNMT-tf/quickstart.html#step-1-prepare-the-data

```
import gzip
from pathlib import Path
from opennmt.utils.misc import count_lines

def create_gzip(original_file):
    gziped_file = f'{original_file}.gz'
    
    with open(original_file, 'rt', newline="\n") as f:
        lines = f.readlines()

    with gzip.open(gziped_file, 'wt', encoding="utf-8") as f:
        for line in lines:
            f.write(line)
            
    return gziped_file

base_path = Path('/home/jovyan/mt/opennmt_toy_example/toy-ende')

print('txt src:', count_lines(base_path / "src-train.txt"))
print('txt tgt:', count_lines(base_path / "tgt-train.txt"))
print('gzip src:', count_lines(create_gzip(base_path / "src-train.txt")))
print('gzip tgt:', count_lines(create_gzip(base_path / "tgt-train.txt")))
```

This will output the following:
```
txt src: 10000
txt tgt: 10000
gzip src: 1841
gzip tgt: 2035
```

We can check that the compressed files have indeed the same number of lines:
```
zcat src-train.txt.gz | wc -l
10000
```
```
zcat tgt-train.txt.gz | wc -l
10000
```


Discussion:

Resolving patch:
diff --git a/opennmt/utils/misc.py b/opennmt/utils/misc.py
--- a/opennmt/utils/misc.py
+++ b/opennmt/utils/misc.py
@@ -3,6 +3,7 @@
 import collections
 import copy
 import functools
+import gzip
 import heapq
 import io
 import os
@@ -221,7 +222,8 @@ def item_or_tuple(x):
 
 def count_lines(filename, buffer_size=65536):
     """Returns the number of lines of the file :obj:`filename`."""
-    with tf.io.gfile.GFile(filename, mode="rb") as f:
+    file_class = gzip.open if is_gzip_file(filename) else tf.io.gfile.GFile
+    with file_class(filename, mode="rb") as f:
         num_lines = 0
         while True:
             data = f.read(buffer_size)


Test patch:
diff --git a/opennmt/tests/misc_test.py b/opennmt/tests/misc_test.py
--- a/opennmt/tests/misc_test.py
+++ b/opennmt/tests/misc_test.py
@@ -1,4 +1,6 @@
+import gzip
 import itertools
+import os
 
 import numpy as np
 import tensorflow as tf
@@ -181,6 +183,16 @@ def testRelativeConfig(self):
         with self.assertRaisesRegex(KeyError, "a_3"):
             _ = config["3"]
 
+    def testCountLinesGzip(self):
+        expected_num_lines = 42
+
+        path = os.path.join(self.get_temp_dir(), "file.gz")
+        with gzip.open(path, "wt") as f:
+            for i in range(expected_num_lines):
+                f.write("%d\n" % i)
+
+        self.assertEqual(misc.count_lines(path), expected_num_lines)
+
 
 if __name__ == "__main__":
     tf.test.main()