Repository: mfranberg/libplinkio (language: python)

Issue:
Improve the tox and CI/CD settings to support a wide range of platforms
# What does this PR do?

This PR provides tox testing against the multiple versions of Python.
In addition, automated builds of sdist and wheels is also available with this.

# Supplementary information

## Tox

### Why was `py38-` removed from `flake8,black,pylint`?
Because such tools is static checker and need not to run in the virtualenv of the multiple versions of Python.

### Why is the codes tested under `skipsdist = false` and `usedevelop = false` when [.github/workflows/tox.yml](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-8a1944567ec5d9e12aaaac28489c47e6567e13b58d861815379f9b8d3aeb2623) runs?
To prevent bugs such as #35.

###  What is `[coverage:paths]`?
This remaps paths in ".coverage" when `skipsdist = false` and `usedevelop = false`.
Without this, coverage.py cannot recognize the actual source codes.

### Why was [tox-requirements.txt](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-e26af6a9816bb45d6caeae5b8a0c27da6e4b6b0a0c7f1090c9b2c51abff69ade) removed?
Requirements should be defined per tox environment and [tox.ini](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449) is suited for this purpose.

## CI/CD

### Why was "workflow_dispatch:" added to some workflows?
This allow workflow to run manually outside the master branch.

### What are  [.github/workflows/build_dist.yml](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-94abbcc666ebf47df3c54baf467943a0798e35895350bbee769268b4b69f6495) and [.github/workflows/pypi_experimental.yml](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-864c9ddc827c82db40c9c6651c8df0b89fcd27e6b1a5cd54373e89cd88513241)?
"build_dist.yml" build sdist and wheels and save them to artifacts.
The files uploaded to the artifact will be available for download by GitHub users for 90 days.
The wheels except for Apple Silicon will be automatically tested.

"pypi_experimental.yml" build sdist and wheels and publish them to PyPI.  (Experimental)
This workflow calls  ".github/workflows/build_dist.yml" to build the distributables and runs `twine upload`.

## Misc

### Why was  `python_requires='>3.7'` added to [setup.py](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7)?
The reason is that Python 3.7 is the earliest release for which security updates still exist and automated testing I added only covers Python 3.7 and above.

### Why were the source codes under py-plinkio/plinkio moved?
This is because the test codes under py-plinkio/tests import them by mistake when testing sdist and wheels.
I also organized the C files to collect in one directory.

### Why was [plinkfile.py](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-04b1669c3f7ce43a948741e54291e95071a95639c5535f21f90b0cf1ad99fd2e) changed?
Because F-Strings are recommended today and the newer versions of Pylint warn if format() is used.

### Why were some options removed from [.pylintrc](https://github.com/mfranberg/libplinkio/compare/master...Kazuki-TAKEDA:libplinkio:new-cicd?expand=1#diff-ba293004ae8f18975051e825bd8d5c70137a8b5d42d16e0f7c9354aa634c446c)?
The reason is that they are now deprecated and removed in the newer versions of Pylint.
`*-name-hint` were merged into `*-rgx` and removed.
`bad-continuation` was removed because it was buggy.



Discussion / hints:

Fix patch:
diff --git a/src/bed.c b/src/bed.c
index 7ca577c..a9eb87a 100644
--- a/src/bed.c
+++ b/src/bed.c
@@ -207,7 +207,7 @@ transpose_file(const unsigned char *mapped_file, size_t num_loci, size_t num_sam
     unsigned char byte_header[ BED_HEADER_MAX_SIZE ];
     size_t byte_header_length = 0;
    
-    FILE *output_file = fopen( output_path, "w" );
+    FILE *output_file = fopen( output_path, "wb" );
     if( output_file == NULL )
     {
         return PIO_ERROR;
@@ -251,7 +251,7 @@ bed_open(struct pio_bed_file_t *bed_file, const char *path, size_t num_loci, siz
     FILE *bed_fp;
    
     memset( bed_file, 0, sizeof( *bed_file ) );
-    bed_fp = fopen( path, "r" );
+    bed_fp = fopen( path, "rb" );
     if( bed_fp == NULL )
     {
         return PIO_ERROR;
@@ -280,7 +280,7 @@ bed_create(struct pio_bed_file_t *bed_file, const char *path, size_t num_samples
     size_t row_size_bytes;
    
     memset( bed_file, 0, sizeof( *bed_file ) );
-    bed_fp = fopen( path, "w" );
+    bed_fp = fopen( path, "wb" );
     if( bed_fp == NULL )
     {
         return PIO_ERROR;


Test patch:
diff --git a/py-plinkio/tests/data/crlf.bed b/py-plinkio/tests/data/crlf.bed
new file mode 100644
index 0000000..17dd0e7
--- /dev/null
+++ b/py-plinkio/tests/data/crlf.bed
@@ -0,0 +1,1 @@
+l
diff --git a/py-plinkio/tests/data/crlf.bim b/py-plinkio/tests/data/crlf.bim
new file mode 100644
index 0000000..5317c5b
--- /dev/null
+++ b/py-plinkio/tests/data/crlf.bim
@@ -0,0 +1,2 @@
+1	chr1:1	1.000000	1	A	C
+2	chr1:2	2.000000	2	G	T
diff --git a/py-plinkio/tests/data/crlf.fam b/py-plinkio/tests/data/crlf.fam
new file mode 100644
index 0000000..5f97380
--- /dev/null
+++ b/py-plinkio/tests/data/crlf.fam
@@ -0,0 +1,4 @@
+fid1	iid1	0	0	2	1
+fid2	iid2	0	0	2	2
+fid3	iid3	0	0	2	1
+fid4	iid4	0	0	2	2
diff --git a/py-plinkio/tests/plinkio_test.py b/py-plinkio/tests/plinkio_test.py
new file mode 100644
index 0000000..044e15c
--- /dev/null
+++ b/py-plinkio/tests/plinkio_test.py
@@ -0,0 +1,107 @@
+import os
+import pytest
+
+from plinkio import plinkfile
+
+HERE = os.path.abspath(os.path.dirname(__file__))
+TEST_DATA_PREFIX = os.path.join(HERE, "../../tests/data/wgas")
+
+##
+# Tests that opening, reading and closing
+# of plink files works from python.
+#
+
+
+##
+# Test open.
+#
+def test_open():
+    test_pf = plinkfile.open(TEST_DATA_PREFIX)
+    assert test_pf is not None
+
+
+##
+# Make sure that the path is correct.
+#
+def test_get_path():
+    path = TEST_DATA_PREFIX
+    test_pf = plinkfile.open(path)
+    assert path == test_pf.get_path()
+
+
+##
+# Make sure that the number of parsed samples
+# is correct.
+#
+def test_get_samples():
+    test_pf = plinkfile.open(TEST_DATA_PREFIX)
+    assert len(test_pf.get_samples()) == 90
+
+
+##
+# Make sure that the number parsed loci is
+# correct.
+#
+def test_get_loci():
+    test_pf = plinkfile.open(TEST_DATA_PREFIX)
+    assert len(test_pf.get_loci()) == 228694
+
+
+##
+# Make sure open throws IOError when
+# an invalid path is specified.
+#
+def test_fail_open():
+    with pytest.raises(IOError):
+        plinkfile.open("/")
+
+
+##
+# Test iteration.
+#
+def test_iter():
+    test_pf = plinkfile.open(TEST_DATA_PREFIX)
+
+    num_rows = 0
+    for _ in test_pf:
+        num_rows += 1
+
+    assert num_rows == 228694
+
+
+##
+# Tests the snp array.
+#
+class TestSnpArray:
+    # pylint: disable=unused-argument,attribute-defined-outside-init
+
+    ##
+    # Common setup.
+    #
+    @pytest.fixture
+    def fixture_setup(self):
+        test_pf = plinkfile.open(TEST_DATA_PREFIX)
+        self.row = next(test_pf)
+        test_pf.close()
+
+    ##
+    # Test access operator.
+    #
+    def test_access(self, fixture_setup):
+        assert self.row[0] == 2
+        assert self.row[2] == 1
+
+    ##
+    # Test contains.
+    #
+    def test_contains(self, fixture_setup):
+        assert 0 not in self.row
+        assert 1 in self.row
+        assert 2 in self.row
+        assert 3 in self.row
+
+    ##
+    # Test allele count on first row.
+    #
+    def test_count(self, fixture_setup):
+        assert self.row.allele_counts() == [0, 22, 67, 1]
diff --git a/py-plinkio/tests/read_write_test.py b/py-plinkio/tests/read_write_test.py
new file mode 100644
index 0000000..cbf71f9
--- /dev/null
+++ b/py-plinkio/tests/read_write_test.py
@@ -0,0 +1,136 @@
+import os
+import tempfile
+import hashlib
+import re
+import contextlib
+
+from plinkio import plinkfile
+from plinkio.plinkfile import Sample, Locus
+
+HERE = os.path.abspath(os.path.dirname(__file__))
+
+
+##
+# Verify that the contents of bed files are the same.
+#
+def compare_bed_files(test_prefix, write_prefix):
+    with open(test_prefix + ".bed", "rb") as file:
+        data = file.read()
+        test_hash_bed = hashlib.sha256(data).hexdigest()
+    with open(write_prefix + ".bed", "rb") as file:
+        data = file.read()
+        write_hash = hashlib.sha256(data).hexdigest()
+    assert test_hash_bed == write_hash
+
+
+##
+# Verify that the contents of bim files are the same.
+#
+def compare_bim_files(test_prefix, write_prefix):
+    with contextlib.ExitStack() as stack:
+        f_test = stack.enter_context(open(test_prefix + ".bim", "r", encoding="utf-8"))
+        f_write = stack.enter_context(open(write_prefix + ".bim", "r", encoding="utf-8"))
+
+        assert len(f_test.readlines()) == len(f_write.readlines())
+
+        for line_test, line_write in zip(f_test.readlines(), f_write.readlines()):
+            tokens_test = re.split(r"\s+", line_test)
+            tokens_write = re.split(r"\s+", line_write)
+            assert int(tokens_test[0]) == int(tokens_write[0])
+            assert tokens_test[1] == tokens_write[1]
+            assert (
+                (float(tokens_test[2]) - float(tokens_write[2])) / float(tokens_test[2])
+            ) < 1.0e-6
+            assert int(tokens_test[3]) == int(tokens_write[3])
+            assert tokens_test[4] == tokens_write[4]
+            assert tokens_test[5] == tokens_write[5]
+
+
+##
+# Verify that the contents of fam files are the same.
+#
+def compare_fam_files(test_prefix, write_prefix):
+    with contextlib.ExitStack() as stack:
+        f_test = stack.enter_context(open(test_prefix + ".fam", "r", encoding="utf-8"))
+        f_write = stack.enter_context(open(write_prefix + ".fam", "r", encoding="utf-8"))
+
+        assert len(f_test.readlines()) == len(f_write.readlines())
+
+        for line_test, line_write in zip(f_test.readlines(), f_write.readlines()):
+            tokens_test = re.split(r"\s+", line_test)
+            tokens_write = re.split(r"\s+", line_write)
+            assert tokens_test[0] == tokens_write[0]
+            assert tokens_test[1] == tokens_write[1]
+            assert int(tokens_test[2]) == int(tokens_write[2])
+            assert int(tokens_test[3]) == int(tokens_write[3])
+            assert int(tokens_test[4]) == int(tokens_write[4])
+            assert int(tokens_test[5]) == int(tokens_write[5])
+
+
+##
+# Tests reading existing plink files.
+# The test bed file contains CR + LF (0x0D 0x0A).
+#
+def test_read_existing_data():
+    test_data_prefix = os.path.join(HERE, "./data/crlf")
+    test_pf = plinkfile.open(test_data_prefix)
+    samples = test_pf.get_samples()
+    loci = test_pf.get_loci()
+
+    correct_samples = [
+        Sample("fid1", "iid1", "0", "0", 0, 0),
+        Sample("fid2", "iid2", "0", "0", 0, 1),
+        Sample("fid3", "iid3", "0", "0", 0, 0),
+        Sample("fid4", "iid4", "0", "0", 0, 1),
+    ]
+    correct_loci = [
+        Locus(1, "chr1:1", 1.0, 1, "A", "C"),
+        Locus(2, "chr1:2", 2.0, 2, "G", "T"),
+    ]
+    correct_rows = [
+        [3, 2, 0, 0],
+        [1, 1, 0, 0],
+    ]
+    assert samples == correct_samples
+    assert loci == correct_loci
+
+    with tempfile.TemporaryDirectory() as temp_dir:
+        write_data_prefix = os.path.join(temp_dir, "test")
+        writer = plinkfile.create(write_data_prefix, samples)
+
+        for row, correct_row, locus in zip(test_pf, correct_rows, loci):
+            assert list(row) == correct_row
+            writer.write_row(locus, row)
+
+        writer.close()
+
+        compare_bed_files(test_data_prefix, write_data_prefix)
+        compare_bim_files(test_data_prefix, write_data_prefix)
+        compare_fam_files(test_data_prefix, write_data_prefix)
+
+
+##
+# Tests writing new plink files.
+#
+def test_write_new_data():
+    with tempfile.TemporaryDirectory() as temp_dir:
+        plink_prefix = os.path.join(temp_dir, "test")
+
+        samples = [Sample("fid1", "iid1", "0", "0", 0, 0), Sample("fid2", "iid2", "0", "0", 0, 1)]
+        loci = [Locus(1, "chr1:1", 1.0, 1, "A", "C"), Locus(2, "chr1:2", 2.0, 2, "G", "T")]
+        rows = [[0, 1], [1, 2]]
+
+        writer = plinkfile.create(plink_prefix, samples)
+
+        for i, locus in enumerate(loci):
+            writer.write_row(locus, rows[i])
+
+        writer.close()
+
+        reader = plinkfile.open(plink_prefix)
+        assert samples == reader.get_samples()
+        assert loci == reader.get_loci()
+        for row, reader_row in zip(rows, reader):
+            assert row == list(reader_row)
+
+        reader.close()
diff --git a/py-plinkio/tests/write_test.py b/py-plinkio/tests/write_test.py
deleted file mode 100644
index ab7dc96..0000000
--- a/py-plinkio/tests/write_test.py
+++ /dev/null
@@ -1,29 +0,0 @@
-import os
-import tempfile
-
-from plinkio import plinkfile
-from plinkio.plinkfile import Sample, Locus
-
-
-def test_read_write():
-    with tempfile.TemporaryDirectory() as temp_dir:
-        plink_prefix = os.path.join(temp_dir, "test")
-
-        samples = [Sample("fid1", "iid1", "0", "0", 0, 0), Sample("fid2", "iid2", "0", "0", 0, 1)]
-        loci = [Locus(1, "chr1:1", 1.0, 1, "A", "C"), Locus(2, "chr1:2", 2.0, 2, "G", "T")]
-        rows = [[0, 1], [1, 2]]
-
-        writer = plinkfile.create(plink_prefix, samples)
-
-        for i, locus in enumerate(loci):
-            writer.write_row(locus, rows[i])
-
-        writer.close()
-
-        reader = plinkfile.open(plink_prefix)
-        assert samples == reader.get_samples()
-        assert loci == reader.get_loci()
-        for row, reader_row in zip(rows, reader):
-            assert row == list(reader_row)
-
-        reader.close()
diff --git a/tests/plinkio_test.py b/tests/plinkio_test.py
deleted file mode 100644
index b21a6e1..0000000
--- a/tests/plinkio_test.py
+++ /dev/null
@@ -1,96 +0,0 @@
-import unittest
-
-from plinkio import plinkfile
-
-##
-# Tests that opening, reading and closing
-# of plink files works from python.
-#
-class TestPlinkIo(unittest.TestCase):
-    ##
-    # Test open.
-    #
-    def test_open(self):
-        pf = plinkfile.open( "./data/wgas" )
-        self.assertNotEqual( pf, None );
-
-    ##
-    # Make sure that the path is correct.
-    #
-    def test_get_path(self):
-        path = "./data/wgas"
-        pf = plinkfile.open( path )
-        self.assertEqual( path, pf.get_path( ) )
-
-    ##
-    # Make sure that the number of parsed samples
-    # is correct.
-    #
-    def test_get_samples(self):
-        pf = plinkfile.open( "./data/wgas" )
-        self.assertEqual( len( pf.get_samples( ) ), 90 );
-    
-    ##
-    # Make sure that the number parsed loci is
-    # correct.
-    #
-    def test_get_loci(self):
-        pf = plinkfile.open( "./data/wgas" )
-        self.assertEqual( len( pf.get_loci( ) ), 228694 );
-
-    ##
-    # Make sure open throws IOError when
-    # an invalid path is specified.
-    #
-    def test_fail_open(self):
-        with self.assertRaises( IOError ):
-            plinkfile.open( "/" )
-
-    ##
-    # Test iteration.
-    #
-    def test_iter(self):
-        pf = plinkfile.open( "./data/wgas" )
-
-        num_rows = 0
-        for row in pf:
-            num_rows += 1
-
-        self.assertEqual( num_rows, 228694 )
-
-##
-# Tests the snp array.
-#
-class TestSnpArray(unittest.TestCase):
-    ##
-    # Common setup.
-    #
-    def setUp(self):
-        pf = plinkfile.open( "./data/wgas" )
-        self.row = next( pf )
-        pf.close( )
-
-    ##
-    # Test access operator.
-    #
-    def test_access(self):
-        self.assertEqual( self.row[ 0 ], 2 )
-        self.assertEqual( self.row[ 2 ], 1 )
-
-    ##
-    # Test contains.
-    #
-    def test_contains(self):
-        self.assertFalse( 0 in self.row )
-        self.assertTrue( 1 in self.row )
-        self.assertTrue( 2 in self.row )
-        self.assertTrue( 3 in self.row )
-
-    ##
-    # Test allele count on first row.
-    #
-    def test_count(self):
-        self.assertEqual( self.row.allele_counts( ), [ 0, 22, 67, 1 ] )
-
-if __name__ == '__main__':
-    unittest.main( )