Repository: openschc/openschc (language: python)

Issue:
pytestifying all tests
We need to have all tests written using the `pytest` so they can be run by GitHub Actions.

I see two locations for tests:
- the `test/` directory. The `README` suggest one needs to run the scripts directly but `test_frag.py` suggests `pytest` is used
- different `test_*.py` script at the root, which I believe should be run directly.

My approach would to migrate *all* tests into the `tests/` directory, using the `pytest` approach.

I can do the heavy-lifting, but @MarinoMtz, please indicate which tests we expect to run.
pytestifying all tests
We need to have all tests written using the `pytest` so they can be run by GitHub Actions.

I see two locations for tests:
- the `test/` directory. The `README` suggest one needs to run the scripts directly but `test_frag.py` suggests `pytest` is used
- different `test_*.py` script at the root, which I believe should be run directly.

My approach would to migrate *all* tests into the `tests/` directory, using the `pytest` approach.

I can do the heavy-lifting, but @MarinoMtz, please indicate which tests we expect to run.
pytestifying all tests
We need to have all tests written using the `pytest` so they can be run by GitHub Actions.

I see two locations for tests:
- the `test/` directory. The `README` suggest one needs to run the scripts directly but `test_frag.py` suggests `pytest` is used
- different `test_*.py` script at the root, which I believe should be run directly.

My approach would to migrate *all* tests into the `tests/` directory, using the `pytest` approach.

I can do the heavy-lifting, but @MarinoMtz, please indicate which tests we expect to run.
pytestifying all tests
We need to have all tests written using the `pytest` so they can be run by GitHub Actions.

I see two locations for tests:
- the `test/` directory. The `README` suggest one needs to run the scripts directly but `test_frag.py` suggests `pytest` is used
- different `test_*.py` script at the root, which I believe should be run directly.

My approach would to migrate *all* tests into the `tests/` directory, using the `pytest` approach.

I can do the heavy-lifting, but @MarinoMtz, please indicate which tests we expect to run.
pytestifying all tests
We need to have all tests written using the `pytest` so they can be run by GitHub Actions.

I see two locations for tests:
- the `test/` directory. The `README` suggest one needs to run the scripts directly but `test_frag.py` suggests `pytest` is used
- different `test_*.py` script at the root, which I believe should be run directly.

My approach would to migrate *all* tests into the `tests/` directory, using the `pytest` approach.

I can do the heavy-lifting, but @MarinoMtz, please indicate which tests we expect to run.


Discussion / hints:
Yes, I agree.
For now we can take the test that are in src/tests, i'll add there the fragmentation ones there progressively 
OK. What I will do is change the GitHub actions configuration so all tests from `test/` run continuously. I will also move test [test_compress.py](https://github.com/openschc/openschc/blob/develop/src/test_compress.py) under `test/` (and pytestify it).
Yes, I agree.
For now we can take the test that are in src/tests, i'll add there the fragmentation ones there progressively 
OK. What I will do is change the GitHub actions configuration so all tests from `test/` run continuously. I will also move test [test_compress.py](https://github.com/openschc/openschc/blob/develop/src/test_compress.py) under `test/` (and pytestify it).
Yes, I agree.
For now we can take the test that are in src/tests, i'll add there the fragmentation ones there progressively 
OK. What I will do is change the GitHub actions configuration so all tests from `test/` run continuously. I will also move test [test_compress.py](https://github.com/openschc/openschc/blob/develop/src/test_compress.py) under `test/` (and pytestify it).
Yes, I agree.
For now we can take the test that are in src/tests, i'll add there the fragmentation ones there progressively 
OK. What I will do is change the GitHub actions configuration so all tests from `test/` run continuously. I will also move test [test_compress.py](https://github.com/openschc/openschc/blob/develop/src/test_compress.py) under `test/` (and pytestify it).
Yes, I agree.
For now we can take the test that are in src/tests, i'll add there the fragmentation ones there progressively 
OK. What I will do is change the GitHub actions configuration so all tests from `test/` run continuously. I will also move test [test_compress.py](https://github.com/openschc/openschc/blob/develop/src/test_compress.py) under `test/` (and pytestify it).

Fix patch:
diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml
index bd00382b..79dc7e6a 100644
--- a/.github/workflows/python-app.yml
+++ b/.github/workflows/python-app.yml
@@ -17,4 +17,6 @@ jobs:
       run: |
         pip install pytest
     - name: Test with pytest
-      run: pytest tests/
+      run: |
+        cd src
+        pytest tests/


Test patch:
diff --git a/src/test_compress.py b/src/test_compress.py
deleted file mode 100644
index 03255482..00000000
--- a/src/test_compress.py
+++ /dev/null
@@ -1,50 +0,0 @@
-from gen_base_import import b2hex
-from compr_core import *
-from compr_parser import *
-from gen_rulemanager import *
-
-import pprint
-import binascii
-
-
-class debug_protocol:
-    def _log(*arg):
-        print(*arg)
-
-p = Parser(debug_protocol)
-
-coap = bytearray(b"""`\
-\x12\x34\x56\x00\x1e\x11\x1e\xfe\x80\x00\
-\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
-\x00\x00\x01\xfe\x80\x00\x00\x00\x00\x00\
-\x00\x00\x00\x00\x00\x00\x00\x00\x02\x16\
-2\x163\x00\x1e\x00\x00A\x02\x00\x01\n\xb3\
-foo\x03bar\x06ABCD==Fk=eth0\xff\x84\x01\
-\x82  &Ehello""")
-
-print (b2hex(coap))
-
-v = p.parse(coap, T_DIR_UP)  # or T_DIR_DW
-pprint.pprint(v[0])
-
-RM = RuleManager(log=debug_protocol)
-RM.Add(file="../examples/configs/comp-rule-100.json")
-RM.Print()
-
-C = Compressor(debug_protocol)
-D = Decompressor(debug_protocol)
-
-r = RM.FindRuleFromPacket(v[0], direction="UP")
-if r != None:
-    print ("selected rule is ", r)
-    schc_packet = C.compress(r, v[0], v[1], T_DIR_UP)
-
-    print (schc_packet)
-    schc_packet.display("bin")
-
-    rbis = RM.FindRuleFromSCHCpacket(schc=schc_packet)
-
-    if rbis != None:
-        pbis = D.decompress(schc_packet, rbis, direction=T_DIR_UP)
-        pprint.pprint (v[0])
-        pprint.pprint(pbis)
diff --git a/src/test_failed/test_ruleman.py b/src/test_failed/test_ruleman.py
deleted file mode 100644
index 59555198..00000000
--- a/src/test_failed/test_ruleman.py
+++ /dev/null
@@ -1,80 +0,0 @@
-import sys
-sys.path.insert(0, ".")
-sys.path.insert(0, "..")
-
-from gen_rulemanager import RuleManager
-from gen_bitarray import BitBuffer
-if sys.implementation.name != "micropython":
-    import pytest
-
-
-context1 = {
-    "devL2Addr":"AABBCCDD",
-    "dstIID":"2001:0db8:85a3::beef"
-}
-context2 = {
-    "devL2Addr":"*",
-    "dstIID":"*"
-}
-bogusRule0 = { # bogus rule with no frag or comp
-    "RuleID": 3
-    }
-rule1 = {
-    "RuleID" : 4,
-    "RuleIDLength" : 5,
-    "Compression" : []
-}
-rule2 = {
-    "RuleID" : 4,
-    "RuleIDLength" : 3,
-    "profile": {
-        "MICAlgorithm": "RCS_RFC8724",
-        "MICWordSize": 8,
-        "L2WordSize": 8
-    },
-    "Fragmentation" : {
-        "dir": "out",
-        "FRMode": "AckOnError",
-        "FRModeProfile": {
-            "dtagSize" : 2,
-            "FCNSize": 3,
-            "ackBehavior" : "afterAll1"
-        }
-    }
-}
-rule3 = {
-    "RuleID" : 7,
-    "RuleIDLength" : 3,
-    "Profile": {
-        "MICAlgorithm": "RCS_RFC8724",
-        "MICWordSize": 8,
-        "L2WordSize": 8
-    },
-    "Fragmentation" : {
-        "dir": "in",
-        "FRMode" :"NoAck"
-    },
-}
-conflictingRule0 = {
-    "RuleID" : 15,
-    "RuleIDLength" : 4,
-    "compression" : { "rule_set": [] }
-    }
-
-@pytest.mark.skip(reason="no way of currently testing this") 
-def test_ruleman_01():
-    # XXX actually, it is not test code right now.
-    RM = RuleManager()
-    RM.add_context(context1, rule1,rule2,rule3)
-    RM.add_context(context2, rule1)
-    print(RM._db)
-    #RM.add_rules(context1, [conflictingRule0])
-    #RM.add_rules(context1, [bogusRule0])
-    print(RM.find_context_bydevL2addr("AABBCCDD"))
-    print(RM.find_context_bydstiid("2001:0db8:85a3::beef"))
-    RM.find_rule_bypacket(context1, BitBuffer(int("10000111",2).to_bytes(1, "big")))
-
-
-# for micropython and other tester.
-if __name__ == "__main__":
-    test_ruleman_01()
diff --git a/src/test_failed/test_simsched.py b/src/test_failed/test_simsched.py
deleted file mode 100644
index 2d1c8f34..00000000
--- a/src/test_failed/test_simsched.py
+++ /dev/null
@@ -1,32 +0,0 @@
-import sys
-sys.path.insert(0, ".")
-sys.path.insert(0, "..")
-
-from net_sim_sched import SimulScheduler
-if sys.implementation.name != "micropython":
-    import pytest
-
-scheduler = SimulScheduler()
-cancel_event_id = None
-
-def callback(arg1, arg2, scheduler=scheduler):
-    global cancel_event_id
-    if cancel_event_id != None:
-        canceled_event = scheduler.cancel_event(cancel_event_id)
-    else:
-        canceled_event = None
-    print("clock=%s callback, argw=%s next_event_time=%s canceled=%s"%(
-        scheduler.get_clock(), repr((arg1, arg2)),
-        scheduler.get_next_event_time(), canceled_event))
-    if arg1 == "1":
-        scheduler.add_event(1, callback, ("2", "two"))
-        cancel_event_id = scheduler.add_event(1, callback, ("none", "none"))
-
-def test_simsched_01():
-    scheduler.add_event(3, callback, ("4", "four"))
-    scheduler.add_event(1, callback, ("1", "one"))
-    scheduler.run()
-
-# for micropython and other tester.
-if __name__ == "__main__":
-    test_simsched_01()
diff --git a/src/tests/Makefile b/src/tests/Makefile
deleted file mode 100644
index 6d8c3b12..00000000
--- a/src/tests/Makefile
+++ /dev/null
@@ -1,12 +0,0 @@
-ifeq ($(PY),)
-	#PY := ../../../micropython/ports/unix/micropython
-	PY := python3
-	# micropython
-endif
-
-all-tests:
-	$(PY) test_bitarray.py
-	$(PY) test_bitarray2.py
-	$(PY) test_bitmap.py
-	$(PY) test_ruleman.py
-	$(PY) test_simsched.py
diff --git a/src/tests/README.rst b/src/tests/README.rst
deleted file mode 100644
index fa7feca8..00000000
--- a/src/tests/README.rst
+++ /dev/null
@@ -1,52 +0,0 @@
-OpenSCHC tests
-==============
-
-Testing, the traditionnal way
------------------------------
-
-Move into the *tests* directory and run the test_xxx.py files directly.
-
-They contain asserts, so that an unexpected result generates an error message.
-Conversely, no error message on the console output means that all results are as expected.
-
-This works with both Python3 and micropython.
-
-You can also run::
-
-  make all-tests
-
-To use another version of python, such as micropython, set it into the PY variable like this::
-
-  PY=../../../micropython/ports/unix/micropython make all-tests
-
-If the path is a relative path, make the path relative to the *tests* directory.
-
-
-Pytest (the modern way)
------------------------
-
-Installation
-************
-
-`Pytest <https://docs.pytest.org/en/latest/>`_ is required.
-
-Install pytest with `pip`::
-
-  pip install pytest
-
-Run
-***
-
-In the src directory, type::
-
-  make pytest
-
-
-Guidelines
-----------
-
-If you generate new pieces of code or alter existing code, please provide/alter test code
-so that they keep working both when executed with pytest and when run directly as the main.
-
-
-
diff --git a/src/tests/__init__.py b/src/tests/__init__.py
deleted file mode 100644
index 8b137891..00000000
--- a/src/tests/__init__.py
+++ /dev/null
@@ -1,1 +0,0 @@
-
diff --git a/tests/coap.pcapng b/src/tests/coap.pcapng
similarity index 100%
rename from tests/coap.pcapng
rename to src/tests/coap.pcapng
diff --git a/src/tests/rule.json b/src/tests/rule.json
deleted file mode 100644
index 9813dcf3..00000000
--- a/src/tests/rule.json
+++ /dev/null
@@ -1,20 +0,0 @@
-
- {
-    "DeviceID" : "lorawan:0000000000000001",
-    "SoR" : [
-	{
-        "RuleID": 11,
-	    "RuleIDLength": 6,
-	    "NoCompression": []
-    },
-    {
-      "RuleID":12,
-      "RuleIDLength":6,
-      "Fragmentation":{
-         "FRMode":"NoAck",
-         "FRDirection" : "DW"
-        }
-   }
-   ]
-}   
-    
\ No newline at end of file
diff --git a/src/test_failed/test_bitarray.py b/src/tests/test_bitarray.py
similarity index 100%
rename from src/test_failed/test_bitarray.py
rename to src/tests/test_bitarray.py
diff --git a/src/test_failed/test_bitarray2.py b/src/tests/test_bitarray2.py
similarity index 100%
rename from src/test_failed/test_bitarray2.py
rename to src/tests/test_bitarray2.py
diff --git a/src/tests/test_bitmap.py b/src/tests/test_bitmap.py
index 047e3606..1c81a7f3 100644
--- a/src/tests/test_bitmap.py
+++ b/src/tests/test_bitmap.py
@@ -7,7 +7,6 @@
 if sys.implementation.name != "micropython":
     import pytest
 
-
 def check_bitmaps(bitmaps, expected):
     print("bitmaps:", bitmaps)
     for i in bitmaps:
diff --git a/src/tests/test_compress.py b/src/tests/test_compress.py
new file mode 100644
index 00000000..95fc2f20
--- /dev/null
+++ b/src/tests/test_compress.py
@@ -0,0 +1,53 @@
+from gen_base_import import b2hex
+from compr_core import *
+from compr_parser import *
+from gen_rulemanager import *
+
+import pprint
+import binascii
+
+
+class debug_protocol:
+    def _log(*arg):
+        print(*arg)
+
+coap = bytearray(b"""`\
+\x12\x34\x56\x00\x1e\x11\x1e\xfe\x80\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
+\x00\x00\x01\xfe\x80\x00\x00\x00\x00\x00\
+\x00\x00\x00\x00\x00\x00\x00\x00\x02\x16\
+2\x163\x00\x1e\x00\x00A\x02\x00\x01\n\xb3\
+foo\x03bar\x06ABCD==Fk=eth0\xff\x84\x01\
+\x82  &Ehello""")
+
+
+
+def test_compress():
+    p = Parser(debug_protocol)
+
+    print (b2hex(coap))
+
+    v = p.parse(coap, T_DIR_UP)  # or T_DIR_DW
+    pprint.pprint(v[0])
+
+    RM = RuleManager(log=debug_protocol)
+    RM.Add(file="../examples/configs/comp-rule-100.json")
+    RM.Print()
+
+    C = Compressor(debug_protocol)
+    D = Decompressor(debug_protocol)
+
+    r = RM.FindRuleFromPacket(v[0], direction="UP")
+    if r != None:
+        print ("selected rule is ", r)
+        schc_packet = C.compress(r, v[0], v[1], T_DIR_UP)
+
+        print (schc_packet)
+        schc_packet.display("bin")
+
+        rbis = RM.FindRuleFromSCHCpacket(schc=schc_packet)
+
+        if rbis != None:
+            pbis = D.decompress(schc_packet, rbis, direction=T_DIR_UP)
+            pprint.pprint (v[0])
+            pprint.pprint(pbis)
diff --git a/tests/test_dummy.py b/src/tests/test_dummy.py
similarity index 100%
rename from tests/test_dummy.py
rename to src/tests/test_dummy.py
diff --git a/tests/test_parse.py b/src/tests/test_parse.py
similarity index 73%
rename from tests/test_parse.py
rename to src/tests/test_parse.py
index e340a290..0e35e60c 100644
--- a/tests/test_parse.py
+++ b/src/tests/test_parse.py
@@ -1,11 +1,14 @@
 import pytest
 import pcapng
 
-import sys
-sys.path.insert(0, "src/")
+#============================ defines =========================================
 
 ETHERNET_HEADER_LENGTH = 14
 
+#============================ helpers =========================================
+
+#============================ tests ===========================================
+
 def test_import():
     import compr_parser
 
@@ -27,5 +30,3 @@ def test_parse():
             direction = 'UP',
             layers    = ["IPv6", "ICMP", "UDP"]
         )
-        with open('poipoi.txt','a') as f:
-            f.write(str(parsedFrame)+'\n')
diff --git a/src/tests/test_ruleman.py b/src/tests/test_ruleman.py
new file mode 100644
index 00000000..466dae49
--- /dev/null
+++ b/src/tests/test_ruleman.py
@@ -0,0 +1,77 @@
+import pytest
+import gen_rulemanager
+import gen_bitarray
+
+#============================ defines =========================================
+
+context1 = {
+    "devL2Addr":             "AABBCCDD",
+    "dstIID":                "2001:0db8:85a3::beef",
+}
+context2 = {
+    "devL2Addr":             "*",
+    "dstIID":                "*",
+}
+bogusRule0 = { # bogus rule with no frag or comp
+    "RuleID":                3,
+}
+rule1 = {
+    "RuleID":                4,
+    "RuleIDLength":          5,
+    "Compression":           [],
+}
+rule2 = {
+    "RuleID":                4,
+    "RuleIDLength":          3,
+    "profile": {
+        "MICAlgorithm":      "RCS_RFC8724",
+        "MICWordSize":       8,
+        "L2WordSize":        8,
+    },
+    "Fragmentation": {
+        "dir":               "out",
+        "FRMode":            "AckOnError",
+        "FRModeProfile": {
+            "dtagSize":      2,
+            "FCNSize":       3,
+            "ackBehavior":   "afterAll1",
+        }
+    }
+}
+rule3 = {
+    "RuleID" :               7,
+    "RuleIDLength" :         3,
+    "Profile": {
+        "MICAlgorithm":      "RCS_RFC8724",
+        "MICWordSize":       8,
+        "L2WordSize":        8,
+    },
+    "Fragmentation": {
+        "dir":               "in",
+        "FRMode":            "NoAck",
+    },
+}
+conflictingRule0 = {
+    "RuleID":                15,
+    "RuleIDLength":          4,
+    "compression": {
+        "rule_set":          [],
+    }
+}
+
+#============================ helpers =========================================
+
+#============================ tests ===========================================
+
+@pytest.mark.skip() 
+def test_ruleman():
+    # XXX actually, it is not test code right now.
+    RM = gen_rulemanager.RuleManager()
+    RM.add_context(context1, rule1,rule2,rule3)
+    RM.add_context(context2, rule1)
+    print(RM._db)
+    #RM.add_rules(context1, [conflictingRule0])
+    #RM.add_rules(context1, [bogusRule0])
+    print(RM.find_context_bydevL2addr("AABBCCDD"))
+    print(RM.find_context_bydstiid("2001:0db8:85a3::beef"))
+    RM.find_rule_bypacket(context1, gen_bitarray.BitBuffer(int("10000111",2).to_bytes(1, "big")))
diff --git a/src/tests/test_simsched.py b/src/tests/test_simsched.py
new file mode 100644
index 00000000..9739ed16
--- /dev/null
+++ b/src/tests/test_simsched.py
@@ -0,0 +1,38 @@
+import pytest
+import net_sim_sched
+
+#============================ defines =========================================
+
+#============================ globals =========================================
+
+scheduler       = net_sim_sched.SimulScheduler()
+cancel_event_id = None
+
+#============================ helpers =========================================
+
+def callback(arg1, arg2, scheduler=scheduler):
+    global cancel_event_id
+    if cancel_event_id != None:
+        canceled_event = scheduler.cancel_event(cancel_event_id)
+    else:
+        canceled_event = None
+    print(
+        "clock={} callback, argw={} next_event_time={} canceled={}".format(
+            scheduler.get_clock(),
+            repr((arg1, arg2)),
+            scheduler.get_next_event_time(),
+            canceled_event
+        )
+    )
+    if arg1 == "1":
+        scheduler.add_event(1, callback, ("2", "two"))
+        cancel_event_id = scheduler.add_event(1, callback, ("none", "none"))
+
+#============================ tests ===========================================
+
+def test_simsched():
+    
+    
+    scheduler.add_event(3, callback, ("4", "four"))
+    scheduler.add_event(1, callback, ("1", "one"))
+    scheduler.run()