Repository: legend-exp/pygama (language: python)

Issue:
Add ORRunDecoderForRun to Orca decoders
I wrote up an initial version of the ORRunDecoderForRun, where the data format was taken from http://orca.physics.unc.edu/orca/Data_Chain/Run_Control.html. I've tested it with build_raw on the L200 test data, and it appears to fit well into the workflow (no errors, and the data shows as expected in the outputted LH5 file).

I started this as a draft PR, since I'm not totally clear how exactly we want to write up new decoders and have them fit into pygama.

A couple of quick comments on basic things I was thinking about when writing up this simple decoder:

- Did I put it in the right spot? (i.e. should it live in orca_flashcam.py, or somewhere else?)
- Is the structure okay? I just based the structure on the ORFlashCamListenerConfigDecoder, since they seemed to both be relatively simple decoders
- I arbitrarily named various decoded values, are there specific conventions that I should follow?

I need to add docstrings, and I'm guessing I should add a test as well (or is there a test I should update?).

---------
Before submitting a pull request, please make sure you've read and understood the pygama developer's guide: https://pygama.readthedocs.io/en/latest/developer.html. In particular, do not forget to:

- [x] Conform to our **coding conventions**
- [x] Update existing or add new **tests**
- [x] Update existing or add new **documentation**
- [x] Address any issue reported by GitHub checks



Discussion / hints:
# [Codecov](https://codecov.io/gh/legend-exp/pygama/pull/421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp) Report
Base: **44.40**% // Head: **44.54**% // Increases project coverage by **`+0.14%`** :tada:
> Coverage data is based on head [(`b07f453`)](https://codecov.io/gh/legend-exp/pygama/pull/421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp) compared to base [(`04120fb`)](https://codecov.io/gh/legend-exp/pygama/commit/04120fb658f95610d977376292522dace29a10ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp).
> Patch coverage: 96.42% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #421      +/-   ##
==========================================
+ Coverage   44.40%   44.54%   +0.14%     
==========================================
  Files          88       89       +1     
  Lines       10371    10399      +28     
==========================================
+ Hits         4605     4632      +27     
- Misses       5766     5767       +1     
```


| [Impacted Files](https://codecov.io/gh/legend-exp/pygama/pull/421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp) | Coverage Δ | |
|---|---|---|
| [src/pygama/raw/orca/orca\_run\_decoder.py](https://codecov.io/gh/legend-exp/pygama/pull/421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp#diff-c3JjL3B5Z2FtYS9yYXcvb3JjYS9vcmNhX3J1bl9kZWNvZGVyLnB5) | `96.29% <96.29%> (ø)` | |
| [src/pygama/raw/orca/orca\_streamer.py](https://codecov.io/gh/legend-exp/pygama/pull/421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp#diff-c3JjL3B5Z2FtYS9yYXcvb3JjYS9vcmNhX3N0cmVhbWVyLnB5) | `84.21% <100.00%> (+0.10%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/legend-exp/pygama/pull/421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=legend-exp).

Nice @slwatkins!

The run records are not specific to FC, it's a general thing for all of Orca. So this decoder should go in its own file, like orca_run_decoder.py

Other than that, it looks good! did you test it?
@jasondet I just moved over the code to a new file, and added some simple docstrings for documentation.

I've been testing on the LEGEND test data file, and it seems to fit into the workflow well. After running `build_raw` on the L200 test file, things seem to be stored as I'd expect after running `show`:

<details>
  <summary>Output from show</summary>

```python
>>> show('./L200-comm-20220519-phy-geds_test.lh5')

/
├── ORFlashCamADCWaveform · table{packet_id,eventnumber,timestamp,runtime,numtraces,tracelist,baseline,daqenergy,channel,ts_pps,ts_ticks,ts_maxticks,to_mu_sec,to_mu_usec,to_master_sec,to_dt_mu_usec,to_abs_mu_usec,to_start_sec,to_start_usec,dr_start_pps,dr_start_ticks,dr_stop_pps,dr_stop_ticks,dr_maxticks,deadtime,waveform,crate,card,ch_orca,fcid}
│   ├── baseline · array<1>{real}
│   ├── card · array<1>{real}
│   ├── ch_orca · array<1>{real}
│   ├── channel · array<1>{real}
│   ├── crate · array<1>{real}
│   ├── daqenergy · array<1>{real}
│   ├── deadtime · array<1>{real}
│   ├── dr_maxticks · array<1>{real}
│   ├── dr_start_pps · array<1>{real}
│   ├── dr_start_ticks · array<1>{real}
│   ├── dr_stop_pps · array<1>{real}
│   ├── dr_stop_ticks · array<1>{real}
│   ├── eventnumber · array<1>{real}
│   ├── fcid · array<1>{real}
│   ├── numtraces · array<1>{real}
│   ├── packet_id · array<1>{real}
│   ├── runtime · array<1>{real}
│   ├── timestamp · array<1>{real}
│   ├── to_abs_mu_usec · array<1>{real}
│   ├── to_dt_mu_usec · array<1>{real}
│   ├── to_master_sec · array<1>{real}
│   ├── to_mu_sec · array<1>{real}
│   ├── to_mu_usec · array<1>{real}
│   ├── to_start_sec · array<1>{real}
│   ├── to_start_usec · array<1>{real}
│   ├── tracelist · array<1>{array<1>{real}}
│   │   ├── cumulative_length · array<1>{real}
│   │   └── flattened_data · array<1>{real}
│   ├── ts_maxticks · array<1>{real}
│   ├── ts_pps · array<1>{real}
│   ├── ts_ticks · array<1>{real}
│   └── waveform · table{t0,dt,values}
│       ├── dt · array<1>{real}
│       ├── t0 · array<1>{real}
│       └── values · array_of_equalsized_arrays<1,1>{real}
├── ORFlashCamListenerConfig · table{readout_id,fcid,telid,nadcs,ntriggers,nsamples,adcbits,sumlength,blprecision,mastercards,triggercards,adccards,gps,ch_boardid,ch_inputnum}
│   ├── adcbits · array<1>{real}
│   ├── adccards · array<1>{real}
│   ├── blprecision · array<1>{real}
│   ├── ch_boardid · array_of_equalsized_arrays<1,1>{real}
│   ├── ch_inputnum · array_of_equalsized_arrays<1,1>{real}
│   ├── fcid · array<1>{real}
│   ├── gps · array<1>{real}
│   ├── mastercards · array<1>{real}
│   ├── nadcs · array<1>{real}
│   ├── nsamples · array<1>{real}
│   ├── ntriggers · array<1>{real}
│   ├── readout_id · array<1>{real}
│   ├── sumlength · array<1>{real}
│   ├── telid · array<1>{real}
│   └── triggercards · array<1>{real}
├── ORRunDecoderForRun · table{subrun_number,runstartorstop,quickstartrun,remotecontrolrun,heartbeatrecord,endsubrunrecord,startsubrunrecord,run_number,time}
│   ├── endsubrunrecord · array<1>{bool}
│   ├── heartbeatrecord · array<1>{bool}
│   ├── quickstartrun · array<1>{bool}
│   ├── remotecontrolrun · array<1>{bool}
│   ├── run_number · array<1>{real}
│   ├── runstartorstop · array<1>{bool}
│   ├── startsubrunrecord · array<1>{bool}
│   ├── subrun_number · array<1>{real}
│   └── time · array<1>{real}
└── OrcaHeader · string
```
</details>
Is this ready for review?
I'd say so! Marking it ready now.
Is this packet type present in any of the Orca test files on legend-testdata?
They were one of the first couple of packets in the L200 commissioning data, here's the Python code below that opens the file and runs `build_raw`:

```python 
import legend_testdata
from pygama.raw import build_raw
from pygama.lgdo import show

LTD = legend_testdata.LegendTestData()
orca_file = LTD.get_path('orca/fc/L200-comm-20220519-phy-geds.orca')

build_raw(
    orca_file,
    in_stream_type='ORCA',
    out_spec='L200-comm-20220519-phy-geds_test.lh5',
    overwrite=True,
)
show('./L200-comm-20220519-phy-geds_test.lh5')
```

So, all the packets in that truncated file have their decoders now. When we add more decoders, we'll have to use a different file for testing/create a new one with more packet types.
OK, should we have a test in this PR or should we wait for a better test file to be uploaded to legend-testdata? We'll have to do it anyways for the Orca decoding validation I guess
I'd be happy to write one up - it looks like there aren't any for the Orca decoders in general. The only one in `tests/raw/orca/` is the below file, which is just a dummy test.

https://github.com/legend-exp/pygama/blob/main/tests/raw/orca/test_orca_packet.py

Maybe @jasondet has an opinion? I'm fine with making one either way.

Fix patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index f6056bd6b..7ed213cfa 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -7,6 +7,7 @@ on:
       - refactor
       - 'releases/**'
   pull_request:
+  merge_group:
   release:
 
 concurrency:


Test patch:
diff --git a/tests/raw/test_daq_to_raw.py b/tests/raw/test_daq_to_raw.py
new file mode 100644
index 000000000..c8e74455f
--- /dev/null
+++ b/tests/raw/test_daq_to_raw.py
@@ -0,0 +1,247 @@
+import json
+import plistlib
+import re
+import struct
+from collections import Counter
+from io import BytesIO
+
+from pygama.lgdo import LH5Store
+from pygama.raw import build_raw
+from pygama.raw.orca import orca_streamer
+from pygama.raw.orca.orca_flashcam import ORFlashCamListenerConfigDecoder
+from pygama.raw.orca.orca_run_decoder import ORRunDecoderForRun
+
+
+class OrcaEncoder:
+    """Encoding function for recreating raw Orca files."""
+
+    def __init__(self, file):
+        """Initialize the encoder for daq to raw testing."""
+
+        self.file = file
+        self.header = None
+
+    def encode_header(self):
+        """Convert orca header back to a byte string."""
+
+        lh5 = LH5Store()
+        test_file, _ = lh5.read_object(
+            "OrcaHeader",
+            self.file,
+        )
+        test_str = str(test_file.value, "utf-8")
+        xml_dict = json.loads(test_str)
+        xml_header = BytesIO()
+        plistlib.dump(xml_dict, xml_header, fmt=plistlib.FMT_XML)
+        xml_header.seek(0)
+        compiled_header = str(xml_header.read(), "utf-8")
+        xml_header.close()
+        fixed_header = re.sub(
+            r"([1-9]\d*\.0)+",
+            lambda x: str(int(float(x.group(0)))),
+            compiled_header,
+        ).replace("1", "1.0", 4)
+
+        self.header = bytes(fixed_header, "utf-8")
+        len_header_true = len(self.header)
+
+        if (len_header_true + 8) % 16 != 0:
+            extra_zeros = 16 - ((len_header_true + 8) % 16)
+            extra_bytes = struct.pack(
+                f"{extra_zeros}b",
+                *([0] * extra_zeros),
+            )
+            self.header += extra_bytes
+
+        len_header_full = len(self.header) + 8
+
+        init_header = struct.pack("i", (len_header_full) // 4)
+        init_header += struct.pack("i", len_header_true)
+
+        self.header = init_header + self.header
+
+    def encode_orflashcamconfig(self, ii):
+        """Convert orca flashcam config data back to byte strings."""
+
+        lh5 = LH5Store()
+        tbl, _ = lh5.read_object(
+            "ORFlashCamListenerConfig",
+            self.file,
+        )
+
+        packets = []
+        packets.append(4 << 18)
+        packets.append((tbl["readout_id"].nda[ii] << 16) + tbl["fcid"].nda[ii])
+
+        decoded_values = ORFlashCamListenerConfigDecoder().get_decoded_values()
+
+        for i, k in enumerate(decoded_values):
+            if i < 2:
+                continue
+            packets.append(tbl[k].nda[ii])
+            if k == "gps":
+                break
+
+        npacks = tbl["ch_boardid"].nda.shape[-1]
+
+        for jj in range(npacks):
+            packets.append(
+                (tbl["ch_boardid"].nda[ii, jj] << 16) + tbl["ch_inputnum"].nda[ii, jj]
+            )
+
+        packets[0] += len(packets)
+
+        return packets
+
+    def encode_orflashcamadcwaveform(self, ii):
+        """Convert orca flashcam ADC waveform data back to byte strings."""
+
+        lh5 = LH5Store()
+        tbl, _ = lh5.read_object(
+            "ORFlashCamADCWaveform",
+            self.file,
+        )
+
+        orca_header_length = 3
+        fcio_header_length = 17
+
+        packets = []
+        packets.append(3 << 18)
+        packets.append(1)
+        packets[1] += orca_header_length << 28
+        packets[1] += fcio_header_length << 22
+
+        packet3 = 0x80000
+        packet3 += tbl["channel"].nda[ii]
+        packet3 += tbl["ch_orca"].nda[ii] << 10
+        packet3 += tbl["crate"].nda[ii] << 22
+        packet3 += tbl["card"].nda[ii] << 27
+        packets.append(packet3)
+
+        # time offsets
+        packets.append(tbl["to_mu_sec"].nda[ii])
+        packets.append(tbl["to_mu_usec"].nda[ii])
+        packets.append(tbl["to_master_sec"].nda[ii])
+        packets.append(tbl["to_dt_mu_usec"].nda[ii])
+        packets.append(tbl["to_abs_mu_usec"].nda[ii])
+        packets.append(tbl["to_start_sec"].nda[ii])
+        packets.append(tbl["to_start_usec"].nda[ii])
+
+        # set the dead region values
+        packets.append(tbl["dr_start_pps"].nda[ii])
+        packets.append(tbl["dr_start_ticks"].nda[ii])
+        packets.append(tbl["dr_stop_pps"].nda[ii])
+        packets.append(tbl["dr_stop_ticks"].nda[ii])
+        packets.append(tbl["dr_maxticks"].nda[ii])
+
+        # set event number and clock counters
+        packets.append(tbl["eventnumber"].nda[ii])
+        packets.append(tbl["ts_pps"].nda[ii])
+        packets.append(tbl["ts_ticks"].nda[ii])
+        packets.append(tbl["ts_maxticks"].nda[ii])
+
+        packets.append(tbl["baseline"].nda[ii] + (tbl["daqenergy"].nda[ii] << 16))
+
+        packets.extend(
+            [
+                xx + (yy << 16)
+                for xx, yy in zip(
+                    tbl["waveform"]["values"].nda[ii, ::2],
+                    tbl["waveform"]["values"].nda[ii, 1::2],
+                )
+            ]
+        )
+
+        wf_samples = 2 * (len(packets) - orca_header_length - fcio_header_length)
+
+        packets[1] += wf_samples << 6
+        packets[0] += len(packets)
+
+        return packets
+
+    def encode_orrun(self, ii):
+        """Convert orca run data back to byte strings."""
+
+        lh5 = LH5Store()
+        tbl, _ = lh5.read_object(
+            "ORRunDecoderForRun",
+            self.file,
+        )
+
+        packets = []
+        packets.append(7 << 18)
+        packets.append(tbl["subrun_number"].nda[ii] << 16)
+        decoded_values = ORRunDecoderForRun().get_decoded_values()
+        for i, k in enumerate(decoded_values):
+            if 0 < i < 7:
+                packets[1] += tbl[k].nda[ii] << (i - 1)
+
+        packets.append(tbl["run_number"].nda[ii])
+        packets.append(tbl["time"].nda[ii])
+
+        packets[0] += len(packets)
+
+        return packets
+
+
+def test_daq_to_raw(lgnd_test_data):
+    """Test function for the daq to raw validation."""
+
+    # open orca daq file and create LH5 file
+    orca_file = lgnd_test_data.get_path("orca/fc/L200-comm-20220519-phy-geds.orca")
+    out_spec = "/tmp/L200-comm-20220519-phy-geds_test.lh5"
+
+    build_raw(
+        orca_file,
+        in_stream_type="ORCA",
+        out_spec=out_spec,
+        overwrite=True,
+    )
+
+    # recreate the header
+    encoder = OrcaEncoder(out_spec)
+    encoder.encode_header()
+
+    rebuilt_orca_data = encoder.header
+
+    # get the order of all of the data IDs
+    orstr = orca_streamer.OrcaStreamer()
+    orstr.open_stream(orca_file)
+    raw_packets = []
+
+    while True:
+        this_packet = orstr.load_packet(skip_unknown_ids=False)
+        if this_packet is None:
+            break
+        raw_packets.append(this_packet.copy())
+
+    orstr.close_stream()
+
+    data_ids = [pack[0] >> 18 for pack in raw_packets]
+
+    # count the number of each ID
+    full_count = Counter(data_ids)
+    buffer_count = {k: 0 for k in full_count}
+
+    # build the raw orca file in the order of data IDs
+    for dd in data_ids:
+        buffer_count[dd] += 1
+        ii = buffer_count[dd] - 1
+
+        if dd == 7:
+            this_packet = encoder.encode_orrun(ii)
+        elif dd == 4:
+            this_packet = encoder.encode_orflashcamconfig(ii)
+        elif dd == 3:
+            this_packet = encoder.encode_orflashcamadcwaveform(ii)
+        else:
+            raise ValueError(f"Encoder does not exist for data ID {dd}")
+
+        rebuilt_orca_data += struct.pack(f"{len(this_packet)}I", *this_packet)
+
+    # load the original raw orca file as a byte string
+    with open(orca_file, "rb") as ff:
+        orig_orca_data = ff.read()
+
+    # assert the byte strings are the same
+    assert rebuilt_orca_data == orig_orca_data