Project: datamanagement
Issue DM-388: Build replicator
Replicator On boot: * Establishes connection with single, pre-configured distributor. * Checks connection with the network outage buffer, the Base raw image cache, and the tape archive. On success, the replicator registers itself with the Orchestration Manager in the fully-operational pool. From K-T: Note that what to check should eventually be dependent on what was specified in the configuration. That way, if the tape library is down, we could still run using the network outage buffer or raw image cache as the redundancy source. In operation: * Receives a job with visit id, exposure sequence number within the visit and raft id. * Queries the Base EFD replica for information needed to process the image. * Subscribes to the Camera Data System (CDS) “startReadout” event and to the OCS telemetry topics. * Waits for a startReadout event * Request the cross-talk corrected exposure for the raft using the CDS client interface and block until it is available. On receipt of crosstalk-corrected image: * Verify integrity of image (hash or checksum) * Send image with telemetry to the distributor, compressing it if the configuration says to do so. At the same time, the image is written to the network outage buffer and the raw image cache using the Data Access Client Framework, retrying until successful for a configured number of tries. All images that are written are tagged with the Archiver mode (by the replicator job). * Request the raw image using the CDS client interface and block until it is available. On receipt of the raw image: * Verify integrity of image (hash or checksum) * Send image to the distributor and simultaneously write to the network outage buffer and the tape archive. ALL image transmission statuses (successful and unsuccessful) are recorded to the Alert Production Database. NOTE: Image some calibration or engineering modes, there may only be raw-image data, not crosstalk-corrected image data. The replicator job configuration will provide for this). NOTE: Replicator jobs will need to detect they are not completing in the expected amount of time. NOTE: Something (the Base DMCS? The Event/Message Monitor?) needs to pay attention to the number of replicator re-registering themselves with the local-only pool and notify NetOps to investigate and resolve the issue. On errors sending or keeping connected to the distributor, the replicator: * unregisters from the “fully-operational” pool * registers with the “local-only” pool The replicator will continue to try and connect to the distributor, and if it is able to reestablish the connection: * unregisters from the “local-only” pool * registers with the “fully-operational” pool Heartbeat messages between replicators and distributors will indicate when the replicators will be able to re-register with the fully-operational pool.
Story points: 10