# ICENOTE 107: IceCube DAQ: Operational Scenario Considerations

May 8, 2003

Authors:

A.  Goldschmidt, K. Helbing, C. McParland, R. H. Minor, D. Nygren,

G.T. Przybylski, R.G. Stokstad, K.-H. Sulanke, ...[^1]

Rhm converted to ICENOTE 5/15/03

## Introduction:

Recent evidence indicating the presence of significant near-end
cross-talk in the sample of Geo Space LP quad cable has stimulated both
concern and various suggestions, including departures from the baseline
hardware configuration. The central concern is that the occurrence of
cross-talk during the time calibration process will corrupt the waveform
integrity of the time calibration pulse. In addition to hardware
changes, possible remedies include changes in firmware/software in the
DOM/DOR nexus. While the DAQ function most vulnerable to waveform
corruption is the timing calibration process, even data transmission may
be degraded, depending on the level of cross-talk and mode of operation.

The main motivations for this note are to clarify the baseline
operational scenario, to examine possible additional requirements, to
define the two distinct, basic modes for **DOM Main Board (MB)** time
calibration/data transmission during normal data-taking operation of the
IceCube array, and to define the circumstances that would force a
transition from one mode to the other. Several complex and disparate
factors such as noise sources, oscillator stability, cable cross-talk
effects, and data rates must be considered together. Communication
protocols and software design impose further constraints. While this
note explores several circumstances of conceivable performance
degradations, the reader is cautioned not to conclude that such
performance degradations are deemed likely. The perspective is that of a
physicist, with the expectation that more professional engineering
expertise will greatly reduce the graininess of this panoramic snapshot.

> The anticipated PMT noise rate specified in the IceCube Preliminary
> Design Document (PDD) was 500 Hz. From recent considerations, it
> appears that \~800 Hz is more likely, and that the average data size
> per hit may be more like 22 bytes, rather than \~10 bytes \[refs:
> Klaus Helbing, UW PMT evaluation\]. In the absence of local
> coincidence operation, the total anticipated rate for IceCube DOMs,
> based on string 18 data is 18 kB/sec/DOM, much higher than the PDD
> estimate[^2]. This increase reduces the bandwidth margin for data
> transmission, and will represent a significant risk factor if the PMT
> rate were to climb much higher.
>
> This study also suggests that system requirements for uniformity of
> array operation and for bandwidth loss to the time calibration process
> should be considered. An underlying goal is to stimulate the creation
> of a fully fleshed-out technical document that shall provide the
> accepted basis for software design and firmware development, and
> ultimately, for array operation.

## The Time Calibration Process:

The experience gained with String 18 is the foundation for the proposed
operational scenarios. The time calibration process is based on
repetitive interrogations of the **DOM MB** local clock at appropriate,
but not highly synchronized, intervals. In IceCube this clock is derived
from a free-running 20 MHz quartz oscillator with excellent short-term
stability. The interrogation process produces data from which the **DOM
MB** local clock frequency **f** and phase **φ** may be continuously
determined. Both long- and short-term frequency drifts are compensated
for in this manner, and, it is unnecessary to require that all
oscillators have a particular frequency **f**. Transformation from **DOM
MB** local time to IceCube master clock time is performed by the
***string processor***. The transformation permits an interpolation of
time between the moments of measurement; the local/master time
transformation must wait for the most recent measurement, rather than
extrapolate.

The concept of this process has been described in a technical note by
Bob Stokstad\[ref?\]. The critical conceptual element in the time
calibration process is the idea that pulses, synchronized to the IceCube
Master Clock, transmitted to the **DOM**s over twisted pair, and
digitized by the **DOM MB**, can provide timing information at the few
ns level, even though the rise-time of the received pulse is much longer
than 1000 ns. Experience with string 18 has shown that resolutions of
\~3 ns rms are routinely possible, and that the cable lengths may be
measured to sub-ns precision with a symmetric \"Reciprocal Active
Pulsing\" (RAP) process. For all this to work well, waveform integrity
is essential, and measurement noise levels must be low.

The timing offset for each **DOM MB** due to the widely varying cable
lengths is determined by the symmetric, back-and-forth RAP process, as
noted. With a properly designed algorithm, the RAP process compensates
automatically for the variations in pulse rise-time due to the variation
in cable length. In principle, a systematic timing error can only arise
from an up-down asymmetry in the hardware elements ( *e.g.,* cable
impedance effects caused by temp and pressure variations, components in
circuitry, termination, impedance glitches, etc...), or in the
firmware + software (not likely). In practice, a number of small
hardware asymmetries will unavoidably exist, some of which may be
explored through laboratory investigation, such as the impact of two
DOMs connected to one pair. Initial tests in the laboratory indicate
that the effect of impedance glitches introduces a waveform distortion
and corresponding systematic offset in rough accordance with
expectations. However, because we are exploiting the timing pulse to a
precision/rise-time ratio of about 1 part per thousand, it must be
admitted that small systematic effects may exist that have yet to be
appreciated.

## Operational Modes:

There appear to be two basic, distinct ***operational modes*** for the
time calibration process:

-   ***Mode A:*** The central element of Mode A is that ***DOR*** cards
    initiate the timing calibration process, either by following a
    command by the ***DOM Hub processor***, or by a scheduler embedded
    within a state machine internal to the ***DOR*** card. Beyond that,
    there are two "sub-modes" which are expected to yield equivalent
    information content.

> In the first sub-mode, the **DOM MB** is instructed to enter time
> calibration mode by a message from the ***DOR*** card. The first pulse
> of the time calibration packet to the **DOM**s is transmitted at an
> instant related robustly to the master clock. **DOM MB**s receive the
> timing pulse and digitize the waveform, applying the local clock
> time-stamp. **DOM MB**s subsequently transmit waveform and time-stamp
> data to ***DOR*** cards. The scheduling rate is a parameter determined
> by the ***string processor***, based on the measured error/stability
> level. This schedule rate parameter is downloaded by the ***string
> processor*** to either the ***DOM Hub processor*** or the ***DOR***
> card, depending upon the implementation.
>
> In the second sub-mode, the **DOM MB** transmits a time calibration
> message to the ***DOR*** card, following an instruction by the
> ***DOR*** card to do so. The ***DOR*** card receives the response
> message (containing the DOM local time-stamp), digitizes the waveform,
> and applies the master clock time stamp. Each of these sub-modes is
> utilized in the RAP method to measure cable lengths. Normally, only
> one of these sub-modes would be used to track **f** and **φ**.
> Assuming a maximum interrogation rate of 5 Hz, and an interrogation
> cycle of 3 ms, the dead-time, or bandwidth impact due to time
> calibration is 5 x 3 x 10^-3^ = 1.5 x 10^-2^ or 1.5%[^3].

-   ***Mode B:*** The **DOM MB**, responding to every command from the
    ***DOR*** card, initiates the timing pulse by transmitting as
    digital data the local clock value related to the leading edge of
    the response packet. The firmware must be designed such that the
    loaded clock value is robustly related to the leading edge. Every
    response by the **DOM MB** is thus a possible time calibration
    waveform. The ***DOR*** cards receive the packet (= timing pulse)
    and digitize the waveform leading edge, applying a master clock
    time-stamp. The ***DOR*** cards must either add the waveform data to
    the data block for transmission to the ***string processor***, or
    could also load this data in a separate buffer for subsequent
    transfer. In mode B, much more data will be created by the ***DOR***
    cards than is needed for most DOMs. The ***string processor*** will
    determine how often it needs to grab timing data from the data
    stream, discarding the excess fraction. For a data block request
    rate of 5 Hz, the additional fractional burden placed on bandwidth
    (1 Mbaud rate) in Mode B is 5 Hz x 6 bytes x 10^-5^sec/byte = 0.03%.
    This is a per DOM burden, rising to 0.06% for both DOMs on a pair.
    This has a negligible impact, remaining negligible even for much
    higher data block request rates.

These two modes should provide equivalent information for the time
calibration process, but have differing operational impact. The first,
**Mode A**, requires the **DOM MB** state machine to regularly shift
from normal data acquisition mode to time calibration mode, and it also
adds waveform data to the up-going bandwidth burden. The time resolution
in string 18 has been observed to be better in **Mode A** relative to
**Mode B**, but this may be related to specific noise environment of
string 18 DAQ, not an intrinsic characteristic. **Mode A** has
flexibility in that the nature of the timing pulse could be different
from the standard message protocol.

In the second, **Mode B**, there is never a transition in the DOM state
machine to time calibration mode during normal data acquisition. The
timing pulse is simply the leading edge of the response message, perhaps
a data block, or perhaps only a \"no data yet\" response. The added data
burden is the value of the local clock, anticipated to be not more than
six bytes (48 bit clock). In contrast, the waveform data may be as much
as 128 samples if testing is necessary to determine waveform integrity.
In this case, the data would be 128 x 10 bits = 160 bytes, if packed, or
as much as 256 bytes, without compression. Roughly speaking, **Mode B**
appears to require about 50 times less bandwidth on the cable. The
waveform data collected by the ***DOR*** card in the second **Mode B**
places no significant data flow burden, since the PCI bus is capable of
100+ Mbyte/second. In principle, the ***DOR*** card could be arranged to
choose how often it needs to digitize, capture, and transmit the
waveform data. The putative advantages of **Mode B** are that there is
almost no bandwidth overhead devoted to timing calibration, which may
permit a higher interrogation rate than **Mode A**, and that there would
be no interrupts solely for the on-going time calibration process.
However, the transition to **Mode B** would have implications for
software design all the way from DOM firmware through the string
processor.

In either Mode A or Mode B, the *string processor* converts the local
time-stamp to master clock units. In the first mode, however, the timing
calibration process is initiated by the *DOM Hub processor/ DOR* card as
a separate request to the DOM MBs to perform waveform capture, or to
initiate a time calibration response message. Based on anticipated
performance, discussed further below, it does not seem necessary at this
time to choose Mode B over Mode A. The RAPCAL procedure to measure cable
lengths (not performed during regular data acquisition) would be the
same in either mode, being similar to Mode A. Current software/firmware
effort is toward the implementation of Mode A. A transition to Mode B
can occur if experience forces it, but, as noted, the software/firmware
effort will be substantial. For either Mode A or Mode B, a synchronized
mode of operation may be considered in the circumstance of significant
cross-talk in the cable.

## The Factors that Affect Interrogation Rate:

It is essential that we understand the allowable range of interrogation
rates for the timing process, as this can affect the mode of array
operation. As the rate of interrogation of a hypothetical unstable or
noisy **DOM MB** is increased ever higher - to maintain the system
timing resolution of 5 ns rms - at some point unacceptable impact on the
**DOM MB** to **DOM** **Hub** bandwidth and system overhead will occur.
Therefore, we need to establish the boundaries beyond which a **DOM MB**
must be placed in the class of non-conforming system elements.

The required rate of interrogation for each **DOM MB** will depend on
two basic factors:

-   the measured [stability]{.underline} of its clock;

-   the [accuracy]{.underline} of each measurement of local clock
    frequency **f** and phase **φ**.

The goal we seek is uniform and efficient array operation for any
combination of:

-   poor oscillator stability,

-   undesirable levels of cable cross-talk,

-   high noise levels,

-   high data rates (noisy PMTs).

However, the boundary of the allowable parameter space among these
factors is correlated. First, however, we consider separately the impact
of each of these factors.

### [Oscillator Stability]{.underline}

> The **DOM MB** clock is currently based on a Toyocom quartz
> oscillator. While these are rather inexpensive parts (\~\$6 each),
> they typically display short-term stability (\~5 sec) about 100 times
> better than typical manufacturer\'s specs. A batch of 50 purchased a
> few years ago (similar to those in string 18) were tested in a special
> setup capable of detecting Allen variance of **δf/f** ≅ 1 x 10^-11^.
> More than 90% displayed short term stability (Allen variance - ref
> GTP\] clustered around **δf/f** ≅ 6 x 10^-11^, as measured over 5
> second intervals. In a test of 100 Toyocom 20 MHz devices recently
> purchased and appropriate for IceCube use, 94 of the devices displayed
> similar short term stability (Allen variance - ref GTP\] of **δf/f** ≅
> 5 x 10^-11^, as measured over 5 second intervals. The remaining 6
> devices were less stable, with widely varying performance and
> behaviors. However, all of those six devices display short-term
> stability better than **δf/f** = 1 x 10^-9^.
>
> The interrogation rate **F** is simply related to the oscillator
> stability by **F = (δf/f)/∆t**, where **∆t** is the timing error
> budget allotment for the running time calibration process. If accuracy
> of individual measurements, **δt,** as determined by the chosen
> algorithm, are at least this good, then these two parameters can be
> regarded as equivalent[^4]. The nature of the algorithm to extract the
> timing information may depend both on the stability behavior of the
> oscillator measurements, and on error sources such as cable cross-talk
> and induced EMI. We do not explore the topic of algorithm design
> further in this note, except to note that an algorithm that permits a
> range of interrogation rates is regarded as "uniform", or "identical"
> across the IceCube array.
>
> Here we shall take **∆t** ≡ **[3 ns rms]{.underline}**. Based on
> expectations for DOM interrogation response latency (software in CPU),
> we shall require that the interrogation rate not exceed 5 Hz. Then the
> required stability must not be worse than **δf/f** = 1.5 x 10^-8^.
>
> This is a fairly relaxed requirement for the short-term stability,
> more than an order of magnitude worse than the poorest one tested in
> an ensemble of 100 devices (20 MHz).
>
> Thus we conclude that it is [unnecessary to test oscillators
> individually]{.underline} to cull out unstable devices prior to
> loading. One further possibility is to place footprints for two
> devices, either jumper selectable during **DOM MB** test, or perhaps
> even firmware selectable during in-ice operation. The chance of having
> two marginal devices, thereby requiring rework, would be negligible,
> on the order of 1 x 10^-4^ or less.
>
> The conclusion is that devices with performance characteristics of the
> Toyocom devices will not present an operational difficulty, unless
> long-term performance deteriorates such that **δf/f** (Allen variance
> for 5 seconds) becomes \~ 2 x 10^-8^. The string 18 experience
> provides some evidence that long-term deterioration of stability at
> this level does not occur. Again, aging, usually taken to mean
> long-term frequency drift, is irrelevant.

### [Cable Cross-talk]{.underline}

This issue has received considerable attention recently with the arrival
of a sample of quad from Geo Space LP. The cross-talk data of Sulanke
and that of Przybylski indicate that undesirably high levels of
intra-quad near-end and far-end cross-talk in this sample are present.
The magnitude of cross-talk is quite frequency-dependent, and is reduced
by employing pulse waveforms with longer rise/fall times. Cross-talk in
the Geo Space LP cable distorts the timing pulses beyond an acceptable
level. In other words, if **DOM MB** [X]{.underline} is transmitting
data on one pair in a quad, then **DOM MB** [Y]{.underline}, on the
other pair in the quad, will be unable to acquire usable timing
measurement data due to near-end cross-talk while the data transmission
is occurring. Near-end cross-talk reduces data integrity at 1 Mbaud
transfer rate as well, if both pairs are simultaneously active in
opposite directions. The present Geo Space LP quad sample provides
inadequate margins for data transmission at 1 Mbaud.

In contrast, the measured performance of string 18 (which uses quad
cable from Ericsson) provides evidence that excellent quads do exist.
The near end crosstalk data taken by Przybylski on string 18 cable
installed in the ice indicate a negligible level of cross-talk. String
18 has not yet been operated in a way that would be prone to this
near-end cross-talk, if it were large. Further tests with higher data
rates are planned, to measure possible effects under realistic
IceCube-like conditions. Of course, it is reasonable to expect that
individual twisted pairs will give excellent or superior performance as
well, but this has not been tested in ice[^5].

The requirement placed on intra-quad near-end cross-talk is quite
stringent. If the arriving timing calibration pulse is taken to have
\~200 mV amplitude (varies with cable length), near-end cross-talk in
the 0.4 to 1 MHz range of more than \~ 1 mV p-p will distort the timing
calibration process at LSB level. If the cross-talk is 2 mV p-p or more,
the error may increase, and perhaps the interrogation rate. For the time
calibration process, it is useful to consider cross-talk in the
framework of bi-directional communication, *i.e.,* in terms of the
[induced near-end cross-talk signal amplitude relative to the received
signal amplitude]{.underline} - which is less than the transmitted
signal by a substantial, frequency-dependent, factor. For the quad
cables used for these measurements, the received signal is about 200 mV
for a 1500 mV drive signal. The cross-talk must be not greater than 2
mV/1500 mV  - **57 db**. Because the insertion loss is
length-dependent, the requirement must ultimately be framed for the
maximum length cable. This requirement is for an asynchronous operating
mode.

It has long been clear that one obvious mitigation for near-end
cross-talk that is too large during the time calibration process is to
arrange that both pairs are quiet at the start of the process. The
question is, then, what negative impact would such an operational mode
have on system performance? Extending arguments put forth by Sulanke,
the steady-state operating scenario is **Mode A**, arranged to
equilibrate during normal data acquisition to nearly simultaneous
requests by the ***DOR*** card for data to a DOM on each of the two
twisted pairs in a quad. With the presumption that the PMT noise rates
are similar, impending data blocks in each of the DOMs will be roughly
the same size. Then one may expect that the DOM software-driven latency
to begin data transmission in each of the targeted DOMs will be about
the same. The latency will depend on the data block size, and it is not
yet known how large this latency is likely to be. At 1 Mbit/s, data
transmission of a 4096 byte packet will require roughly 41 ms.[^6] We
assume that the data request interrogation rate will always occur
frequently enough that the impending data block does not exceed 4096
bytes. Some data bandwidth loss must occur, in the sense that the
twisted pair of the DOM that finishes first will be "idle", until the
second DOM completes transmission. We will assume this difference in
data block size is not greater than 20%, corresponding to \~8 ms.[^7]
Whether this is a genuine bandwidth loss is somewhat debatable.

Nevertheless, at the moment when both DOMs have completed data transfer,
the ***DOR*** card in the DOM Hub can initiate the time calibration
procedure for all four DOMs on the quad. No software intervention is
required, as the ***DOR*** card can perform the sequence in firmware.
All this may require about 6 ms, perhaps much less. Assuming a maximum
interrogation rate of 5 Hz, the dead-time, or bandwidth impact due to
time calibration is 5 x (8 + 6) x 10^-3^ = 7%. This is acceptable, if
the proposed system requirement for this is adopted.

Considering a quad, there exist two conceivable \"levels\" of
synchronization of activity on the two pairs. The \"weak\" level permits
bi-directional communication for data acquisition and messaging
purposes, whereas the \"strong\" level permits only uni-directional
communications. It is an implicit requirement of DAQ functionality that
weak synchronization is permissible, whereas strong synchronization may
not be imposed. This places a performance requirement on [conventionally
defined near-end]{.underline} [cross-talk]{.underline} somewhat less
stringent than that for the time calibration process of about -46 db.

Another suggestion is for the ***DOR*** card to simultaneously acquire
waveform data on [both]{.underline} pairs during the time calibration
process (not relevant for **Mode A** sub-mode 1, which has down-going
timing pulses). The cross-talk effects could in principle be subtracted,
using individually tailored recipes acquired for each quad. This level
of complication is regarded as beyond the level of acceptable
complexity, and could be considered only for serious
function-threatening circumstances.

The issue of non-negligible inter-quad or inter-pair cross-talk is much
more complicated. In that circumstance, there seems to be no practical
way to synchronize activity among an ensemble of ***DOR*** cards. The
avoidance of inter-quad or inter-pair cross-talk imposes probably the
most important performance criterion, independent of **Mode A** or
**Mode B**. Both near-end and far-end inter-quad cross-talk have a
pernicious impact on the time calibration process for this reason. This
has not yet been studied much, but again, a performance requirement of 1
mv induced noise signal appears to be appropriate. This roughly
corresponds to a conventionally defined near-end cross-talk criterion of
about 2 mV/1500mV  - **57 db**, assuming a 1500 mV drive signal. This
situation is complicated to analyze, since the quads/pairs within an
assemble string cable are of varying lengths, with breakouts along the
way.

### [Noise]{.underline}

The careful design for differential signal transmission and reception on
the pair has been demonstrated by the high level of rejection for
common-mode noise from the VLF transmitter, and presumably also for
other noise sources. The impact of noise has been partly considered in
another systems requirement note \[ref \"Deep-Ice System Requirement for
Induced Electrical Noise on **DOM** Twisted Pair\", Feb 15, 2003\]

For the purposes of this note, we will assume that the primary impact of
noise is:

-   to distort some fraction of waveforms, such that a fraction of the
    timing measurements will fail a test for waveform integrity (pulsed
    or spikey noise), and/or;

-   to increase the measurement error for all waveforms (continuous
    noise).

> In either case, an increase in the interrogation rate is required to
> maintain the overall calibration accuracy.

For continuous noise, the rate increase will rise roughly as (**δt
/∆t)^2^,** where **δt** is the measurement error, and **∆t** is the
error budget allocation. The algorithm must now form a running average
of several consecutive measurements, rather than just two. It is
conceivable that the algorithm can be framed to accommodate a variable
number of measurements in the running fit, such that no deviation from
"uniform" or "identical" operation exists for those DOMs displaying
noisy waveform capture.

### [Data Rate Impact]{.underline}

Currently, the baseline scenario for array operation is:

-   without local coincidence in effect;

-   two DOMs per twisted pair.

The present estimate for data flow, as noted above, is that each DOM is
expected to produce about 800 Hz x \~22 bytes = \~18 kB/second of
compressed data. For two DOMs/pair, the data rate on the wire will be
\~360 Kbits/second. K-H Sulanke has tested data transmission in both Geo
Space LP and Ericsson quads. The Ericsson quad performs well in either
bi-directional or uni-directional transmission modes \[ref: K-H. Sulanke
note on cable tests\] at least up to 1 Mbit/s. Uni-directional data
transmission at 1 Mbit/s is possible in the Geo Space LP quad. The Geo
Space LP cable provides inadequate margins for data transmission at this
rate in a bi-directional mode. The fact that the anticipated data rate
is now \~40% of the demonstrated transmission rate permits little
flexibility in optimizing both transmission rate and the interrogation
rate needed for timing calibration; the margin is near the prudent
limit.

Because data transmission rate margin is now much less than envisaged in
the PDD, it has been suggested that perhaps IceCube should revert back
to the string 18 configuration, in the sense that only one DOM is
connected to a twisted pair, rather than two DOMs/pair. This choice
would obviously increase the data transmission bandwidth margin or S/N
ratio, since a greater latitude would exist for transmission rate.
However, the costs for such a choice are substantial. First, the cable
mass appears likely to increase, perhaps to the point that two
cables/string would be needed; this might have significant cost impact.
Or, if constant mass is imposed to maintain a single cable/string,
perhaps the smaller diameter of the copper wire might increase the risk
of breakage during handling and deployment, or possibly degrade timing
somewhat since the received signals will be smaller; this has not been
analyzed. Second, each ***DOR*** card will be supporting just four DOMs,
half as many DOMs as in the case of to DOMs/pair; thus, the number of
DOM Hubs would be doubled, or, each DOM Hub will need to support 15
***DOR*** cards, rather than eight. This may have a substantial cost,
space, and power dissipation impact. Clearly, effort to reduce the
observed PMT noise rate and/or the average data size per hit will be
well placed. Investigations aimed at increasing radio-purity and
suppressing the scintillation mechanism in the Benthosphere seem
particularly well motivated. Finally, it is worth noting that the
software design now is based on a master/slave relationship, with the
***DOR*** card as master, in order to accommodate two DOMs/pair. While
this will work as well with only one DOM/pair, considerable effort has
been expended to change from string 18.

The use of local coincidence to reduce upward data flow will be an
inherent capability in the IceCube array. However, on-going religious
debates have not resolved whether this should be preferred over the "no
local coincidence" operation. The issue may be stated that, as a
discovery instrument, IceCube should not discard data *a priori*.
However, the exploitation of this data flow, which is \~99% PMT SPE
noise due to scintillation in the benthosphere, remains totally unclear.
The resolution of this issue requires a fairly mature Monte Carlo to
assess whether significantly isolated hits add anything to the
scientific reach. Slow-moving, but highly luminous, objects would be
easily detected in local coincidence mode by the non-SPE character of
the signals. In contrast to string 18, where only nearest or
next-to-nearest neighbor correlations could be created, local
coincidence in IceCube is foreseen to operate as a programmable
"coherence length", extending well beyond nearest- or next-to-nearest
neighbor if needed. If chosen, local coincidence operation will reduce
the upward data flow by about two orders of magnitude. Local coincidence
operation may well turn out to be an attractive option. The task of
histogramming PMT noise rates for the IceCube Supernova Search goals
would be transferred to the DOMs, which would report the data in
fine-bin (\~10 ms) summaries. Transformation to master clock units would
require a simple interpolation procedure.

## Interrogation Rate:

> With the context presented above, we now return to the topic of
> interrogation rate. Within the expected range of **DOM MB** clock
> performance, the needed rate of interrogation may be expected to vary
> over a fairly wide range. For example, if the timing error for
> measurement in some **DOM MB** is **δt** is 2 ns rms, and the short
> term stability is **δf/f** ≅ 5 x 10^-11^, then a measurement might
> need to occur only every \~30 seconds. On the other hand, for an
> observed timing measurement error for a particular **DOM MB** is
> **δt** = 5 ns rms, and that particular **DOM MB** clock short term
> stability is **δf/f** \~1 x 10^-9^, then a measurement will need to
> occur perhaps as often as 1 or 2 Hz.
>
> Consequently, we note that:

-   A wide range for the interrogation rate can still be accommodated
    within the suggested systems requirement of [uniform
    operational]{.underline} mode, if each **DOM MB** is assigned a
    [interrogation rate]{.underline} parameter based on measured
    performance. In other words, if identical algorithmic processing is
    maintained with either more or less frequent calculations of **f**
    and **φ**, it seems that the proposed systems requirement of
    identical processing will be met.

```{=html}
<!-- -->
```
-   It will be necessary to establish a stability criterion, such that
    DOMs displaying clock stability poorer than the criterion will be
    placed in a non-conforming class, at least for high precision data
    analysis such as directional reconstruction. We propose that the
    maximum allowable interrogation rate is 5 Hz in **Mode A**. It is
    worth emphasizing that oscillator instability does not seem to be a
    serious risk.

```{=html}
<!-- -->
```
-   It is important to note that the factors of clock instability and
    induced EMI noise both lead to higher interrogation rates as the
    natural path of mitigation, whereas cross-talk effects are not
    ameliorated by this tactic and require a different approach. Induced
    continuous EMI noise has a potentially pernicious impact, as the
    frequency **F** of measurements may need to rise quadratically to
    maintain the desired running accuracy. This assertion assumes noise
    is dominant over clock instability, and thus that the desired
    running accuracy may be maintained by a fit that uses many less
    accurate measurements.

##  Proposed DAQ System Requirements:

Given the incomplete state of understanding of the timing calibration
process - at least at the parts per thousand level - it seems prudent to
require that:

**1. All DOMs in the array shall be [operated uniformly]{.underline},
*i.e.,* in the same Mode and with "identical" algorithmic processing for
time calibration, in order to ensure that any unknown systematic timing
offset is common to all DOM data.**

The precise meaning of [uniform operation]{.underline} needs
elaboration. The relevance of this proposed requirement becomes clearer
if we consider the possibility that some subset of **DOM MB**s may
display significantly variant behavior or perhaps sub-standard
performance. If this variant subset is large enough that its absence
would create a negative impact for scientific reach, then an operational
scenario that maintains viability for most of this subset will become
important. It is prudent to consider this circumstance, since it is
conceivable that performance degradation may occur either through the
appearance of new noise sources (*e.g., HF pulsed radar*), or possibly
through aging of various components that could conceivably affect
oscillator stability.

The new expected data rate also suggests that constraints be considered
for any process which requires some fraction of available bandwidth on
the quad/pair. As a first approximation to the establishment of a DAQ
systems requirement for the time calibration process, we suggest that:

2\. The loss in bandwidth (≡ twisted pair availability for data
transmission) to the time calibration process shall not exceed 10%. DOMs
that fail to meet performance requirements at this limit fall into the
class of non-conforming elements.

Finally, DAQ operation has an implicit requirement that cable
performance could compromise, aside from the timing process. Explicitly,
it is required that:

3\. The level of cross-talk must be sufficiently low that bi-directional
communication for data acquisition and messaging is not degraded. This
corresponds to a weaker criterion for near-end cross-talk of \~ 46 db.

## Conditions and Scenarios:

In this section, we consider how various operational conditions affect
the choice of system operation mode. In all cases, we presume that local
coincidence is not in effect.

The terms are defined as follows:

-   [Cross-talk negligible]{.underline} ≡ at least -**57 db**, defined
    as above ≡ [near-end cross-talk]{.underline} signal relative to
    [transmitted signal]{.underline}; asynchronous operation of
    intra-quad twisted pairs is feasible for all purposes.

-   [Stable clocks]{.underline} ≡ **δf/f** ≤1 x 10^-9^ (5 second
    interval); **F** less than 1 Hz.

-   [Negligible noise]{.underline} ≡ not more than 1 mV rms differential
    noise on the wire pair in the frequency range 0.3 MHz to 1 MHz at
    the time calibration pulse receiver.

-   [Cross-talk significant]{.underline} ≡ near-end cross-talk
    sufficiently large that time calibration measurement errors exceed 6
    ns, forcing a large increase in **F**.

-   [Less Stable clocks]{.underline} ≡ 1 x 10^-8^ \> **δf/f** \> 1 x
    10^-9^, requiring 1 Hz \<**F** \< 5Hz.

-   [Significant noise]{.underline} ≡ differential noise level on wire
    pair above 1 mV, requiring a increase in **F**, but still less than
    5 Hz.

**A. Cross-talk negligible, stable clocks, negligible noise:** In this
most optimistic set of conditions, either mode is feasible. In **Mode
A** (either sub-mode) the ***DOM Hub processor/ DOR*** card schedules
time calibration interrogation for each **DOM MB** once every few
seconds. Small variations in the interrogation rate, based on measured
performance, do not lead to operational difficulties. The timing
measurements can be interleaved occasionally with requests for regular
data acquisition. **Mode B** can also work well here, but is not needed.

**B**. **Cross-talk negligible, less stable clocks, negligible noise:**
In this perhaps more realistic scenario, the critical issue for **Mode
A** is system or **DOM MB** app overhead if clock stability deteriorates
beyond 5 Hz. **Mode B** offers an advantage, but the number of DOMs in
this category is expected to be very small.

**C. Cross-talk negligible, less stable clocks, significant noise:** The
impact of many conceivable forms of noise may lead to rapidly increasing
interrogation rates, with a potentially lethal consequence for non-local
coincidence **Mode A** system operation. **Mode B** should offer a
significant advantage, as the subset may now be significant. This
situation seems rather unlikely, but not inconceivable.

**D. Cross-talk significant, stable clocks, negligible noise:** In this
case, system operation is coupled, at least across the companion pairs
of the quad. The design complications are substantial. More than one
sub-scenario is possible. It is necessary to enforce some sort of
synchronization, to ensure that the companion pair of the quad is quiet
while a timing pulse is being transmitted. The ***DOR*** card must now
control the sequencing and timing of the commands to the **DOM MB**s. In
this scenario, all four **DOM**s connected to a quad would be
sequentially addressed in an appropriate pattern such that each gets the
timing waveform while the diagonal pair is quiet. In **Mode A** system
latency is a threat since the ***DOM Hub processor*** may have to wait
after issuing a request for data until the ***DOR*** card completes its
current sequence. **Mode A** is viable, since the ***DOR*** card
controls the sequencing. In **Mode B**, the ***DOR*** firmware requires
a more sophisticated state machine, controlling the timing of commands
to the individual **DOM**s through an internal scheduler such that each
**DOM MB** gets the share it needs. System latency may of course be
mitigated by buffering within the ***DOR*** card. A cycle of four steps
is needed to provide opportunity for each **DOM MB** on the quad to
respond while the companion pair is quiet. In either **Mode A or B**,
there is a kind of unavoidable inefficiency since a particular **DOM
MB** participates in the calibration process only once every four
commands (assuming equal needs, and that two DOMs are never addressed
simultaneously). If a particular DOM needs a substantially greater
interrogation than its cable mates, then the scheduler can in principle
provide this capability, but the firmware complexity increases.

**E. Cross-talk significant, less stable clocks, negligible noise:** In
this case, the interrogation rates now vary by much more than an order
of magnitude, reaching several Hz for many **DOM MB**s. It may now
evolve that calls by the ***DOM Hub processor*** to initiate the timing
calibration sequence are much too costly to support, in terms of system
overhead. **Mode A** may not be viable, even with the ***DOR*** card
handling the action on the wires. In addition, the bandwidth required to
transmit the waveform data (128 x 10 bits = 160 bytes) is becoming a
substantial fraction of the regular data. Again, it seems rather
unlikely that a large fraction of oscillators will display degraded
stability.

In **Mode B**, the **DOM MB** must load only the status of the clock
into the header of the data block, consisting of only 6 bytes of data.
Even at 10 Hz, only 60 bytes of data/second are transmitted for this
purpose. The leading edge of the first pulse of the data block must be
robustly and consistently correlated with this clock status value for
the timing calibration process to work.

**F. Cross-talk significant, less stable clocks, significant noise:** In
this least desirable, but not completely unthinkable scenario, the
viable option is **Mode B**. Relative to **E**, the presence of noise
means that even more timing measurements are needed. As the ***DOR***
card is controlling this process, and if the **DOM MB** application does
not display too much latency in response, rates significantly exceeding
5 Hz may be tolerable.

## Discussion:

The **DOM**s have been shown to be impervious to the VLF common-mode
signals. The VLF noise at the **DOMCOM** card is visible, although only
at the least significant bit. One important factor affecting the
viability of **Mode B** is whether the noise at the surface, *i.e.,* the
counting house, is significant or negligible. It seems important and
prudent to take those measures in the design and layout of the counting
house to try to minimize the presence of noise from all likely sources.
The initial tests of the ***DOR*** card itself by Sulanke have shown
excellent noise performance, so it seems possible to maintain the "equal
viability option" of either **Mode A or Mode B**. The DOM Hub will also
need to be tested at full capacity of ***DOR*** cards for noise levels.

## Summary Perspective:

-   A systems requirement should be imposed such that all **DOM MB**s in
    the array be operated "uniformly"- either all in **Mode A or in Mode
    B**, but not mixed, with "identical time calibration algorithmic
    processing", as discussed above.

-   The variations among the DOM ensemble of the interrogation rate
    **F** do not appear to violate this proposed requirement, nor would
    the need to exploit a running fit of *n* measurements.

```{=html}
<!-- -->
```
-   Should a systems requirement be imposed such that the loss in
    bandwidth (≡ twisted pair availability for data transmission) to the
    time calibration process shall not exceed 10%? DOMs failing to meet
    this requirement will be classed as non-conforming elements. The use
    of local coincidence would make this requirement less compelling.

```{=html}
<!-- -->
```
-   The **mode** of array operation may be driven by the need to operate
    successfully the subset of **DOM MB**s that display the poorest
    stability or the poorest measurement accuracy, if this subset is a
    non-negligible fraction of the array (not deemed likely).

-   The \"weak\" level of synchronization is permissible for DAQ
    operation, but the \"strong\" level of synchronization is not an
    acceptable operational scenario.

```{=html}
<!-- -->
```
-   The observed stability of the Toyocom oscillators is sufficiently
    high that the fraction of non-conforming **DOM MB**s is expected to
    be small, probably much less than 1%, and completely negligible if
    two oscillators are on-board selectable.

-   The issue with Toyocom devices is qualification. Oscillators from
    other qualified vendors may perform as well, or perhaps worse, and
    cost much more.

-   The VLF noise rejection of string 18 indicates that rejection of
    low-frequency environmental noise will be very high for the IceCube
    design.

```{=html}
<!-- -->
```
-   Near-end cross-talk above **-57 db** complicates the time
    calibration process of the array, possibly forcing companion pairs
    within a quad to both become quiet.

-   The impact of time calibration for data bandwidth is not
    insignificant, and has not yet been quantitatively defined.
    Nevertheless, this complication appears to be manageable.

```{=html}
<!-- -->
```
-   The demonstrated data transmission rate of 1 Mbaud provides
    adequate, but not large, margin for both data acquisition and the
    time calibration process for an 800 Hz PMT noise rate.

-   An upper limit for PMT noise rate should be considered.

-   System performance goals appear to be met using either Ericsson
    quads, or twisted pair cable. Factors such as cost, weight, and
    deployment may determine the choice.

-   Reversion to only one DOM/pair may have substantial cost, space and
    power impact, as well as impact for cable mass. Small-diameter
    copper wires need study to ascertain effects on reliability and
    timing.

-   The Ericsson quads appear to provide adequate performance for both
    data transmission and time calibration.

-   Further tests to ascertain cross-talk impacts on time calibration in
    string 18 will occur in the near future, using realistic
    IceCube-like baud rates.

-   Inter-quad cross-talk above -57 db will not be correctable by choice
    of mode or synchonization tactics.

-   Efforts to suppress the scintillation mechanism in the benthosphere
    glass, as well as increased radio-purity, should receive high
    priority.

-   The use of local coincidence, and impact on scientific reach, if
    any, should be evaluated as a priority activity within the
    simulation effort.

-   The flexibility of the IceCube DAQ concept and design permits two
    distinct **Modes** for operation.

-   **Mode A** is the preferred implementation for IceCube operation. A
    maximum of 5 Hz for the time calibration process is imposed.

-   **Mode B** may mitigate some conceivably serious instrumental or
    environmental effects, but its development is not warranted at this
    time.

-   Effort should be taken to minimize noise sources that could
    penetrate the ***DOR*** cards or the twisted pairs connected to
    them.

## Conclusion:

The IceCube DAQ design is tolerant of significant departures from
optimum performance expectations for oscillator stability, noise,
intra-quad cross-talk, and data rates, each factor considered
separately. However, certain combinations of degraded performance may
lead to non-continuous departures in array operation, such as a
transition from **Mode A to Mode B.** Inter-quad or inter-pair
cross-talk levels above **-57 db** will be difficult to deal with.
Efforts to reduce the anticipated data rate from **DOM to *DOR***,
either by improvements to data compression/feature extraction in the
DOM, or by reducing the background noise rate (radio-purity +
scintillation suppression in the benthosphere), or by validating the use
of local coincidence, are highly motivated and may increase the safety
margins where most needed.

[^1]: Authors Stokstad and Minor have not had an opportunity to approve
    current text.

[^2]: The actual value of the average data size may become smaller with
    a tuned data compression algorithm.

[^3]: The value of 3 ms is assumed here to include timing calibration
    cycles for both DOMs per twisted pair. The precise value for this is
    still not known accurately.

[^4]: A measurement will involve at least two consecutive
    interrogations. Depending upon the algorithm, there may be numerical
    factors like √2 , *etc,.* not specified here.

[^5]: Helix-wound pairs would have two neighbors, rather than one. For
    equivalent conductor size, etc, the assembled cable of twisted pairs
    will be substantially larger, possibly forcing the use of two main
    cables.

[^6]: We assume that the data request interrogation rate will always
    occur frequently enough that the impending data block does not
    exceed 4096 bytes.

[^7]: It may be worth imposing an upper limit on the noise rate of PMTs
    prior to DOM integration.