Subject :Effect of buffer overflows on the IDE data set

From : John Oliver

Date : 13 June 1993

Charles Simon and I have recently looked carefully at the buffer
overflows in the IDE impact data set. We have satisfied ourselves that
the buffer overflow events can be connected unambiguously to specific
high activity events. Thus it is possible to add these missing impacts
back into the IDE data set. I have now done so for the data sets used
for the IDEplot and IDEview programs, and by me in my various data set
analyses. In this TB I will report on the process used.

**Background:**

The data record for IDE made provision for the buffering of 43 impact
records for each LDEF surface (North, South, East, West, Up, Down)
during each 2.5 hour (668 IDE tick) data dump period. A single impact
during a given 13.6 second IDE tick period resulted in a single impact
record word consisting of the time, labeled by detector thickness
(either 0.4 or 1.0 micron). Multiple impacts during a single tick
resulted in two impact record words, the first giving the time, and the
second giving the number of hits at that time on each detector
thickness. Each data record also contains a separate running total count
of impacts on both the 0.4 and 1.0 micron detectors.

If more than 43 impact record words were generated during a data dump
period, the last records were written over the earliest records. Thus
time tagged data on impacts could be lost during periods of high impact
activity.[^1] Because of the separate running total counts, he total
number of impacts was not lost, even during the "buffer overflow" events
described above. Buffer overflow events (detected by a comparison of the
running count of events recorded in impact record words with the running
total impact counts) occurred seven times in the IDE data set.
Information on the buffer overflows was contained in the file
LDEFHITS.TXT which was provided by LaRC along with the LDEF data set.
The specific location in time of the missing impacts can not be
determined. It is possible to infer that they lie after the instant of
the preceding data dump, and prior to the earliest impact still
remaining in the buffer. There is an added possible ambiguity if one
word of a multiple impact record word pair was overwritten.

**Implementation:**

I have written software which has inserted added impacts into the data
sets used by my IDEplot and IDEview programs. I have assigned all added
impacts to the time of the earliest impact recorded in the buffer by
duplicating that event the required number of times. As will be
apparent, buffer overflows occurred generally at the time of a spike or
an intense event in an MOES. In the following paragraphs, I have
provided "snipits", generated by IDEview, of each of the buffer overflow
events. The added points are indicated by gray arrows. I have included
the pertinent text from LDEFHITS.TXT describing the location of the
missing impacts. The numbers on the right of the figures indicate the
number of impacts at the IDE tick marked by the cursor. The text tells
how many impacts were added to the data set.

![](media/image1.emf){width="2.698611111111111in"
height="2.6458333333333335in"}**Buffer overflow 1:\
Record #159**

Inserted:

> \* 29 hits on 0.4 NORTH between 105542 and 105948

Note that the impacts were probably spread out over several IDE ticks
instead of being concentrated in a single tick.

**Buffer overflow 2:\
Record #488**![](media/image2.emf){width="2.6777777777777776in"
height="2.6354166666666665in"}

Inserted:

> \* 1 hit on 1.0 EAST between 325314 and 325422

Not shown here is the start of a weak MOES later in this buffer.

**Buffer overflow 3:\
Record #573**![](media/image3.emf){width="2.6875in"
height="2.6458333333333335in"}

Inserted:

> \* 24 hits on 0.4 EAST between 382094 and 382273
>
> \* 2 hits on 1.0 EAST between 382094 and 382273

This is the June 4 A MOES. Note that the impacts were probably spread
out over several IDE ticks instead of being concentrated in a single
tick.

**June 4A MOES:**![](media/image4.emf){width="5.0569444444444445in"
height="2.3180555555555555in"}

Buffer overflow 3 occurred at the time of the 4th peak of the June 4A
MOES.

**Buffer overflow 4:\
Record #784**![](media/image5.emf){width="2.698611111111111in"
height="2.6458333333333335in"}

Inserted:

> \* 10 hits on 0.4 EAST between 523042 and 523173
>
> \* 5 hits on 1.0 EAST between 523042 and 523173

A bifurcated spike

**Buffer overflow 5:\
Record #1087**![](media/image6.emf){width="2.6777777777777776in"
height="2.6458333333333335in"}

Inserted:

> \* 27 hits on 0.4 NORTH between 725446 and 725485
>
> \* 2 hits on 1.0 NORTH between 725446 and 725485

A bifurcated spike

Note that the impacts were probably spread out over several IDE ticks
instead of being concentrated in a single tick.

**End of tape track
one:**![](media/image7.emf){width="4.008333333333334in"
height="1.9708333333333334in"}

Five IDE data dumps were not recorded.

> \* Data Dump Time
>
> \* Lost Record 1 1131590
>
> \* Lost Record 2 1132258
>
> \* Lost Record 3 1132926
>
> \* Lost Record 4 1133594
>
> \* Lost Record 5 1134262

data inserted as Lost Record 5

**Buffer overflow 6:\
Record 1694**![](media/image8.emf){width="2.698611111111111in"
height="2.6458333333333335in"}

Record 1694 is first record on return track of tape\
and cannot be decoded unambiguously.

Data lost between 1130922 and 1134930:

Inserted (includes **all** data lost at end of tape):

> \* 2 hits 1.0 SPACE
>
> \* 4 hits 1.0 EAST
>
> \* 5 hits 0.4 EAST
>
> \* 4 hits 1.0 SOUTH
>
> \* 17 hits 0.4 SOUTH
>
> \* 3 hits 0.4 NORTH

**Buffer overflow 7:\
Record #2472**![](media/image9.emf){width="2.698611111111111in"
height="2.6458333333333335in"}

Inserted:

> \* 15 hits on 0.4 NORTH between 1653966 and 1654055
>
> \* 11 hits on 1.0 NORTH between 1653966 and 1654055

A bifurcated spike

**Modified Data sets:**

IDEview uses a single compacted working data set HITS.HEX. The old
(prior to 13JUN93) HITS.HEX has been saved as HITSORG.HEX. A new file
has been created with the missing impacts added as described above, as
HITSBOF.HEX. A copy of HITSBOF.HEX was renamed HITS.HEX for future use
with IDEview. If it is necessary to use IDEview with the original data
set, a copy of HITSORG.HEX should be renamed HITS.HEX.

IDEplot uses separate data sets for each panel and detector type:

  ------------- ------------------- ------------------- -------------------
                Working             Original            Impact Added

  North 0.4     no04.hex            no04org.hex         no04bof.hex

  North 1.0     no10.hex            no10org.hex         no10bof.hex

  South 0.4     so04.hex            so04org.hex         so04bof.hex

  South 1.0     so10.hex            so10org.hex         so10bof.hex

  East 0.4      ea04.hex            ea04org.hex         ea04bof.hex

  East 1.0      ea10.hex            ea10org.hex         ea10bof.hex

  West 0.4      we04.hex            we04org.hex         we04bof.hex

  West 1.0      we10.hex            we10org.hex         we10bof.hex

  Space 0.4     up04.hex            up04org.hex         up04bof.hex

  Space 1.0     up10.hex            up10org.hex         up10bof.hex

  Earth 0.4     dn04.hex            dn04org.hex         dn04bof.hex

  Earth 1.0     dn10.hex            dn10org.hex         dn10bof.hex
  ------------- ------------------- ------------------- -------------------

The original data sets have been renamed [aannORG.HEX]{.smallcaps} and
saved. The modied data sets have been named [aann]{.smallcaps}BOF.HEX.
Copies of the [aann]{.smallcaps}BOF.HEX data sets were renamed
[aann]{.smallcaps}.HEX for future use with IDEview. If it is necessary
to use IDEview with the original data set, copies of
[aann]{.smallcaps}ORG.HEX should be renamed [aann]{.smallcaps}.HEX.

[^1]: note that since data from both types of detectors were mixed, high
    activity on the 0.4 micron detectors could result in data loss for
    either 0.4 or 1.0 micron detectors.