sot/eng_archive — pull request #43 by taldcroft (2012-08-12 12:33:30+00:00)
Title: Update computation of 5min and daily stats

Summary
--------

- Use weighted mean and stddev for calculating stats.
- Use float64 to accumulate sum for computing stats mean.
- Rebuild stats files for the full mission.
- Fix bug that number of samples for daily stats was incorrect.
- Add notes and regression testing code for re-building stats files.
- This release fixes issues #38, #39, and #41.

Testing
--------

The newly-generated 5min and daily stats for all MSIDs were compared to the existing flight values. The maximum difference was examined and converted to a relative change by dividing by the "range" of that MSID. This is in the `rebuild_stats/validate_stats.py` code. Out of ~6000 MSIDs a total of ~600 show differences more than 1% in the mean or sampled value. This is expected at least for some MSIDs (e.g. to address #39 the Time-weighted mean values issue).

A detailed investigation of one MSID `DP_RW_MOM_TOT` with a large (15%) difference showed that the change was expected because of the sampling during that day.

The list of all MSIDs with large changes can be found in `rebuild_stats/diffs.5min` and `rebuild_stats/diffs.daily`. Reviewers may wish to scan through these files for anything unexpected. A sample entry is:

AWD3TQI means=0.09112829 2010:324:23:58:53.816 0.218795210123 6.73643112183

This means the max relative difference in the mean value for `AWD3TQI` is 0.0911 (9.11%) and occurred at the given date. The final two numbers are the median of all daily mean values and the `range` of the MSIDs which was used to compute the relative difference. The `range` is the maximum of
`[median of means, max_value_in_telemetry - min_value_in_telemetry)]`

Finally, confirmed that the repeated data issue (#39) has been fixed.

Review
--------

@aarvai @matthewdahmer @jeanconn - please review the testing. No significant code changes have been put in since the PR #42 version.

The new database statistic values can be accessed for testing using:

setenv ENG_ARCHIVE /proj/sot/ska/tmp/eng_archive

Then for example:

```python
>>> import Ska.engarchive.fetch_eng as fetch
fetch: using ENG_ARCHIVE=/proj/sot/ska/tmp/eng_archive for archive path
>>> dat = fetch.Msid('AWD3TQI', '2000:001', stat='daily')
>>> dat.plot()
```

@nrawolk @Gregg140 - I do not expect any backward incompatible changes, but as eng archive users you may be interested.