# Presentation: 134382

## Luminosity Database

**Elizabeth Gallas**
**Fermilab Computing Division /**
**D0 Computing and Analysis Group**

- D0 Database ‘Taking Stock’ meeting
- July 10, 2006

- ElizabethGallas

- July 10, 2006

## Outline

- Luminosity Database Status
- Client/Server Status
- Short Term activities
- Long Term activities
- Storage Requirements
- Conclusions
- Appendix
- Purpose: Luminosity Offline Software
- Lum Database is central to Lum Offline Software
- These systems are extensively documented

- ElizabethGallas

- July 10, 2006

## Luminosity Database Status 

_**Luminosity Database Status **__****_

- Schema is stable 
- Legacy Data is loaded 
- Online Data is loaded automatically  (lumGrabber)
- evolving with other short term work (next slide)
- Data Quality (DQ by LBN is waiting...)
- Test system online 30-March-2006 
- No problems found
- waiting for feedback from DQ group 
- Cannot go to production without specifications

- ElizabethGallas

- July 10, 2006

## Luminosity Client/Server Status

_**Luminosity Client/Server Status**_

- Operations Reports
- Need to:
- understand operation and discrepancies
- address performance
- lmAccess
- Need to:
- understand operation and discrepancies
- address performance
- Short term help: luminosity db server
- Long term help: replace db server (?)
- The corresponding legacy systems still in place.
- Goal for coming year: Need to phase out legacy systems.

- ElizabethGallas

- July 10, 2006

## LumDB: Short Term Activities

_**LumDB: Short Term Activities**_

- Current work on the database:
- Luminosity Constant – many years of studies changing all luminosity measurements back to start of Run 2A
- Other improvements:
- Eliminating duplicate records, impose unique keys and constraints
- Understanding Special Values and Status Flags
- Fix Transition Times
- Get Begin/End LBN for every Store in the database
- loading of fuzzy or other times (transition records)
- Development database fully loaded
- Makes testing/implementation of these changes more robust
- Thanks to DBAs for making this possible.

- ElizabethGallas

- July 10, 2006

## LumDB: Long Term Issues

_**LumDB: Long Term Issues**_

- Procedures for updating the database
- Examples:
- luminosity constant change
- discrepancies found
- In future: establish on case by case basis
- Examples exist: provide templates for future changes
- Program: check database vs online Scalar files
- at regular intervals and especially after updates (above)
- Expect to start this project when ‘constant change’ is complete
- Online data loading
- will require some maintenance/supervision
- Thanks Vladimir for help so far !
- In the absence of change, these systems have become more and more stable, not requiring much supervision.

- ElizabethGallas

- July 10, 2006

## 1. LumDB Table Counts (Corrections and DataQuality)

| TABLE | CURRENT | START
ROWS | END
ROWS |
| --- | --- | --- | --- |
| LBNFactors | 0 |   | no use case |
| RangedFactors | 0 |  | no use case |
| QualFlags | 5 |  | No growth expected |
| QualGroups | 9 |  | No growth expected |
| QualUsers | 8 |  | No growth expected |
| LBNQual | 0 * | 684 k | 1,823 k |
| LBNQualParams | 0 * | 219 k | 583 k |

*** Waiting for additional requirements from Data Quality Group.**

- ElizabethGallas

- July 10, 2006

## 2. LumDB Table Counts (PK’s, Transitions)

| TABLE | CURRENT | START
ROWS | END
ROWS |
| --- | --- | --- | --- |
| LBNS | 2.7 M | 2.0 M | 5.2 M |
| ExpGroups | 8 |  | No growth
expected |
| Lmtypes | 9 |  | No growth
expected |
| TICKS | 160 |  | No growth
expected |
| Transitions | 705 k * | 1 K | 8 k |

*** Transitions – decided to include all transitions except 60 second timer, some duplicates will be eliminated.**

- ElizabethGallas

- July 10, 2006

## 3. LumDB Table Counts (the big ones)

| TABLE | CURRENT | START
ROWS | END
ROWS |
| --- | --- | --- | --- |
| STREAMS | 2.3 M | 1.4 M | 3.8 M |
| PrdFileChks | 43 K | 4 M | 11 M |
| LmRates | 385 M | 217 M | 579 M |
| Delivered | 95 M | 72 M | 192 M |
| ExpFraction | 755 M | 513 M | 1,370 M |
| ExpLums | 355 M | 436 M | 1,162 M |
| Triggers | 294 M | 239 M | 639 M |

- ElizabethGallas

- July 10, 2006

## 4. LumDB Table Counts (Status Conditions)

| STATUS
TABLE | DISTINCT
CONDITIONS | CURRENT | START
ROWS | END
ROWS |
| --- | --- | --- | --- | --- |
| ELTS | 8 | 213 K |  |  |
| LBNS | 37 | 3.0 M | 0.6 M | 1.6 M |
| LFS | 0 | 0 | 100 | 800 |
| LSS | 5 | 1.3 M | 140 K | 374 K |
| LTS | 42 | 550 M | 24 M | 64 M |
| PFCS | 0 | 0 | 420 K | 1.1 M |
| RFS | 0 | 0 | 100 | 800 |
| TKS | 24 | 6 K | 618 K | 1.6 M |
| TRS | 0 | 0 | 100 | 800 |

- ElizabethGallas

- July 10, 2006

## Conclusions

- Luminosity Database
- Stable for some time
- Activity in short term will make it robust for the foreseen future
- Storage: within expectations
- Luminosity Offline Software
- client(s)/db server systems using LumDB in operation
- complementary to legacy systems
- corresponding legacy systems still in place.
- D0 & D0 Luminosity group empowered
- will need advice from DBA’s (as always)
- need specific help from FNAL CD REX on Lum DB Server issue
- conversations underway, need more specifications
- Goal for coming year: Phase out legacy systems:
- Operations Reports,
- lmAccess stage files

- ElizabethGallas

- July 10, 2006

## Appendix

- Backup slides

- ElizabethGallas

- July 10, 2006

## Purpose – Luminosity Offline Software

**Calculate/Deliver a Luminosity Normalization**
**for any physics data set**
**Verify and certify data integrity**
**for normalization purposes**
**Improve error tracking and characterization**
**Produce Operations Reports and Graphs**
**Produce LBN Quality Reports**

- ElizabethGallas

- July 10, 2006

## Purpose – Luminosity Database

- The Luminosity Offline Software is centered on the Luminosity Database
- The Luminosity Database stores information from:
- Luminosity Monitor Detector
- Trigger Framework
- Level 3
- Fermilab Tevatron Accelerator
- Offline Data Processing (SAM)
- "Group" assigned quality quality flags
- "Consistency" determined quality
- Plays administrative role cross checking information from all online and some offline subsystems

- ElizabethGallas

- July 10, 2006

## Documentation

- Luminosity Group – Home page
- www-d0.fnal.gov/phys_id/luminosity/
- links to documentation on all related Online and Offline systems
- Luminosity Offline Software Documentation
- www-d0.fnal.gov/phys_id/luminosity/software/offline.html
- Loading online and offline Data
- Luminosity database description (below)
- clients and servers
- Luminosity Database Documentation (D0 note 4915)
- _[/d0dist/dist/packages/lm_db/devel/www/](http://www-d0.fnal.gov/d0dist/dist/packages/lm_db/devel/www/LumDB.html)__[ LumDB](http://www-d0.fnal.gov/d0dist/dist/packages/lm_db/devel/www/LumDB.html)__[.html](http://www-d0.fnal.gov/d0dist/dist/packages/lm_db/devel/www/LumDB.html)_
- the luminosity database schema
- database definition language in cvs
- descriptions of database tables and columns, relationships, data types
- and links to database access methods in production and development

- ElizabethGallas

- July 10, 2006

## The Luminosity Database “Schema”

_**The Luminosity Database “Schema”**_

- ElizabethGallas

- July 10, 2006