# Presentation: 416473

## Event Data History

- David Adams
- BNL
- Atlas Software Week
- December 2001

## Contents

- Definitions
- Event Processing
- Use cases
- Requirements
- Design issues
- Status
- Future

- Event Data History           David Adams   BNL

- December 2001

## Definitions

- Data object
- Unit of data transfer to and from data store
- Event data object (EDO)
- Data object associated with a particular event
- Restrict to the highest level (not subobjects)
- Event data contained object
- Objects contained in EDO’s
- E.g. cluster, track or electron
- Only found in and accessed  through an EDO

- Event Data History           David Adams   BNL

- December 2001

## Definitions (cont)

- Event
- A collection of EDO’s (and their histories) associated with one beam crossing
- Plus virtual EDO’s (histories without data)
- Not necessarily all such data anywhere
- Depends on scope; for example:
- All EDO’s in a file
- All EDO’s accessible at a site
- All EDO’s registered in central store
- Someone’s view of the data
- All EDO’s visible to an algorithm

- Event Data History           David Adams   BNL

- December 2001

## Definitions (cont)

- Algorithm
- Event data is processed by running a series of algorithms
- Input to an algorithm are EDO’s and possibly non-event data such as geometry or calibration
- Output is typically one EDO (can be more)
- Characterized by type, version and a collection of run-time parameters
- Similar to the Gaudi Algorithm class except does not include specification of input data
- Gaudi definition might depend on event

- Event Data History           David Adams   BNL

- December 2001

## Definitions (cont)

- Parent EDO
- Each EDO is constructed by an algorithm from a well-specified collection of input EDO’s
- These input EDO’s are the parents
- Each EDO has a well defined ancestry
- Parents
- Parents of parents
- And so on

- Event Data History           David Adams   BNL

- December 2001

## Definitions (cont)

- Replicated data
- Copy a collection of EDO’s
- Levels:
- File replication
- EDO replication (more difficult)
- Regenerated data
- Data reconstructed by providing input and then running a collection of algorithms equivalent to those used in the original data generation
- Regenerate at EDO level

- Event Data History           David Adams   BNL

- December 2001

## Definitions (cont)

- Event data history
- Includes
- Input data
- Algorithms used to produce the data
- Run-time environment
- Nonessential information (time stamp...)
- Provide history for each EDO
- Combine with ancestor histories to recover the full production chain
- Share information to save space

- Event Data History           David Adams   BNL

- December 2001

## Event Processing

- Raw
- Data

- Track
- Clusters

- Tracks 1

- Tracks 2

- Find 1

- Cluster

- Refit

- Find 2

- Tracks 3

- EDO + history

- Algorithm

- const

- non-const

- Event Data History           David Adams   BNL

- December 2001

## Use cases

- 1. Check history
- User wishes to discover the track fitting algorithm used for an electron
- Electron consists of a track and an EM cluster
- The track and EM cluster EDO’s are parents of the electron EDO
- The history for the electron EDO is used to find the history for the parent track EDO
- The specification of the track fitting algorithm is obtained from the history of the track EDO

- Event Data History           David Adams   BNL

- December 2001

## Use cases (cont)

- 2. Select on history
- User has a collection of track EDO’s and wishes to find those generated with particular algorithm characteristics
- User iterates over the associated EDO histories
- Extract the algorithm data for each
- Save histories with matching algorithm characteristics
- User fetches the EDO’s associated with the saved histories

- Event Data History           David Adams   BNL

- December 2001

## Use cases (cont)

- 3. Virtual (regenerated) data
- User wishes to reproduce refit tracks which have been deleted (or never created)
- Original (parent) track EDO is present
- Refit track history is present and provides
- fitting algorithm (including runtime parameters)
- parent track EDO
- The fitting algorithm is (re)run
- Original tracks are used as input
- Regenerated refit tracks as output

- Event Data History           David Adams   BNL

- December 2001

## Use cases (cont)

- 4. Replicated data
- Track EDO’s for interesting events are replicated in a file at a remote site
- Later a user desires to refit these tracks:
- Parent clusters are replicated in a separate file
- Job is run with both files as input
- The cluster EDO’s in the second file are recognized as parents of the track EDO’s in the first file
- These clusters are used to refit the tracks

- Event Data History           David Adams   BNL

- December 2001

## Use cases (cont)

- 5. History creation in Gaudi
- The history data is extracted while running in the Gaudi framework
- A historian is created at the beginning of the job and it extracts job level history from the OS and Gaudi
- The historian extracts algorithm-specific history from each algorithm
- Each time an EDO is created, the historian uses the algorithm and input data (EDO’s and global) to construct history for that EDO
- The job and algorithm histories can be shared

- Event Data History           David Adams   BNL

- December 2001

## Requirements

- 1. History includes essential information:
- Parent EDO’s
- Relevant global data (calib, alignment, ...)
- Algorithm
- Type, version and run-time parameters
- Release version
- Run time environment
- OS, OS and shared lib versions
- 2. Above must be sufficient to reproduce EDO

- Event Data History           David Adams   BNL

- December 2001

## Requirements (cont)

- 3. History includes nonessential information:
- Event identifier
- Time stamp
- Computer identifier
- Job identifier
- CPU time consumed
- Algorithm return status
- Checksum to verify data

- Event Data History           David Adams   BNL

- December 2001

## Requirements (cont)

- 4. History can exist even if its EDO is deleted
- 5. EDO index
- There must be a way to label (index) an EDO so references to parent EDO’s can be persistent
- 6. EDO indices provide identity
- Indices are unique
- Indices span files, federations, DB technologies and geographical locations
- References to parent EDO’s remain valid when parents are replicated or regenerated

- Event Data History           David Adams   BNL

- December 2001

## Requirements (cont)

- 7. EDO’s can be replicated or regenerated
- Copies have the same data, essential history and index as the original
- For regeneration, much of the nonessential history will differ
- 8. Replicated and regenerated EDO’s are equivalent to the originals
- Either may be provided in place of the original
- 9. The history for a regenerated EDO should indicate its secondary nature

- Event Data History           David Adams   BNL

- December 2001

## Design issues

- Object identifiers
- We require a mechanism to assign a unique index to each EDO and its associated history
- Here is an example implementation:
- Use 64 bits (20 years of 109 events/yr with 5k EDO’s/event uses 47 bits)
- A central source serves collections of unused indices to local disks
- Each job gains exclusive access to a collection
- Collection hands out unique indices

- Event Data History           David Adams   BNL

- December 2001

## Design Issues (cont)

- 2. Distributing history information
- Much history info is common to many EDO’s
- Within a file share data without duplicating it
- Separate out job and algorithm histories
- Job history
- Release version
- Runtime environment
- OS, shared libraries and their versions
- CPU identifier (e.g. hostname)
- Start time

- Event Data History           David Adams   BNL

- December 2001

## Design issues (cont)

- Algorithm history
- Type
- Version
- Name or identifier (Gaudi name)
- Run time parameters (Gaudi properties)
- Subalgorithm histories

- Event Data History           David Adams   BNL

- December 2001

## Design issues (cont)

- Data history (for each EDO)
- Event ID
- EDO
- Job history
- Algorithm history
- Parent EDO’s
- Global data indices (calibration, alignment, ...)
- Start and stop times
- CPU time consumed
- Algorithm return status
- Data checksum

- Event Data History           David Adams   BNL

- December 2001

## Design issues (cont)

- 3. Transient interface
- For now we define transient classes describing the three types of histories
- StoreGate converters will make these persistent
- 4. Historian
- Provides a convenient mechanism for generating history objects

- Event Data History           David Adams   BNL

- December 2001

## Status

- Current implementation
- All classes in package Control/AthenaHistory
- JobHistory
- AlgorithmHistory
- DataHistory
- Historian
- Each class has a component test
- Builds and tests successfully in 2.4.1
- Tests must be run by hand
- (waiting for support from ATLAS/CMT)

- Event Data History           David Adams   BNL

- December 2001

## Future

- 1. Modify athena to create history objects
- 2. Make history objects persist
- 3. Teach algorithm to specify parents
- Or should this come from athena?
- 4. Teach algorithm to return parameters
- Relevant parameters instead of all properties
- 5. Design and implement EDO identifiers

- Event Data History           David Adams   BNL

- December 2001

## Future (cont)

- 6. Add history for secondary objects
- Flag for replicas
- More data for regenerated data
- 7. Missing history
- Because history never written or was deleted
- Ancestry chain is broken
- Merge missing history into that of the child

- Event Data History           David Adams   BNL

- December 2001

## Future (cont)

- 8. Mutable EDO’s
- Updates
- If an EDO is updated in a separate algorithm, then history must include the updates
- Early references
- Complications if an EDO is updated after it is used as a parent.
- Child may not be reproducible from the update
- Reference to parent will need to be extended to include the state of the EDO
- Or treat each state as a separate EDO

- Event Data History           David Adams   BNL

- December 2001