# Presentation: 809991

## ATLAS Distributed Analysis:Overview

- David Adams
- BNL
- December 8, 2004

- Distributed Analysis working group
- ATLAS software workshop

## Contents

- ADA Architecture
- Components
- Datasets
- Transformations
- Services
- Changes
- Generic dataset schema
- Hierarchical content
- DIAL catalog interfaces
- Goals for this release
- Current status

- Goals for the next release
- Transformation interface
- Conclusions

- Event Data History           David Adams   BNL

- December 2001

## ADA Architecture

*Generalized*

- Event Data History           David Adams   BNL

- December 2001

## Components

- ADA model
- Data described by a dataset
- Location of data, e.g. files
- Content, e.g. list of event ID’s and the type of data for each event
- Transformation describes an operation that can act on a dataset to produce a new dataset
- Application = code shared by multiple transformations
- Task = user-supplied configuration (parameters or code)
- Job is an instance of a transformation acting on a dataset
- User preferences may be provided
- Should not affect the essential result
- Typically run as a collection of sub-jobs by splitting the input dataset
- Each sub-job applies the same xform its sub-dataset
- Results (output datasets) must be merged
- More generally the transformation might be a DAG (future)

- Event Data History           David Adams   BNL

- December 2001

## Components (cont)

*Transformation*

- Event Data History           David Adams   BNL

- December 2001

## Datasets

- Datasets enable users to examine and access data
- For ATLAS data, we identify
- Types of data
- Used to define dataset categories
- Category is part of the content specification
- Types of datasets
- Currently C++ classes with XML data representation
- Third column indicates if this class exists
- Parameter in the new dataset XML
- See table on following page for ATLAS examples
- There is now a single XML schema for all types of datasets

- Event Data History           David Adams   BNL

- December 2001

## Datasets

| Name | Type | ? | Description |
| --- | --- | --- | --- |
| EVIDS | EventDataset | × | List of event ID’s |
| EVGEN | AtlasPoolEventDataset | × | From event generator |
| HITS | AtlasPoolEventDataset | × | Hits, e.g. from GEANT |
| DIGITS | AtlasPoolEventDataset | × | Digitization of hits |
| BYTE | AtlasByteStreamEventDataset |  | Raw data |
| ESD | AtlasPoolEventDataset | × | Event summary data |
| AOD | AtlasPoolEventDataset | × | Analysis oriented data |
| TAG | AtlasPoolTagEventDataset |  | Event metadata |
| NTUP | RootNtupleDataset |  | Ntuples |
| HISTO | RootHistogramDataset | × | Histograms |
| CBNT | CbntDataset | × | DC1 combined ntuples |
| TEXT | TextDataset |  | Text data, e.g. log files |

- Event Data History           David Adams   BNL

- December 2001

## Datasets (cont)

- acas001> dataset_property -i 10003-20151 print AtlasPoolEventDataset 10003-20151 with no parent is locked and not empty   Content includes 1 block:     AtlasPoolEventDataset:AOD       Content ID list has 17 entries:         type MissingET with with key MET_Base         type MissingET with with key MET_Calib         type MissingET with with key MET_Truth         type ParticleBaseContainer with with key BCandidates         type ParticleBaseContainer with with key ElectronCollection         type ParticleBaseContainer with with key MuonCollection         type ParticleBaseContainer with with key ParticleJetContainer         type ParticleBaseContainer with with key PhotonCollection         type ParticleBaseContainer with with key TauJetCollection         type LVL1_ROI with with key LVL1_ROI         type VxContainer with with key VxPrimaryCandidate         type CTP_Decision with with key CTP_Decision         type INavigable4MomentumCollection with with key MuonboyTrackParticles         type INavigable4MomentumCollection with with key TrackParticleCandidate         type Rec::TrackParticleContainer with with key MooreTrackParticles         type Rec::TrackParticleContainer with with key MuidCombnoSeedTrackParticles         type Rec::TrackParticleContainer with with key MuidStandAloneTrackParticles       Event count is 1073   Location has 1 logical file:     Logical file:       Catalog: MagdaFileCatalog:Atlas       ID: AOD_3401_MultiLeptonGamma.AOD.pool.root       State: READONLY

*Content*

*Location*

*ID*

*Type*

- Example dataset

*Too many events to list*

*Content type*

*No sub-datasets*

- Event Data History           David Adams   BNL

- December 2001

## Transformations (cont)

- For ATLAS we identify the above transformations
- Characterized by input and output dataset categories
- Most common ones listed above
- Others likely
- Those available now are highlighted
- See talks by F. Fassi and C. Haeberli

*Soon*

*Now*

- Event Data History           David Adams   BNL

- December 2001

## Services

- Services enable users to find and examine existing data and create new data.
- Services include:
- Analysis services to submit and monitor jobs
- Catalog services to
- Select data
- Record data, metadata and transformations
- Examine and record data provenance
- Data management services to access the data (files)
- Clients provide the user interface to these services
- ROOT command line
- Python command line (back soon)
- GUI (based on Python) planned

- Event Data History           David Adams   BNL

- December 2001

## Changes

- Move from DIAL 0.92 to 0.94 (almost)
- Generic dataset schema (see following)
- Hierarchical content (see following)
- Unique ID service
- Many changes to catalog interface (see following)
- Transformations
- Integration with production system (C. Haeberli)
- Integrate analysis algorithm from the analysis tools group (F. Fassi)
- Package management
- Define user/application interface (G. Rybkine)
- Provide reference implementation (G. Rybkine)

- Event Data History           David Adams   BNL

- December 2001

## Changes (cont)

- Analysis services
- Continued integration with GLite (D.Liko)
- Begin work on prodsys analysis service (F. Brochu)
- Data management
- Improved understanding of SRM
- Integration of gLite prototype file catalog into DQ (F. Orellana)

- Event Data History           David Adams   BNL

- December 2001

## Generic dataset schema

- Version 0.94 of DIAL include a class GenericDataset
- Means to write to and read from an XML description
- All ADA datasets inherit from this without adding persistent data
- Advantages
- Processing system does not need to know the full dataset type
- Much easier to make use of datasets outside of DIAL
- Including languages other than C++, e.g. python
- Other components already have generic schema
- I.e., the application, task, job
- Schema for the first two need work

- Event Data History           David Adams   BNL

- December 2001

## Hierarchical content

- Each dataset description includes content:
- List of event ID’s if relevant and not too large
- List of type-keys describing the contained object
- For each event in an event dataset
- Like the type-keys in StoreGate
- New release sorts this list into groups
- typically one per processing stage
- For ATLAS: RDO, ESD, AOD, ...
- Dataset can now hold both ESD and AOD with clear distinction

- Event Data History           David Adams   BNL

- December 2001

## DIAL catalog interface

- Much work in DIAL to rationalize the interface through which users interact with catalogs
- Class interface for standard catalog types
- XyzRepository stores string (XML) descriptions of Xyz objects
- XyzSelectionCatalog associates metadata with Xyz ID and name
- XyzReplicaCatalog associates replica-logical ID’s for Xyz
- Here Xyz = Dataset, Job, Application, Task, ...
- Generic interface for each of the above
- String ID instead Xyz ID
- So implementation of GenericRepository interface can be shared by DatasetRepository, JobRepository, ...
- Generic implementations include
- File based (only for GenericRepository)
- MySQL table
- AMI
- Web service (so far only GenericRepository)

- Event Data History           David Adams   BNL

- December 2001

## Goals for this release

- User should be able to
- Select dataset from DSC (dataset selection catalog)
- Run aodhisto transformation
- Input is any AOD (or other event collection) dataset
- Output is a dataset containing root histograms
- Makes use of the analysis tools algorithm
- User can supply their own job options and analysis algorithm
- Run atlasreco transformation
- Input is any RDO dataset
- Output is ESD dataset
- Makes use of the production system transform for release 9.0.x
- Monitor job status for running jobs
- Get description including location of any output dataset
- Easily view the histograms in a root histogram dataset

- Event Data History           David Adams   BNL

- December 2001

## Current status

- Releases
- DIAL release 0.94 is on hold until everything else needed for the release goals is in place
- Dial 0.93 changes often but is now close to what 0.94 will be
- Functionality
- Root demos 4 and 5 have been added to illustrate use of aodhisto and atlasreco, respectively
- aodhisto has only been run with one dataset at one site
- atlasreco cannot use 9.0.2 and is flaky with 9.0.1 due to ATLAS SW problems
- Magda is being used to catalog and move files
- A few demo single-file datasets have been cataloged
- See http://www.atlasgrid.bnl.gov/dialds/dlShowMain.pl

- Event Data History           David Adams   BNL

- December 2001

## Goals for the next release

- Transformations
- Clarify transformation interface (see following)
- So users can add transformations
- Continue development of aodhisto (F. Fassi)
- Complete suite of prodsys transformations (C. Haeberli)
- Catalogs
- Build catalog of datasets from existing production and user data
- Add transformation catalogs
- Add local (to server) and global job catalogs
- Provide catalog interface integrated with job submission client(s)
- Analysis services
- Enable ADA production with DC2 production system (F. Brochu)
- Enable ADA production and analysis with gLite WMS (D. Liko)

- Event Data History           David Adams   BNL

- December 2001

## Goals for the next release (cont)

- Data management
- User anywhere can put and get data from a storage element (SE)
- SE can retrieve requested data from other SE’s
- Integrate DIAL with DQ and SRM (F. Orellana)
- Package management
- Continue development of ADA package management interface and implementations (G. Rybkine)
- Integrate DIAL with ADA PM system
- Deploy PM at processing sites, i.e. integrate with existing systems
- AJDL
- Revisit transformation specification
- Integrate with GANGA and DC2 production system
- Better error reporting

- Event Data History           David Adams   BNL

- December 2001

## Transformation interface

- Clarify and document transformation interface
- How xform is packaged and released
- How analysis service finds xform
- Runtime environment that a xform can expect
- How xform is called
- How xform finds input dataset and extracts it files
- How transform locates software (including itself and its task)
- How transform stores output files and creates output dataset
- How transform indicates job status (running, failed, done, ...)
- How task (user code) is built and accessed
- Make it easy for users to add their own transformations
- E.g. run my athena algorithm
- Keep task mechanism for runtime configuration

- Event Data History           David Adams   BNL

- December 2001

## Conclusions

- Status
- Much progress since last meeting but more to do
- Still in demo mode
- Releases
- Expect DIAL 0.94 soon
- When other pieces are in place
- Then like to get feedback on interface and functionality
- Aim for ADA/DIAL 1.0 in February
- Useful system: more than  demo
- Meeting the short-term goals outlined earlier
- Need more people
- Within ADA
- More attention from external providers (DQ, AMI, prodsys)
- Physics contributions of data and algorithms

- Event Data History           David Adams   BNL

- December 2001

## More information

- For more information on ADA, see the home page
- http://www.usatlas.bnl.gov/ADA
- Includes status of subprojects, relevant talks and documents, and links to associated projects
- DIAL release 0.94 is described at
- http://www.usatlas.bnl.gov/~dladams/dial/releases/0.94/index.html
- To try it out, run DIAL root demos 4 and 5 in that release
- Comments and questions
- ADA mailing list
- ADA Savannah coming soon
- DIAL Savannah (with bug reporting) linked from DIAL page

- Event Data History           David Adams   BNL

- December 2001