# Presentation: 001026

## ATLAS Distributed Analysis and proposal for ATLAS-LHCb system

- David Adams
- BNL
- March 22, 2004

- ATLAS-LHCb-GANGA Meeting

## Contents

- Definitions
- Architecture
- AJDL
- Application
- Task
- Dataset
- Job
- High-level services
- Analysis service
- Job management service
- Catalog services

- Implementation Strategy
- Effort providers
- ARDA
- Role of GANGA
- Connection to LHCb
- More information

- Event Data History           David Adams   BNL

- December 2001

## Definitions

- Analysis (not necessarily distributed)
- Supports the manipulation and extraction of summary data (e.g. histograms) from any type of event data
- AOD, ESD, ...
- Supports user-level production of event data
- e.g. MC generation, simulation and reconstruction
- Distributed analysis
- Extends the extraction and production support to include distributed users, data and processing.
- Natural extension of non-distributed analysis
- Easily invoked from any ATLAS analysis environment
- including Python, ROOT, command line
- easily ported to any future environment (e.g. JAS)

- Event Data History           David Adams   BNL

- December 2001

## Architecture

- Event Data History           David Adams   BNL

- December 2001

## AJDL

- Acronym: Analysis Job Definition Language
- Used to define interfaces for high-level services
- Components include:
- Application – executable to process data
- Task – user configuration of application
- Dataset – describes input and output data
- Job – Activity to perform on (or off) the grid
- Typical: app, task and input dataset  output dataset
- Following diagram shows typical component interactions

- Event Data History           David Adams   BNL

- December 2001

## Analysis

- Framework

- Job 1

- Job 2

- Application

- Task

- Dataset 1

- Service

- 1. Locate

- 2. select

- 3. Create or select

- 4. select

- 5. submit(app,tsk,ds)

- 6. split

- Dataset

- Dataset 2

- 7. create

- e.g. ROOT

- e.g. athena

- Result

- 9. create

- 10. gather

- Result

- 9. create

- exe, pkgs

- scripts, code

- ADA/DIAL user interface

- Event Data History           David Adams   BNL

- December 2001

## AJDL (cont)

- Components must be extensible
- Use subtypes
- E.g. HistogramDataset, EventDataset, AtlasEventDataset
- Generic interface
- For use by (shared) generic high-level services
- Experiment-specific interface
- For application and users
- Nature of components
- Persistent representation of data (e.g. XML)
- Classes to interpret this data (C++, Python, java,...)
- Language bindings or re-implementations
- Service or resource (as in WSRF)

- Event Data History           David Adams   BNL

- December 2001

## Application

- Application specifies executable used to process data
- Two entry points
- Extract and build task
- Process input dataset to produce output dataset
- Application + Task = Dataset transformation
- Carries enough information to
- Locate entry points
- Or carry the corresponding scripts
- Enable installation of all required software
- E.g. list of packages for use with package management system
- Might be subtypes for different package management systems

- Event Data History           David Adams   BNL

- December 2001

## Task

- Task carries the user configuration for an application
- E.g. runtime configuration or code for shared library
- Nature of the task specified by the corresponding application
- At present the task is a collection of embedded text files
- Task plus application (transformation) should specify the content of input and output datasets
- Enable users and processing system to
- Verify transformation is suitable for given input dataset
- Avoid staging unneeded parts of input dataset
- Predict the content of output dataset

- Event Data History           David Adams   BNL

- December 2001

## Dataset

- Provides data view
- Generic properties for use in high-level services:
- Location of data (files, DB, ...)
- So data can be staged
- Content
- E.g. for ATLAS events: event ID’s and type-keys (e.g. good electrons) for each event
- EventDataset is an important generic subtype
- Constituents for compound dataset
- Natural boundaries for dataset splitting
- Subtypes provide interface for users and applications to access the data

- Event Data History           David Adams   BNL

- December 2001

## Job

- Interface enables users (and high-level services) to monitor and manage jobs on the grid
- Generic properties
- State: running, succeeded, failed, paused, ...
- Input parameters (e.g. application, task and dataset)
- Result (e.g. output dataset) after completion
- Management
- Pause/resume
- Kill
- Update status
- Job management service to implement these

- Event Data History           David Adams   BNL

- December 2001

## High-level services

- High-level services use AJDL components
- Middleware does not
- Typically high-level services are generic
- Only use generic properties of AJDL components
- Same service for different applications and datasets
- Different experiments or realms can share services
- E.g. LHCb and ATLAS
- Examples
- Analysis (transformation) service
- Job management
- Catalogs

- Event Data History           David Adams   BNL

- December 2001

## Analysis service

- Transformation service might be a better name
- Provides means to create a concrete dataset
- Interface functions
- Request dataset
- Input is application, task and dataset
- Output is job ID
- Associated job carries ID for output dataset
- Fetch job description
- Input is job ID
- Output is job

- Event Data History           David Adams   BNL

- December 2001

## Analysis service (cont)

- Example scenario for processing a high-level job
- Input is application, task, dataset and job configuration
- Map input virtual dataset to concrete representation
- Split into sub-datasets
- Create sub-job for each sub-dataset
- Stage files for each sub-job
- Locate and possibly install application
- Build (e.g. compile) task
- Run sub-jobs
- Gather and merge results to create output dataset
- Register output dataset (including replica)
- Job provides connection to output dataset and detailed job provenance

- Event Data History           David Adams   BNL

- December 2001

## Job management service

- Provide means to manage jobs
- Analysis service creating the job provides this
- May also want this functionality elsewhere
- Accessed from job interface to implement management functions
- Might create job service (OGSI)
- Or job is a resource (WSRF)

- Event Data History           David Adams   BNL

- December 2001

## Catalog services

- Repositories
- Store AJDL components indexed by ID
- Selection (metadata) catalogs
- Help user to select input data, task , ...
- VDC – Virtual Dataset Catalog
- Prescriptions for creating datasets
- Application, task input dataset
- DRC – Dataset Replica Catalog
- Mapping between virtual and concrete datasets
- Job catalog
- Detailed provenance for concrete datasets

- Event Data History           David Adams   BNL

- December 2001

## Implementation strategy

- Define AJDL
- Components, nature, interfaces
- Implement catalogs
- Tables in AMI
- Programmatic interface
- (C++ with Python binding)
- Analysis services
- Start with existing services or analogs
- DIAL, ATCOM, Capone, GANGA, ...
- Different implementations for different strategies
- At least one using ARDA middleware

- Event Data History           David Adams   BNL

- December 2001

## Implementation strategy (cont)

- User interface
- Programmatic interface to high-level services and AJDL components
- C++, python and eventually java bindings
- GANGA will provide python binding and use it to deliver a GUI
- Extensible design: client tools plug into python bus
- Middleware
- Whatever works to begin
- ARDA services will be used in that context
- Like to see better integration with other middleware efforts

- Event Data History           David Adams   BNL

- December 2001

## Implementation strategy (cont)

- Web service infrastructure
- Short term use independent persistent services
- Mid-term follow ARDA strategy
- GAS – grid access service
- Long term follow standards such as WSRF
- Dataset and job become resources?
- Releases
- Deliver working prototype in May
- Robust enough for average physicist
- Regular releases adding functionality, improving performance and incorporating new middleware

- Event Data History           David Adams   BNL

- December 2001

## Effort providers

- Look to the following for effort:
- GANGA for user interface and more
- DIAL for interactive analysis service
- ARDA integration team for ARDA analysis service
- ARDA/EGEE and US grid projects for middleware
- POOL for datasets and metadata?
- SEAL for python-C++ integration
- Later java as well?
- ATLAS physics and computing groups for ATLAS-specific pieces
- ATLAS applications and datasets
- System testing and evaluation

- Event Data History           David Adams   BNL

- December 2001

## ARDA

- ARDA begins April 1
- Two areas in LCG:
- Middleware development (1st report delivered)
- Integration team
- ATLAS ARDA prototype
- Collaboration in context of integration team
- Deliver at least one analysis service base on ARDA middleware
- We would also like to collaborate on AJDL and other high-level services

- Event Data History           David Adams   BNL

- December 2001

## Role of GANGA

- Look to GANGA to provide
- Python binding (or implementation) for AJDL
- Client tools
- Job submission
- Job monitoring and management
- Task management
- Including JOE
- Comprehensive graphical analysis environment
- Including the above client tools
- LCG analysis service?
- Help with system integration and testing
- And more...

- Event Data History           David Adams   BNL

- December 2001

## Connection to LHCb

- To be determined
- This meeting?
- My ideal is that ATLAS and LHCB share a system
- Along lines of the architecture described here
- Most GANGA effort directed toward delivering *generic* high-level services and client tools
- Implications
- Most of the effort expended by GANGA developers is directly usable by both experiments
- Easy for others outside GANGA to contribute pieces
- Use by two experiments validates the idea of generic tools and services

- Event Data History           David Adams   BNL

- December 2001

## More information

- ADA home page:
- _[http://www.usatlas.bnl.gov/ADA](http://www.usatlas.bnl.gov/ADA)_
- This page has links to other projects

- Event Data History           David Adams   BNL

- December 2001