# Presentation: 009422

## Virtual Data Technologies in ATLAS

**Alexandre Vaniachine**
**Pavel Nevski**
**              ****US-ATLAS Core/GRID software workshop**
**                     ****Brookhaven National Laboratory **
**May 6-7, 2002**

## Pre-VDC Experience

**Recipes for producing the data (DC0 jobOptions, kumacs) has to be fully tested, the produced data has to be validated through a QA step**
**Preparation production recipes takes time and efforts, encapsulating considerable knowledge inside. In DC0 more time has been spent to assemble the proper recipes than to run the production jobs.**
**When you got the proper recipes, producing the data is straightforward**
**After the data have been produced, what do we have to do with the developed recipes? Do we really need to save them?**
**Data are primary, recipes are secondary**

## Virtual Data Perspective

**GriPhyN provides a different perspective:**
**recipes are as valuable as the data**
** ****production recipes are the Virtual Data**
**If you have the recipes you do not need the data (you can reproduce them)**
**recipes are primary, data are secondary**
**Do not throw away the recipes,**
**save them (in VDC)**
**Methods (recipes) should be encapsulated with the data in VD Objects**

## VDC Evaluation Strategy

**Gradual implementatation plan has been proposed:**
***Suppose***** the tools are there (imitate them when necessary)**
**Collect as much components as possible and try to classify there by their functionality**
**Start with event generation and GEANT simulation**
**...**

## DC0 Production System

**Used in DC0 Geant3 simulations**
- High-throughput features:
- scatter-gather data processing architecture
- Fault tolerance features:
- agent-based implementation
- local caching of output and input data (except Objy input)
**Processing time per job: ~24 hours **
**Typical output file size is 200–300 MB**
**	**(170–320 events with hits and digits)
**Typical fault: lost LSF jobs (2-3%)**
- Sometimes LSF batch machines are suddenly switched off
- Occasionally LSF counts time several  (2-3)  times faster then Linux  ***getrusage***
**2 jobs lost due to malloc error in C++ new operator (Objy interface)**
**1 job stopped in GCALOR**

## Data Challenge Features

**ATLAS DC1 phase 1 (starts beginning of May)**
**10****8**** Generator particles events **
**	****(all data produced at CERN)**
**10****7**** Geant3 detector response events (atlsim framework)**
**	****10****7**** reconstructed objects events (Athena framework)**
**Data production is for physics purposes (Trigger TDR)**
**2/3 of data produced outside of CERN**
**production on a global scale:**
**	****	****Asia, Australia, Europe and North America**
**	****	****19 countries**

## VDC Status in DC1

**Approved as an R&D activity (parallel to the production scripts not using VDC)**
**Templated jobOptions approach was used for Generator events production **
**USA site (BNL) will use VDC in simulation production transformation**
**Participants from Canada and UK expressed interest in using VDC-based scripts**

## Production Policies in VDC

**Allocation of unique event ID**
- implementing the event ID allocation policy
**Allocating random number seeds**
- providing unified random number seed allocation policy
**Support for automatic generation of jobs**
- Unique partition numbering
- Encapsulation of environment variables
**VDC database backend ****guaranties**** uniqueness of **
**event ID**
**output PFN**
**random number seeds**
**which is difficult with a non-VDC “perl script” approach in a massive parallel production environment**

## VDC Integration in Production

- Production System is extended in DC1 with features provided by few “ortogonal” VDC component:
**Data reproducibility**
**	****	****SIGNATURE (application software version)**
**Grid dimension:**
**	****	****LOCATION (site)**
**Application complexity**
**	****	****CONFIGURATION (application configuration)**
**VDC-based automatic “garbage collection”:**
**Agents (jobs) get the next derivation from VDC**
**After the data has been materialized**** ****agents register “success” in VDC**
**if some previous invocation has not been completed within the specified timeout period, it is invoked again**

## Virtual Data Catalog

**Proven two-tier architecture:**
- Database backend,  Interface front-end
**That should work! ****Works indeed: in CMS, LIG0, SDSS**