# Presentation: 860615

## Concatenation status

- Rob SnihurSimulation mtg., Jan. 27, 2005.

- Old way (Summer2004)
- Uses 2 tape loggers: fcdfdata004 & fcdfdata0045
- Uses 2 disk buffers: fcdfdata004 & fcdfdata005
- Backlog
- Jqcd1i (800 GB): 50+% done
- Sexo8t (500 GB): 0% done
- Cause: distracted human (working on new concat scheme)
- Additional causes:
- Large samples harder to concatenate
- More jobs crashing
- Disks full  lose 1 tape logger
- Testing new scheme ate up disk space
- Solution:
- Use new disks

##  One DCAF segment:  simulation makes many (small) output files, totaling X GB (where X > 1)
 At the end, concatenate all small files within the DCAF segment
 MCfull:
For (i=istart;i<ied;i++) {
	Call MCProd;
	cp $outfile $outdir
}
Concatenate $outdir

-  At the end, concatenate all small files within the DCAF segment
-  MCfull:
- For (i=istart;i<ied;i++) {
- 	Call MCProd;
- 	cp $outfile $outdir
- }
- Concatenate $outdir

_**New proposal**_

##  Simple
 self-contained
 No need for durable cache
 instead use disk available on worker nodes: how much ?
 No need for separate submission of concatenation jobs
 Less book-keeping
 Average user can do it
 Fewer segments to submit
 Drawbacks
 Need to know event size for each sample in advance
 Could run a 100-event test job

-  self-contained
-  No need for durable cache
-  instead use disk available on worker nodes: how much ?
-  No need for separate submission of concatenation jobs
-  Less book-keeping
-  Average user can do it
-  Fewer segments to submit
-  _**Drawbacks**_
-  Need to know event size for each sample in advance
-  Could run a 100-event test job

_**Features**_

## Ran 80 wtop3i segments in 2 jobs (50+30)
Total segments = 301 (904 run sections)
Expected final output file sizes of 1 GB
Got: 0.2 – 1.4 GB
Used large max runsection size (2000 events),so this probably explains large variation
Kept final output files on disk, allowing validation before inserting to DFC.

First job (50 segments):
45 OK
5 failed to copy output tarball (though output data may be OK)
Got 49 output data files
Missing 6 expected output files
I suspect my knowledge of file naming
Name for a file which starts with run section #2 for lowest run number?

Need to catch failure modes.  Must carefully analyze log files, file sizes, names, etc.

- Total segments = 301 (904 run sections)
- Expected final output file sizes of 1 GB
- Got: 0.2 – 1.4 GB
- Used large max runsection size (2000 events),so this probably explains large variation
- Kept final output files on disk, allowing validation before inserting to DFC.
- First job (50 segments):
- 45 OK
- 5 failed to copy output tarball (though output data may be OK)
- Got 49 output data files
- Missing 6 expected output files
- I suspect my knowledge of file naming
- Name for a file which starts with run section #2 for lowest run number?
**Need to catch failure modes.  Must carefully analyze log files, file sizes, names, etc.**

_**Testing**_

## Submitted   : Thu Jan 20 06:05:35 2005
Ended       : Fri Jan 21 16:06:40 2005
Job duration:  34:01:05

     Wait times:       Max           Min        Abs.Avg.        Start Avg.
               :    12:16:23        0:00:48       0:14:43         0:14:42

  Segment times:      Total           Mean           RMS             Max
           Real:   897:19:30       17:56:47       3:44:15        27:08:19
            CPU:   890:59:51       17:49:11       7:16:57        62:22:12
IO Data summary:
         WaitDH:     0:01:06        0:00:01       0:00:00         0:00:03
           Read:    51015.8M        1020.3M        277.8M         1460.0M

- Ended       : Fri Jan 21 16:06:40 2005
- Job duration:  34:01:05
-      Wait times:       Max           Min        Abs.Avg.        Start Avg.
-                :    12:16:23        0:00:48       0:14:43         0:14:42
-   Segment times:      Total           Mean           RMS             Max
-            Real:   897:19:30       17:56:47       3:44:15        27:08:19
-             CPU:   890:59:51       17:49:11       7:16:57        62:22:12
- IO Data summary:
-          WaitDH:     0:01:06        0:00:01       0:00:00         0:00:03
-            Read:    51015.8M        1020.3M        277.8M         1460.0M

_**CAF email output**_