Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
at the Electronics/Online Users’ Workshop

R ECENT B ELLE R ESULTS

fb

32M

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 2 of 25

 events — recorded by July 2001

R ECENT B ELLE R ESULTS

fb

32M

 events — recorded by July 2001

sin2f 1 sin(D mdD t)

Observation of CPV in

decay
Combined

1
0
-1

-8

 -4 0
D t (ps) Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 2 of 25

4First step towards
CKM unitarity triangle

8 !

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 2 of 25

R ECENT B ELLE R ESULTS

sin2f 1 sin(D mdD t)

Observation of CPV in

decay

Observation of"# $&%'$

Belle Xsµµ
Belle Xsee
CLEO Xsµµ
CLEO Xsee

Combined
1

Belle K*µµ
Belle K*ee
CDF K*µµ
BaBar K*ll
CLEO K*ll

0
-1

-8

 decays-4 0
D t (ps) 

4

Belle Kµµ
Belle Kee
BaBar Kll
CLEO Kll

8 !
10First step towards
CKM unitarity triangle

-7

10

-6

10

-5

10

-4

First step onvery rare decays as BSM probe(S UPER KEKB
Activities)*)

Just started — first workshop in Aug. 2001
) )*(note: none of these activities / plans are official yet)Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 3 of 25

(S UPER KEKBSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 3 of 25

Activities()*)

Just started — first workshop in Aug. 2001

Motivation — Flavor physics)*)

Beyond the precision Measurement of CKM matrix)*)
Tool to understand the physics of higher energy and beyond SM (BSM)— As kaon experiments have been.
)*)Complementary to the energy frontier experiments (LHC, LC, . . . )Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 3 of 25(

S UPER KEKB
Activities()*)

Just started — first workshop in Aug. 2001

Motivation — Flavor physics)*)(

Beyond the precision Measurement of CKM matrix)*)
Tool to understand the physics of higher energy and beyond SM (BSM)) )*Complementary to the energy frontier experiments (LHC, LC, . . . )Solution — Luminosity frontier)*)+-,/.

+Upgrade KEKB and Belle forcm 10 s ' ( 2of the current design))*) 3 3%experiment is better in many channels than hadron machines— (BTeV, LHCb, ATLAS, CMS)

ROADMAP
First Step
Discovery of CPV
in -decay




576

2

4 

fb '
+ ,8,cm 90 s '2001 (Now)
Achieved

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 4 of 25

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 4 of 25

ROADMAPFirst Step

Second Step

Discovery of CPV
in -decay




576

2

4 

fb '
+:,8,cm 90 s '2001 (Now)
Achieved

Precision testof KM Model

;

;
<

A B

<

= >>

fb ?'@
A+>:CED
cm ?1F s ?'@2005
Current KEKB and BelleSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 4 of 25

ROADMAPFirst Step

Second Step

Discovery of CPV
in -decay




576

2

4 

fb '
+:,8,cm 90 s '2001 (Now)
Achieved

Precision testof KM Model

;

;
<

A B

<

= >>

fb ?'@
A+>:CED
cm ?1F s ?'@2005
Current KEKB and Belle

Third Step
Flavor physics on BSM;;<< =>>>fb ? @+A >:C/G
cm ?1F s ? @2010?
Super KEKB

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 5 of 25

O KADA M ATRIXO P
QRTS Unitarity
UWVYXZV\[ triangle
Vd[
O S ] ^`_bacVYXZinclusive
RTS Ufe X e [O S ] ^`_bahgjg i
RTS UYk
asymmetryk
O S ]mlndirect
RTS o/k indirect asymmetryO S pqk
O S r ^`_ba e g
O S e gO S st]vu::

H/—SUSY—mSUGRA Minimal General SUSY
Flavor
FlavorMixing Mixing

IKJ
LNM

extra 2HDM extra
quark MHDM dim. . . .

Se

s
e
i
d ressuty s progti
tiv inisnSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 6 of 25

R)

ACCELERATOR U PGRADE

More beam current
0.9 A/0.8 A x 10 A / 3 A(LER/HER)
Squeeze beam moreLo
fo w ErP n
os ergitr y R
on
in
g

(L
E

TSUKUBA Area (Belle)Rgyer
En trongh c
Hi r Ele
fo

HER
LER
Interaction Region

w

g
in
E
(H

y*y
y y
*

RF

NIKKO Area

RFw

OHO Area

y*y
y y
*

(TRISTAN Accumulation Ring)

LE
R
c

na

Li

RF
RF FUJI Areaw-

Positron/eElectron

+

R

e

HE

Squeeze z|}'{ ~/ mm 
Bunch length  mm 

 mm

 mm

New IR design

RF

RF

WIGGLERR)

WIGGLERw

Crossing q mrad 
Crab cavity

Electron cloud

y*y
y y
*

Swap \&

Ante-chamber  mrad

IR UPGRADE

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 7 of 25

New interaction region (IR) design is needed.
(

QCS (final focus quad) closer to IP — No EFC (extreme forward calorimeter)(((

Beam mask design is essential for radiation tolerance, cooling design, and beam
background estimation.
Aiming for 1.0 cm radius beampipe (for better vertexing) Beam-pipe with 1.5
cm radius is OK.

B ELLE DETECTOR

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 8 of 25

1. silicon vertex detector (SVD)
3 layers of double sided sensors
2. central drift chamber (CDC)
50 anode layers
3. aerogel cherenkov counters (ACC),1188 cells, toE` E`
4. time-of-flight counters (TOF)s
4cm thick scintillator, 128 in
5. electromagnetic calorimeter (ECL)
8736 CsI(Tl) crystals]6.
and  detector (KLM)
14 layers of glass RPC in iron yoke
7. superconducting solenoid
1.5 Telsa
8. extreme-forward calorimeter (EFC)
320 BGO on top of final focus quad

B ELLE DETECTOR

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 8 of 25(

1. silicon vertex detector (SVD)
3 layers of double sided sensors
2. central drift chamber (CDC)
50 anode layers
3. aerogel cherenkov counters (ACC),1188 cells, toE` E`
4. time-of-flight counters (TOF)s
4cm thick scintillator, 128 in
5. electromagnetic calorimeter (ECL)
8736 CsI(Tl) crystals]6.
and  detector (KLM)
14 layers of glass RPC in iron yoke
7. superconducting solenoid
1.5 Telsa
8. extreme-forward calorimeter (EFC)
320 BGO on top of final focus quad

Physics goal(
Radiaion tolerance(
Occupancy(
Readout deadtime

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 9 of 25

DAQ35 DESIGN SPEC
4 × 1033 cm-1s-1 (now)More background
More physics
Data size

200 Hz1 - 3 kHz (?)(design: up to 500 Hz)(design: up to 5 kHz)

1 kHz (!)

(BB+qq: 15 Hz)(BB+qq: 400 Hz (!))

100 kB(SVD: 15 kB)(pixel SVD: 50 kB)

10 MB/s

200 - 400 MB/s(design: 15 MB→40 MB)

at storage

1

40 Hz
40 kB

Data flow at L1

1 × 1035 cm-1s-1 (×25)

5 MB/s4(design: 15 MB→24 MB)

2(design: 600 MB/s (!))

50 - 100 MB/s 3(design: 240 MB/s)

4
1 Background rate should not scale to luminosity... probably scale to the beam current.
2 Data size will increase due to a larger occupancy.
3 A factor of 2 more reduction with software trigger, prescaling two-photon/ττ/µµ/eeγ.
4 After DAQ upgrade in summer 2001 (now)Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 10 of 25

E LECTRONICS AND DAQ NOW

SVD
front-end
electronics

VME
Flash ADC

CDC
front-end
electronics

Q-to-T
converter

TDC

MasterVME(

Q-to-T
converter

TDC

MasterVME

Q-to-T
converter

TDC

MasterVME

Q-to-T
converter

TDC

MasterVME

hitmultiplexor

TDC

MasterVME

Q-to-T
converter

TDC

MasterVME

ACC(

Front-end systemsince beginning
New event building farm + data
storage system
just replaced

front-end
electronics

Basic features of Belle trigger/DAQ
Level-1 trigger
Redundant triggers (track/neutral)
2.2 µs latency
Simple and robust DAQ
no pipeline readoutunified TDC readout with Q-to-T

TOF
front-end
electronics

ECL
front-end
electronics

KLM
front-end
electronics

EFC
front-end
electronics

TDC

TRG
subsystem
trigger

globaldecision
logic

MasterVME
Sequence
Controldata flow
trigger flow

Event
Building
Farm

3km

Data
Storage
SystemTape
Library

Basic specifications
200 Hz L1 trigger rate
(500 Hz maximum)
30 KB/event (15 MB/s),
down to the tape<200 µs deadtime(<10% at 500 Hz)

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 11 of 25

L1 TRIGGER NOW
Level-1 trigger

Pipelined— 32/64 MHz clock
SVD trigger— to be added soon

Cathod Pads
Track countSegment

Track count

TOF

Hit

back-backtimingCluster count

ECL

Threshold

TimingEnergy sum

Threshold

4x4 Sumtiming decisionmultiplicity

48 prescale & mask

Axial Wires48 subtrigger decision

CDC

Z finderStereo Wires

BhabhaGlobal Decision
Logic (GDL)

KLM

Hit

µ hit

EFC

PreampLow threshold

BhabhaHigh thresholdTwo photon

2.2 µsec after event crossing

to readout
seqeunce
controller(SEQ)

Q- TO -T + FASTBUS SYSTEM NOW
Trigger (2.5us)(+ delay)Event
preamp.
signalSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 12 of 25

Q-to-T (charge to time) converter
LeCroy MQT 300A Q-to-T chip
CDC / ACC / TOF / ECL / EFC

TDC readout
LeCroy LRS1877S FASTBUS multihit-TDC
built-in sparsification functionegr
CDC / ACC / TOF / ECL / EFC / KLM / TRGchaDis
KLM : hit map is serialized
TRG: all the timings are recorded
Unified VME readout system
Down
Up
CommonMotorola MVME162(VxWorks) controls
edge
edge
stop
Homemade FASTBUS/VME interface with DPM.
The same software for all the TDC systems
5.5 MB/s TDC readout / 3.5 MB/s to Event builder
Two timings are digitize and read out.
80µs deadtime/event

Hold
Q-to-T
capacitor

Q-to-T
outputmultihit
TDC

trigger from SEQ

Controller

fromQ-to-T
frontend

FASTBUS (up to 6 crates)

Dual Port Mem.

Event Bulder Tx

FASTBUSPROCESSOR
Timing Dist.

TDC

DMA
VME (1 crate per detector)

to EB
switchSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 13 of 25

VA1 + FADC SYSTEM NOW
Flash ADC to readout frontend VA1-chipsparsification by onboard DSP
4 VME crates to readout flash ADC
VA1 latch at 1µs by the TOF cluster pre-trigger (a few kHz)trigger

150 µs / event fixed deadtime for delayed VA1 readout.

CDC gate for Q-to-T
SVD VA1 readout
control VME
150 µsController
Timing Dist.

VA1 readout clock noise (crosstalk) in the CDC readout.
readout VME

from
VA1
frontend

Flash ADC

Event Builder Tx

triggerfrom SEQ

to EB
switchSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 14 of 25

E VENT B UILDING FARM NOW
100Base-T local area network
100Base-T Data flow (0->1)
100Base-T Trigger flow (1->2)
SEQ

1000Base-SX Data flow

TRG

1000Base-LX Data flow (3km)

TOF

PC

PC

Switchless EB-farmfrom 2001 fall
first physics run
from today!

CDC 1
Experiment
hall

CDC 2

Tunnel Computer
center

layer0  1— 5 MB/s/link

SVD 1
SVD 2

PC

PC

PC

SVD 3
SVD 4

sw

sw

sw

sw

layer1  2— 15 MB/s/link

WS

ACC
ECL



PC

PC

PC
(DQM)

KLM

Ultra Wide SCSIEFC
SONY DTF2...
Local area network

layer 0
VME readoutlayer 1
partial event
building + level-2
reconstruction

layer 2level-2
filteringlayer 3
final event
building + level-3
filteringlayer 4
mass storagelayer2  3—  50% reduction
layer3  4— 24 MB/s

E LECTRONICS FOR SUPER KEKBSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 15 of 25

Current DAQ system will not work with 10 times higher background rate.
At 5 kHz, deadtime  + s is needed.


L1 latency (2.2  s) is too long if trigger comes at 5 kHz.
Need pipeline buffer (analog/digital) before L1 decision.

No software based readout system.
Need large FIFO buffer (with sparcification) for a delayed readout.
Individual systems:
*

SVD — Occupancy will be critical (4%  40%?)
Analog pipeline should be in front-end
*
ECL — Need faster shaping time
Pileup effect *
CDC — Inner part should be replaced with SVD.
  ¡!T¢
measurement. (similar system for PID/KLM).



SVD ELECTRONICSClosely related to the new SVD design(vertex resolution, occupancy,
radiation tolerance, cost, . . . )Pixel or not — finer innermost layer
to reduce occupancyExtended outer layer — drift chamber
will not work there
Analog pipeline on the readout chip— not so easy task

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 16 of 25

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 17 of 25



ECL ELECTRONICS

Use the same CsI(Tl)— OK in terms of radiation(too costly to find and replace with a
new/better crystal)The optimal shaping time is too slow,
if pile up is considered.
Lots of low energy pile-ups( a few MeV  £  ¤ ¥ GeV ¦ ).
Need much shorter shaping
multi-range flash-ADC+ waveform sampling.

Hit
preamp.
signal
Hold

6µs
until
peak
max

100ns
hold time1§ s shaping time¨
6§ s peak time¨
2 MeV equiv. pile up

CDC ELECTRONICS

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 18 of 25

Use similar drift chamber — OK in terms of occupancy.


TMC (time-memory chip) is a good candidate for readout.
9 ¡!Q¢

should be carefully considered.

Similar readout can be used
for PID and KLM (but a new
PID detector may have to berebuilt).

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 19 of 25

E LECTRONICS SUMMARY

almost of all detectors should beredesigned
hit rate/ch or
occupancy

typicaltime scaledetector@x100

of detector

present x10x100

B:4Hz B:40Hz <5kHz
KLM E:>40Hz E:>400Hz

CsI

noise

<3

noise

<10

noise

1msec/cm2 <1/10:op-mode & electronics
op-mode)(<1/factor:segmentation>900nsec

TOF 30kHz 300kHz

3MHz~50nsec<1/50:segmentation&electronics

10kHz
ACC B:1kHz
E:5kHz 50kHz

100kHz500kHz~50nsec<1/10:segmentation

50kHz
CDC 5kHz*w/o inner <10%500kHz~400nsec(cell size)

100%

0.1~
1µsec
(noise)SVD 20kHz
4%
EFC

40%

even ifx10 fine granularity
is necessary©

DAQ BACKEND

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 20 of 25

Readout system©

ª*ª

Non-VME system can be used if realtime software is not needed.
— Serial line readout (USB2.0 or IEEE1394)

EB-farm©

ª*ª

No much room to add EB-farm nodes to cope with « 10 data rate.
— Event distribution and multiple EB-farms running in parallel.

Storage systemª*ª

No single drive can write at a rate of « ¬+­ data rate.
— Disk caching and delayed writing into multiple tape drives.

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 21 of 25

R EADOUT UNIT

map onto a linear memory,or align into a serial line.

Read out by a PC

10-20 readout module / crate~1-2K channel / unit for ECL / CDC~10-20K channel / unit for SVD

CPU?
Timing system?
Readout module?
Readout module?
Readout module?
Readout module?
Readout module?
Readout module?
Readout module?
Readout module?
Readout module?
Readout module?

Data readout/transfer
using serial-bus(USB2.0 or IEEE1394).

Asynchronous
trigger-id buffer

Readout Unit
Synchronous
1 PC + 1 box + 1 timing systemring buffer
trigger-id
Backplane system (9U VME?)
store if data upon trigger
is convenient for misc. purposes.

Asynchronous
FIFO buffer

digitization
every clock

Daisy chain

GbE

10-20 MB/s (5kHz) throughput
No other conenction to PC(no timing info at PC)

PC

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 22 of 25

M ULTIPLE EB- FARM

~30 PCsx 20 MB/s~600 MB/s
CDC
TOF
TRG
SEQ
SVD1
SVD2
SVD3
SVD4
ECL
KLM
ACC
EFC

~30
switches
150
cables

~15 PCsx 40 MB/s
L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

~5 PCsx 48 MB/s~240 MB/s

One of the possible configuration,with no much challenge for per-node speed.

L3

1-to-5 splitting (GbE to FastE) and10-to-1 bundling (FastE to GbE),
with inexpensive network switches.

~5x7 PCs

Scalable up to 1-to-N (N~O(10)) splitting
L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

~15
switches layer layer
1
2

L3

L3

layer
3

May need some network techniquessuch as QoS (RSVP) and/or VLAN.
(need to confirm).

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 23 of 25

D ISK CACHING
Tape drive candidates:®
SONY DTF3 (48 MB/s) in 2002
DTF4 (96 MB/s) in 2006?
Switching disks

No single drive at 240 MB/s
No data transfer, either?
Disk cache and delayed write
If input speed = output speed, 3 PC is enough

Need multi-port disk ?
PC with multi-port
network + network switch

If input speed = N * output speed, N+2 PC are needed
If 10 PC for 5 drives, disk-to-tape connections are fixed,
and some margin for storage is provided.

Unit PC for disk caching
200 GB
2-port~50 MB/s x GbE

GbE-SW

x50 PC (200 GB disk/PC)
Fast enough disk? If not, RAID?

x10 1-to-10 full GbE SW
GbE-SW

Super KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 24 of 25

DAQ35 FULL CONFIGURATION

Belle Detector~30 PCs~5x7 PCs

SEQ
CDC1
CDC2
CDC3
CDC4
CDC5
TOF
TRG1
TRG2
TRG3

many
cables

SVD1
SVD2
SVD3
SVD4
SVD5
SVD6
SVD7
SVD8
SVD9
SVD10

150
cables

ECL1
ECL2
ECL3
ECL4
ECL5
ECL6
KLM1
KLM2
ACC1

Trigger andTiming Systems

ACC2

30 fibresfrom EH~45
switches

L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

L1
TRK

L2
TRK

L1
VTX

L2
VTX

L1
NEU

L2
NEU

~50 PC with disks
L3

5 PC withtape drives

L3

50
cablesL3

50cables
5 fibresto CRC
L3

L3

©

S UMMARYSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 25 of 25

The Belle data acquisition system has been so simpleª*ª

Working very stably with quite a small number of operation crews.
ª ª*Reasonably good choice for the Belle I.

©

S UMMARYSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 25 of 25

The Belle data acquisition system has been so simpleª*ª©

Working very stably with quite a small number of operation crews.
ª ª*Reasonably good choice for the Belle I.
The Super KEKB DAQ demands a complete replacement of the electronicsª*ª

New idea is needed, new R&D efforts have to be made.
ª*ª
The dataflow part will be rather easy (thanks to IT).
ª ª*
Still the system is much simpler than hadronic machines— technologies should be borrowed from those people.

©

S UMMARYSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 25 of 25

The Belle data acquisition system has been so simpleª*ª©

Working very stably with quite a small number of operation crews.
ª ª*Reasonably good choice for the Belle I.
The Super KEKB DAQ demands a complete replacement of the electronicsª*ª

New idea is needed, new R&D efforts have to be made.
ª*ª
The dataflow part will be rather easy (thanks to IT).
ª ª*
Still the system is much simpler than hadronic machines— technologies should be borrowed from those people.
ª ª*Lots of help from Electronics group.

©

S UMMARYSuper KEKB Data Acquisition System
Mikihiko Nakao (KEK)
5 October, 2001
Page 25 of 25

The Belle data acquisition system has been so simpleª*ª©

Working very stably with quite a small number of operation crews.
ª ª*Reasonably good choice for the Belle I.
The Super KEKB DAQ demands a complete replacement of the electronicsª*ª

New idea is needed, new R&D efforts have to be made.
ª*ª
The dataflow part will be rather easy (thanks to IT).
ª ª*
Still the system is much simpler than hadronic machines— technologies should be borrowed from those people.
ª ª*Lots of help from Electronics group.
So far, and so will be.