Booster Space Charge Modeling/Simulations Review

September 17, 2003

The results of a Booster Space Charge Modeling/Simulations review are
reported. It was an internal Fermilab review organized by the Computing
Division. The topic was a major project to simulate particle beam
behavior, including a full three-dimensional treatment of space charge
effects, in circular accelerators such as the Fermilab Booster.

The members of the committee are:

Chuck Ankenbrandt (Chair) \<ankenbrandt@fnal.gov\>

Paul Lebrun \<lebrun@fnal.gov\>

David McGinnis \<mcginnis@fnal.gov\>

Jean-Francois Ostiguy \<ostiguy@fnal.gov\>

Vladimir Shiltsev \<shiltsev@fnal.gov\>

Mike Syphers \<syphers@fnal.gov\>

At the beginning of the review, the committee received the following
"Abstract of the Work" to be reviewed:

> *The Advanced Accelerator Simulation Project, which is a
> multi-laboratory collaboration effort, is funded by the DOE SciDAC
> program. The goal of the project is to develop a suite of scalable and
> portable simulation software that can be used to solve challenging
> problems in accelerator design, analysis and optimization.*
>
> *Fermilab physicists on the project have concentrated their efforts on
> modeling space charge effects in circular machines, specifically in
> the FNAL Booster, for high intensity operation scenarios where the
> effects are extremely important. The current program has focused on
> modeling of the Booster as well as performing beam studies with the
> machine. Data and simulations are compared in an attempt to both
> validate the model as well as to understand the performance of the
> Booster. In the process of carrying out the program, the Ion Profile
> Monitor, which is essential for understanding the evolution of the
> beam during the Booster cycle has been calibrated.*
>
> *Modern software techniques have been introduced into the
> beam-modeling package, Synergia. Synergia re-uses existing beam
> physics codes, spans multiple languages, and provides an extensible
> framework. The code has a flexible human interface based on python and
> a portable build system which uses the GNU Autotools.*

On May 27^th^, the committee heard a series of presentations according
to the following agenda:

> *\* Presentation of Charge - Pushpa Bhat*
>
> *\* Overview of the Project - Panagiotis Spentzouris*
>
> *\* Technical Implementation/Code Development - Jim Amundson*
>
> *\* Applications and Plans - Panagiotis Spentzouris*
>
> *\* Q & A*

In addition, the committee had access to project documents contained on
this web page:

> *http://cepa.fnal.gov/psm/aas/Advanced_Accelerator_Simulation.html*

The following documents in the Beams Documents Database,
\<http://beamdocs.fnal.gov/\>, were also made available:

> *\> Beams-doc-495-v2*
>
> *\> Beams-doc-581-v1*
>
> *\> Beams-doc-499-v1*
>
> *\> Beams-doc-498-v1*
>
> *\> Beams-doc-497-v1*
>
> *\> Beams-doc-496-v1*
>
> *\> Beams-doc-615-v1*
>
> *\> Beams-doc-616-v1*

The charge to the committee consisted of the following items:

> *(1) Are the simulation techniques used in Synergia appropriate for
> evaluating space charge in the Booster and other circular machines?*
>
> *(2) A well planned validation with data is critical for any
> simulation program. Has the validation program using Booster IPM data
> resulted in a useful test of the Synergia simulation package?*
>
> *(3) Have sufficient beam studies with the Booster been made?*
>
> *(4) Are the extrapolations to high intensity operation scenarios
> valid?*
>
> *(5) Is the software suite user-friendly and efficient?*
>
> *(6) Please comment on the plans for further simulations and studies.*

## General remarks

A few general remarks are in order before responding specifically to the
charge.

Charged particle beams generate electric and magnetic fields that act
back upon the particles in the beam, causing so-called space charge
effects. Except in the trivial case of an unbunched cylindrical beam of
uniform density, the forces resulting from these fields are nonlinear.
These intensity-dependent nonlinear forces can disrupt the beam in a
synchrotron via a variety of mechanisms. These phenomena are understood
qualitatively, but only simulations can provide quantitative estimates
of the relative importance of the various effects in realistic
situations.

Synergia is a package based on IMPACT, a code originally developed at
Los Alamos to study space-charge induced halos in high intensity linacs.
A substantial part of the code has in fact been re-used, and the
Synergia team wrote "glue" components, reducing development cost.
Despite the technical difficulties encountered in assembling Synergia
from components originating from different sources and implemented in
different languages, the committee is supportive of the approach. An
advantage is that users can begin using the code immediately while the
team continues to acquire experience.

In addition to writing a simulation package, Panagiotis Spentzouris and
Jim Amundson have also conducted experiments at the Booster and compared
the results with their simulations. They should be commended for a
significant amount of high quality work. Very nice preliminary results
were shown.

The Synergia team has, in a relatively short time, had a significant
influence on some Booster operational issues. They communicate regularly
and collaborate closely with Beams Division personnel. This type of
collaboration has historically been difficult to accomplish and should
be supported. Simulation of space charge effects is important not only
for existing machines like the Booster, but also for possible future
high intensity proton synchrotrons. Participation of the laboratory in
SciDAC is important in terms of scientific contributions to the field of
computational accelerator physics and in terms of the establishment of a
critical mass of local expertise. Such expertise is important for the
study of future projects and also will determine in large measure how
successful the Laboratory will be in attracting future government
funding for computational activities.

## Specific responses to the charge

Here are the responses to the specific issues raised in the charge to
the committee:

\[1\] Are the simulation techniques used in Synergia appropriate for
evaluating space charge in the Booster and other circular machines?

The simulation techniques are sound. The space-charge algorithm used in
Synergia/IMPACT is well established and suitable to run on large
parallel systems:

the electric potential is calculated in the beam frame by convolution of
a free-space Green\'s function with the bunch charge distribution. The
convolution itself is performed using 3D FFTs.

While the convolution algorithm is certainly appropriate, one should not
get the idea that this is the only competitive method available (the
Fast Multipole Method is an example of another viable algorithm). One
issue is that in order to preserve accuracy, the spatial discretization
must be performed on a relatively uniform grid. Since bunches in
circular machines are typically much longer than their transverse size
and since it is important to model the details of the potential
variation along the transverse dimension,

a correspondingly high sample density is required in the longitudinal
dimension, substantially increasing computation time. The developers of
IMPACT have recently attempted to improve the algorithm by using special
non-Fourier longitudinal expansions, but so far, the reported
performance has been disappointing. Computational efficiency is an
important issue for simulations over a complete acceleration cycle. For
example, a single Booster turn with a million macroparticles currently
represents about an hour of computer time on a modest number of
processors (8-16). Granted that transverse space charge effects are most
important early in the cycle, still the beam makes about 20,000 turns
during an entire acceleration cycle in the Booster.

\[2\] A well-planned validation with data is critical for any simulation
program. Has the validation program using Booster IPM data resulted in a
useful test of the Synergia simulation package?

Panagiotis and Jim have done a very good job of understanding,
calibrating, and using the Ion Profile Monitor in the Booster. The IPM
measures the transverse beam size distribution, and hence indirectly the
emittance, by collecting ions created by the passage of the beam through
the residual gas. In the presence of space charge, one expects the
emittance to grow with a rate that increases with beam intensity. This
is a direct and observable measurement of the effect of space charge.

Numerically, insufficient macroparticle statistics cause \"artificial\"
emittance growth that must be distinguished from real emittance growth.
This may be a problem when growth due to physical mechanisms is already
relatively small. The IPM is not necessarily the best device to observe
the tails of the beam distribution, the halo region, with sufficient
resolution. Another problem is that the relative gains of the channels
where the ions are collected vary. Typically, the channels that receive
the most ions have a reduced gain.

The main challenge for interpreting IPM data is that the ions do not
travel in parallel paths to the collection grid. Rather, the beam
self-field causes the trajectories to curve. As a result the measured
transverse profiles are distorted in an intensity-dependent way. A
correction must be computed and applied a posteriori. To compute the
correction, one must necessarily make an initial assumption about the
(unknown!) beam shape. The analytical and numerical work of the Synergia
team has led to a much improved IPM correction algorithm.

The committee (favorably!) views the comparison of Synergia results with
Booster IPM data primarily as the (successful!) beginnings of an attempt
to do physics rather than a software validation program. The effort to
simulate a real machine like the Booster is a powerful motivator for the
addition of needed features to the software. The Booster study results
might also be construed as evidence for the validity of Synergia.
However, the Booster is too complicated and not well enough understood
to provide convincing code validation. Also, there are large
uncertainties in the measurements and in the knowledge of beam initial
conditions. The IPM is a relevant diagnostic; however, good agreement
with IPM data should not be regarded as a conclusive test.

The Synergia team should participate in the overall SciDAC code
validation program. This is in their work plan, and they should be
strongly encouraged to do so, even if such validation programs have
already been run on IMPACT, or have no direct bearing on the Booster
performance. Such validation tasks not only are crucial to keep
re-assessing the validity of Synergia as it keeps evolving, but also
will strengthen their reputation in the accelerator simulation
community. These validation tests should also be maintained and
documented as \"potential examples\".

At an ICFA Mini-Workshop in Oxford, England in April of this year,
several of the participants agreed to a joint effort on space charge
code benchmarking. Data from a CERN PS experiment on the Montague
resonance will be used. The Synergia team should take part in this
comparison activity.

It would also be important to validate the code using some standard
simple cases. The most obvious one would be a simulation of a beam with
a K-V distribution propagating in a continuously focusing machine. In
that case, all the space charges forces are linear and calculable. The
steady state solution is an unstable equilibrium.

\[3\] Have sufficient beam studies with the Booster been made?

The data acquired so far during dedicated beam study periods have proven
very useful for stimulation of code development and then for comparison
with Synergia results. However, dedicated Booster beam study time is a
scarce commodity. More Booster beam studies are warranted, but
pragmatism requires that most future studies be parasitic. Parasitic
study time is readily available, but such studies cannot deviate too
much from normal running conditions. That is not much of a limitation
because those are the most interesting conditions anyway if the goal is
to understand the real machine.

In particular, "normal running conditions" means magnets ramping and rf
cavities accelerating. Although acceleration has been implemented in
Synergia, much of the data and Synergia simulations to date have been
performed with magnet currents DC and rf off. Future studies should
involve mostly parasitically acquired data during normal acceleration
cycles with rf on. The effects of varying the intensity and other
parameters early in the cycle should be studied under those conditions.
Once there is confidence in the ability to properly handle acceleration,
then transition crossing could also be studied. Although longitudinal
space-charge effects at transition are well known, the possible
interaction of transverse and longitudinal effects at transition is
virgin territory.

\[4\] Are the extrapolations to high intensity operation scenarios
valid?

The committee is not in position to answer this question regarding the
specific work under review because the Synergia team did not present
numerical results related to high intensity extrapolations to the full
committee. (A private conversation with one committee member after the
presentations revealed that they actually ran the code at higher
intensity and saw more halo created, but did not present quantitative
results because no experimental data are available for comparison.)

Although the committee lacks specific information regarding Synergia
extrapolations to high intensity, some general comments are in order. A
machine like the Booster is a complex object. Space charge is only one
of several factors influencing performance. A complete model would
include various magnetic field errors at injection (remanent fields),
field defects due to saturation, magnet alignment errors, beam coupling
impedances, the effects of transition crossing and coherent
instabilities, and so forth. All these factors must be studied both
separately and concurrently, as they can sometimes conspire. Reliable
quantitative predictions about performance at intensities well beyond
current experience would require a model that not only incorporates all
the important physics but also includes correct "initial conditions"
such as an accurate knowledge of field errors. At the present state of
the art, quantitative agreement of such predictions with reality would
probably be fortuitous in most cases.

Nevertheless, the committee believes that Synergia will be quite useful
for understanding the role played by space charge in high intensity
circular accelerators. The results of such simulations are particularly
valuable for guiding design decisions for future accelerators and
upgrades of existing ones. Such design decisions can be based on the
relative performance of various possibilities, even if the absolute
predictions are uncertain.

\[5\] Is the software suite user-friendly and efficient?

As mentioned above, the algorithm at the heart of Synergia is a standard
and well-established one. Efficiency in a parallel context is a matter
of dynamically balancing the load on each processor. IMPACT load
balancing has been carefully optimized over the years.

Panagiotis and Jim have made significant efforts to integrate existing
software components. The technique employed, a python wrapper
communicating through files, was the most expeditious choice. Although
the committee has not tried to use the software, the presentations
suggested that running Synergia should be a straightforward matter for a
knowledgeable person.

Because Synergia is a mixed language package (f90/C++/python), some
increased complexity in terms of installation, support and maintenance
is to be expected. That should not be a serious issue; furthermore, a
reasonable plan was presented for better integration of the python front
end in the future. Some web-based documentation is already available and
needs to be expanded.

\[6\] Please comment on the plans for further simulations and studies.

In addition to the measures embodied in the discussions above, the
committee recommends that the Synergia team pursue the following future
activities. The.key words here should be consolidation and strengthening
via a vigorous program of Booster studies and software augmentations.

Regarding simulations, a plan for rewriting the main IMPACT loop has
been presented and looks good. The Synergia team should consider
rewriting this main loop in C++ rather than Python, thereby restricting
the usage of Python to the interface between the input script and the
core package, because this main loop may contain more interfaces than
presently foreseen, and C++ looks more appropriate for such a
\"framework\" task.

As previously emphasized, the ability to simulate acceleration must be
established. (This is implemented at some level, but has not been
extensively tested.) RF capture, transition phase jumps, and energy
dependent field errors should be included. Also, Synergia could possibly
be expanded to include more realistic machine impedances.

Improved numerical diagnostics should be implemented. This is important,
since at the moment Synergia provides only phase space coordinates for
all particles. To compare with experimental and theoretical results,
users expect to be able to compute moments at various locations around
the ring through an acceleration cycle. This can be done offline by
dumping phase space data periodically, but that arrangement is
cumbersome. Another important diagnostic is the ability to produce
single particle tune footprints since those can be correlated with
theoretical results.

Regarding beam studies, a comprehensive long-term plan that emphasizes
parasitic studies should be developed.

After the present shutdown a vertical IPM will be available in addition
to the previously used horizontal one. The horizontal beam profiles are
complicated because they reflect both transverse radial and longitudinal
phase space distributions. Separating out the evidence for transverse
emittance growth from the effects of the rapidly evolving momentum
distribution is an interesting challenge. However, interpretation of the
vertical profiles in terms of space charge induced emittance growth will
be more straightforward.

The Synergia team should also develop a coherent plan to consolidate
their analysis and studies. In particular, so far they have avoided
drawing definite and quantitative conclusions regarding the emittance
dilution due to space charge in the Booster as a function of time and
intensity. Various plots show that information implicitly. However, a
quantitative and conclusive summary, with a serious attempt at
determining the systematic errors in both measurement and simulation,
could be written up. Maybe they need more beam studies to reach that
stage. If so, they definitely deserve the necessary support for
conducting such studies.

Finally, the idea of adding a graduate student or postdoc to the
Synergia team was briefly discussed. Such an expansion is warranted not
only because the project would make faster progress with more manpower,
but also because the activity would be very educational. The committee
encourages lab management to seriously consider that possibility.