24 january 2011 / Concrete international

by Jeffrey W. Bullard, Edward J. Garboczi, William L. George, Nicos S

Steven G. Satterfield, and Judith E. Te

Advancing the 

Materials Science of 

Concrete with 

Supercomputers

Adding to industry knowledge using complex and interactive

simulations of rheology and hydration

Simulations on supercomputers are used to enhance 

our understanding of complex systems such as global 

weather patterns, nuclear explosions, the birth of galaxies, 

molecular structure, and the rheology and hydration 

of concrete. These simulations are our current best 

representations based on fundamental physical, chemical, 

and mathematical principles.

Why does concrete need to be simulated on a supercomputer? Concrete is arguably, from a materials science 

point of view, the most complex material produced by 

humankind with distinct, important, and random structural 

features on length scales ranging from nanometers to 

hundreds of millimeters. It has many mineralogical 

phases; its behavior is time dependent; it’s formed from 

natural materials whose properties are quite variable; 

and it’s sensitive to its surrounding environment, which 

of course drastically varies around the world. For such 

a material, reliably and optimally manufacturing its 

constituents, mixing and placing the actual concrete, and 

then guaranteeing the appropriate finished properties are 

great challenges. Simulating this complex material is an 

even greater challenge.

Correctly modeling the rheology and hydration of 

concrete can help solve many problems that continue to 

bother the concrete industry. The National Institute of 

Standards and Technology (NIST) has made progress on 

these two problems by using massively parallel computers 

that drive advanced scientific models. This work has 

required sophisticated physical science to develop the 

models and sophisticated computer science and hardware 

to run the models effectively. Basic insights from these 

models will drive the development of more accurate and 

reliable standard measurement techniques. 

This supercomputer-based research is closely coupled 

to NIST’s Virtual Cement and Concrete Testing Laboratory 

(VCCTL) consortium, which was established in 2001 to 

bring concrete industry engineers and scientists together 

with NIST researchers to accelerate the development of a 

virtual testing software environment, mainly working at 

the micrometer-millimeter scale.1

 In its ninth year, the 

VCCTL project has recently been awarded the U.S. 

Department of Commerce Silver Medal “for creating the 

unprecedented capability to predict the performance of 

concrete—the key material used in the U.S. physical 

infrastructure.” The consortium has also won two 

Department of Energy (DOE) Office of Advanced Scientific 

Computing Research (OASCR) awards for excellence in 

3-D visualization of the cement and concrete models 

described herein.

Rheology and Hydration M

Modeling the rheology (flow) of concrete is very 

difficult because fresh concrete is composed of particles 

that range from less than 1 mm to 10 mm in size. Our 

multiscale models simulate flow at different length scales 

by adding particles to a representative matrix fluid that 
Concrete international/ january 2011 25

models the particles’ behavior. At the smallest scales of 

interest, water is the matrix fluid, and cement powder is 

added to the water to model a cement paste. The cement 

paste then becomes the next continuum matrix fluid, to 

which sand particles (and air voids) are added to model a 

mortar. The mortar’s flow properties are input into a flow 

model of concrete, where fresh mortar is the matrix fluid 

and coarse aggregates are the inclusions.

Particle shape at all length scales must be considered 

when predicting flow properties. For example, riverbed 

aggregates are somewhat rounded but crushed aggregates 

can be angular with faceted sides; this shape difference 

affects the rheology. The same holds true at the cement 

particle scale. Our computational models are based on 

novel methods for modeling fluid flow.2,3 We’ve developed 

models that simulate flow at the particle scale, taking into 

account variable particle shape and size, forces between 

particles in a dense suspension, and matrix fluids having 

complex flow properties.

The hydration of cement paste depends on the cement 

particle size distribution; cement mineralogy; the watercement ratio; the presence of any chemical admixtures 

such as retarders, accelerators, or high-range waterreducing admixtures; and curing conditions such as 

temperature and relative humidity. A model of cement 

hydration must be able to predict the rate of hydration 

and the development of the binder structure as a function 

of all of these variables. The hydration process involves 

combining the diffusion of molecules in solution with 20 

or more chemical reactions. Most of the reactions are not 

well understood, and there is some debate about their 

reaction mechanisms. In addition, hydration reactions 

occur at the nanometer scale but eventually give rise to 

micrometer-scale solid features in the microstructure 

that must also be accurately modeled so that setting time 

and mechanical and transport properties can be computed.

To address these challenges, we have developed a 

kinetic cellular automaton model that simulates the first 

few hours of cement hydration, when the chemical and 

microstructure changes are most pronounced. This 

model simulates the fully coupled diffusion and reaction 

phenomena, and we have used it to gain insight into 

reaction mechanisms and their influences on structure 

development in concrete binders at early ages. Other 

related models exist for simulating later-age structure 

and property development. 

Models and Computer S

Both of the aforementioned models have been designed 

to run on large multiprocessor (also known as highly 

parallel) machines (Fig. 1). To use a “parallel” computer, 

a large computation must be split up into smaller parts 

with each part run by a separate processor. In principle, 

this can speed up the program by a factor equal to the 

number of processors used, although in practice there 

are always practical limitations. We could not run models 

that were large enough for concrete without the power of 

parallel algorithms and parallel computers.

For both models, we are simulating a physical system, 

contained in a 3-D volume; we are moving the simulation 

forward in time by small time steps. The algorithms use 

the local conditions (such as chemical concentrations 

and fluid velocities) surrounding any particular point in 

the model volume to predict the state of the system at 

that point for the next time step. This means that we can 

divide the simulation volume into subvolumes that are 

Fig. 1: Two highly parallel machines that have been used in the 

NIST simulation program: (a) Columbia, a cluster of 24 SGI Altix 

machines, each containing 512 processors, maintained by the 

National Aeronautics and Space Administration (NASA); and (b) 

Intrepid, an IBM Blue Gene/P with 163,840 processors, maintained by Argonne National Laboratory. Time on these machines 

was awarded to us by the NASA National Leadership Computing 

System initiative and by the DOE Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, 

respectively. The machines are identified only to adequately 

describe the project—the identification is not an endorsement or 

recommendation by NIST, nor does it mean that the equipment is 

necessarily the best available for the purpose

(a)

(b)
26 january 2011 / Concrete international

each controlled independently by a single processor. So, 

for example, if we’re running a simulation on 1000 

processors, we can arrange these processors into a 103

size mesh, where each processor handles one subvolume, 

potentially speeding the computation by a factor of 1000.

Figure 2 shows a simple four-processor (potential 

speed-up of four times) setup in two dimensions. Each 

processor controls its own square area, and the points 

are some places where data is generated and updated at 

each time step. The circles around the points denote the 

local region that affects how the data at the point changes. 

The black points and their surrounding regions are 

totally within a processor’s subvolume, so it has all the 

information it needs to predict its next time step. However, 

the red point has its data region crossing the boundary 

between processors, so Processors 1 and 2 will need to 

pass information back and forth to update the data at the 

red point, reducing the increase in speed to something 

less than a factor of 4. Because of this communications 

overhead, high-speed networks for information flow are a 

crucial part of supercomputing; and because communication 

rates are usually slower than computation rates, the real 

speed-up is typically less than the number of processors. 

In fact, as we use more processors to simulate a particular 

system, each processor’s subvolume becomes smaller 

and at some point, adding more processors to a simulation 

will no longer improve the execution time because the 

overhead of interprocessor communication outweighs 

the benefit of having more processors to do the work.

In the flow model, both the fluid and the suspended 

inclusions (cement, sand, or gravel) are composed of 

smaller particles. As mixing forces are applied, the fluid 

particles and the inclusions begin to move within the 

volume. As these particles move within the simulation 

volume, they pass from one subvolume (or processor) to 

the next. The location and velocity of each particle are 

then updated based on the forces computed. The fluid 

stress on each inclusion and the total system stress are 

computed to give global rheological variables that can be 

compared to experiments.

When simulating cement hydration and setting, a 

cement paste system (cement particles and water) is 

represented by a 3-D voxel (volumetric picture element) 

digital image grid, and chemical components such as the 

various minerals, water, and dissolved species in solution 

are modeled as small cells of concentration at points of the 

grid. On each simulation time step, both diffusion of mobile 

species and chemical reactions can occur at each voxel. 

The diffusion of mobile species is accomplished by 

random movements of those components between 

neighboring grid points, and chemical reactions are 

considered at each voxel. Chemical and structural 

changes in the system are tracked by updating the number 

of cells of each type at the end of every time step. 

In addition, the model keeps track of the net heat 

released by the reactions; the solution composition, 

including pH; and the setting time. Updating diffusion and 

reactions at a grid point requires only localized information 

about the system around that point. Neighboring processors 

therefore require only as much communication with each 

other as necessary to exchange information about 

neighboring points at their subvolume boundaries.

Models and Visualiz

A lot of data comes out of these 3-D, multiprocessor 

simulations. Fully understanding the output of a supercomputer-based concrete flow or cement hydration 

simulation is a difficult problem that requires highperformance visualization and analysis. In the field, 

trying to look through fresh concrete is not easy, and 

with only simple-minded visualization of model outputs, 

researchers will end up fruitlessly “sticking their head 

into a bucket of concrete.” Our team meets this challenge 

using a state-of-the-art immersive visualization environment 

(IVE). To understand immersion, consider the following 

analogy: looking at 3-D data on an ordinary computer 

screen is like looking at goldfish in a bowl; using an 

immersive system is like swimming with the goldfish at 

the scale of the goldfish. 

The NIST IVE consists of hardware and software 

implemented to create a virtual laboratory. In this 

laboratory, the researcher is immersed into a qualitative 

and quantitative environment with tools for visualization, 

analysis, and measurement (Fig. 3). The system consists 

of three projectors (two rear projection, one top down) 

Fig. 2: A schematic picture of how a parallel computer with four 

processors treats the hydration or rheology models described in 

this article. The processors are numbered 1 to 4, and the squares 

are the regions they control. The points represent sample 

locations where data are updated as the simulation progresses. 

The circles represent the surrounding region from which 

information is drawn to update the data points in the models
Concrete international/ january 2011 27

arranged behind three 3.05 x 3.05 m (10 x 10 ft) square 

left, right, and floor screens arranged in a corner of a 

cube. Active stereo glasses produce 3-D images. A third, and 

very important, hardware component is location tracking. 

This subsystem tracks the direction and orientation in 

which a single user is looking. This information is used 

to update the 3-D images being generated in real time. 

From the three screens, this user perceives a fused 

visual experience of being surrounded by the data. Within 

the physical constraints of the screen surfaces, the user 

can walk around and through the data representation. 

Using this environment allows the researcher to interact 

virtually with the system under study in ways that are 

physically impossible.

The creation of an immersive application requires 

collaboration between the physical scientist with knowledge 

of the application and the computer scientist with 

knowledge of the visualization techniques. The challenge 

is to find a graphical representation that “speaks” to the 

research scientist and facilitates new understanding 

through interactive navigation, analysis, and measurement 

in the virtual environment. A successful application will 

either confirm or contradict the scientist’s preconceived 

notions. Both outcomes have benefits and point the way 

to future progress.

Concrete flow re

Figure 4 shows a snapshot of a visual simulation of 

aggregate flow inside (transparent) cement paste. 

Navigation in the immersive environment provides 

insight into the behavior of the flow. The system allows 

the researcher to “swim around” inside the concrete flow.

A technique recently implemented on the NIST system 

allows the researcher to focus on an area of interest while 

still providing visual context. In Fig. 5, the aggregates 

carrying low stress are shown as outline shapes while the 

aggregates carrying most of the stress are solid. This 

technique allows the researchers to focus on interesting 

high-stress areas while providing an overall view of the 

visual context, which is crucial for understanding. 

Isolated images without a visual context are not very 

instructive and can even be misleading.

Using this kind of visualization and further analysis of 

the supercomputer data, the concrete flow model has 

provided insight on how particle shape and size 

distributions can strongly influence rheological properties 

like yield stress (which inversely correlates to slump) 

and viscosity at cement paste, mortar, and concrete 

length scales. For example, in the case of a cement paste, 

it has been noted in the field that replacing a fraction of 

the cement with ultrafine fly ash can reduce yield stress. 

From large-scale simulations, it became clear that the 

reduction of yield stress was due to changes in how 

stress is transmitted throughout the suspension. When 

Fig. 3: A user in the NIST IVE interacting with a 3-D simulation 

(photo courtesy of Denease Anderson, NIST)

Fig. 4: Simulation of coarse aggregate flowing inside a fresh 

mortar. The numbers on the aggregates indicate shear stress 

levels exerted via the fluid and the other aggregates

Fig. 5: This is the same simulation of coarse aggregate flowing 

inside a fresh mortar as in Fig. 4, with only the most highly 

stressed coarse aggregates shown. The rest are represented by 

only a wire frame to give context
28 january 2011 / Concrete international

shearing during mixing, pumping, and flowing, stress can 

be transmitted via compressive and extensional forces; 

and the transmission of stress in compression can make a 

significant contribution to the apparent yield stress. 

The ultrafine fly ash is smaller than the cement 

particles and also round in shape. The model, with its 

detailed particle-level data and visualization, has shown 

that the inclusion of such particles makes it more difficult 

to establish “stiff” networks or chains of particles that 

can lead to jamming, making it harder to transmit stress in 

compression (that is, you can push a rod but not a rope). 

As a result, cement particles move about each other 

more easily, resulting in lower yield stress and viscosity. 

A similar result was found for higher-scale models of 

concrete where the matrix fluid is now a mortar and the 

solid inclusions are the coarse aggregates.

Our current simulations are investigating the effectiveness 

of different size gradations (aggregates at the concrete 

scale and cement particles at the paste scale) on viscosity 

and yield stress, which will lead to technical data that can 

be used to optimize these gradations for concrete flow. 

Because the models can handle any kind of external 

geometry, we also plan to simulate flow in common 

concrete rheometers using reference materials to obtain 

fundamental rheological parameters using standard 

measurements that can be used to optimize and control 

concrete placement.

Cement hydration r

Microstructures generated by our 3-D cement hydration 

model can be interactively visualized and analyzed at any 

time in the simulation. Figure 6 is a snapshot from an 

interactive visualization of a hydration run at a very early 

stage of the hydration process. Only the important 

calcium sulfate and calcium aluminate phases are shown.4 

Using a virtual interactive tool, the sulfate ion concentration 

in solution can be measured at any spot in the pore 

space. For this example, the probe returns a value of 

0.01039 mol/L at the point sampled. The availability of a 

concentration probe makes it possible to visualize 

concentration gradients and to determine, for example, 

whether calcium sulfate carriers are spatially distributed 

well enough in the cement to prevent undesirably rapid 

aluminate reactions. As the researcher probes the data, 

the numerical values are continuously displayed. Depending 

on the application requirements, the values can be saved 

for later use or dynamically fed to a separate process for 

real-time analysis.

The hydration model has been used to gain insights 

into the mechanisms that control the induction period of 

cement hydration, which is the time crucial to the 

placement and finishing of fresh concrete. The setting 

time of concrete is determined primarily by the length of 

the induction period, which can be controlled by the 

addition of chemical admixtures. Systematic design of 

admixture molecules will be made possible through a 

better understanding of the chemical processes that 

control the length of the induction period. The hydration 

model has been used to confirm that the induction period 

is controlled by the rate of growth of calcium-silicatehydrate5

 and has also shown that the nucleation of calcium 

hydroxide plays an important role in determining the rate 

of that growth and the end of the induction period.6

Work is now beginning on simulating the mechanism 

by which calcium sulfate controls the aluminate reactions 

in portland cement and prevents flash setting. Proper 

sulfate balance is crucial to control setting, especially 

when mineral admixtures are present. With these insights, 

we will be able to predict the optimum sulfate level in 

cement and determine the answers to questions such as 

why does undersulfation strongly retard hydration?

Prosp

At sub-mm length scales, scientists have recently made 

strides in unraveling the structure and properties of C-S-H 

gel, the glue that holds concrete binder together at the 

nanometer and molecular scale via molecular modeling.7

A framework is currently being built for connecting all 

these realistic models at different length scales and 

linking them to the macroscopic scale where engineering 

properties of concrete such as fresh concrete rheology, 

setting, mechanical properties, and transport properties 

can be computed as a function of time. We are already 

thinking of modeling degradation processes in concrete—

this next step is not too far off. This approach of combining 

physical science and computer science—powered by 

supercomputers, interpreted by sophisticated visualization, 

and applied at many length scales—requires intense 

Fig. 6: Calcium sulfate and calcium aluminate phases during initial 

cement hydration at a very early age, when chemical admixtures 

added to change setting behavior can often be effective
Concrete international/ january 2011 29

Jeffrey W. Bullard is a Materials Scientist i

the Materials and Construction Resear

Division of theNIST Engineering Labor

He was awarded a 2009 Department o

Commerce Silver Medal for his ceme

hydration simulations work in the 

and the 2008 FRL Communication 

for a technical publication that clarified the 

mechanism of cement hydration.

Edward J. Garboczi, FACI, i a NIST F

the Materials and Construction Resear

Division of theNIST Engineering Labor

He is a member ofACI Committees 

Hydraulic Cements; 236, Material Scie

of Concrete; and 365, Service Life Predictio

He is a Fellow of the American Cera

Society and received the Robert L’Herm

Medal fromRILEM in 1992 and a

Silver Medal from the Department 

Commerce for 

William L. George is a Computer Scientist i

the High Performance Computing an

Visualization Group in the Informatio

Technology Laboratory a NIST and dev

scientific programming using massively 

parallel computing. He was awarded a 2009

Department of Commerce Silver Medal 

his scientific programming work in the V

Nicos S. Martys is a Physicist in the

Materials and Construction Resear

Division of theNIST Engineering Labor

In 2009, Martys was awarded a Departme

of Commerce Silver Medal for his concre

rheology simulation work in the V

Steven G. Satterfield is a Computer

Scientist in the High Performance Computi

and Visualization Group in the Informatio

Technology Laboratory a NIST and ca

out research in scientific visualization. He

is the main developer of the immersive 

visualization system at N

Judith E. Terrill is the Leader of the Hig

Performance Computing and Visualizatio

Group in the Information Technolog

Laboratory at NIST, which carries

research in parallel computing and 

scientific visualization. She was awarded a

2009 Department of Commerce Silv

Medal for her work in the 

and sustained effort, producing important results at 

present and leading to great promise for the future—

especially as supercomputers keep on growing in size, 

speed, and availability.

Acknowledgments

We thank NASA and DOE for having a vision for concrete and for 

providing supercomputer time for these simulations. We express 

deep gratitude to the many past and present members of the VCCTL 

consortium for intellectual, moral, and financial support.

References

1. Bentz, D.P.; Garboczi, E.J.; Bullard, J.W.; Ferraris, C.F.; and 

Martys, N.S., “Virtual Testing of Cement and Concrete,” Significance 

of Tests and Properties of Concrete and Concrete-Making Materials, 

ASTM STP 169D, Joseph Lamond and James Pielert, eds., 2006, pp. 38-50.

2. Martys, N.S., “Study of a Dissipative Particle Dynamics Based 

Approach for Modeling Suspensions,” Journal of Rheology, V. 49, 

2005, pp. 401-424.

3. Martys, N.S.; George, W.L.; Chun, B.-W.; and Lootens, D., “A 

Smoothed Particle Hydrodynamics-Based Fluid Model with a 

Spatially Dependent Viscosity: Application to Flow of a Suspension 

with a Non-Newtonian Fluid Matrix,” Rheologica Acta, V. 49, No. 10, 

Oct. 2010, pp. 1059-1069.

4. Bullard, J.W.; and Flatt, R.L., “New Insights into the Effect of 

Calcium Hydroxide Precipitation on the Kinetics of Tricalcium 

Silicate Hydration, Journal of the American Ceramic Society, V. 93, 

No. 7, July 2010, pp. 1894-1903.

5. Bullard, J.W.; Enjolras, E.; George, W.L.; Satterfield, S.G.; and 

Terrill, J.E., “A Parallel Reaction-Transport Model Applied to Cement 

Hydration and Microstructure Development,” Modelling and 

Simulation in Materials Science and Engineering, V. 18, No. 2, 2010.

6. Bullard, J.W., “A Determination of Hydration Mechanisms for 

Tricalcium Silicate Using a Kinetic Cellular Automaton Model,” Journal 

of the American Ceramics Society, V. 91, No. 7, 2008, pp. 2088-2097.

7. Pellenq, R. J.-M.; Kushima, A.; Shahsavari, R.; Van Vliet, K.J.; 

Buehler, M.J.; Yip, S.; and Ulm, F.-J., “A Realistic Molecular Model of 

Cement Hydrates,” Proceedings of the National Academy of Sciences, 

V. 106, No. 38, Sept. 22, 2009, pp. 16102-16107.

Selected for reader interest by the editors.