# The ALMA correlator

R. P. Escoffier, G. Comoretto, J. C. Webber, Alain Baudry, C. M. Broadwell, J. H. Greenberg, R. R. Treacy, Ph. Cais, Benjamin Quertier, Pascal Camino, et al.

► To cite this version:

R. P. Escoffier, G. Comoretto, J. C. Webber, Alain Baudry, C. M. Broadwell, et al. The ALMA correlator. Astronomy & Astrophysics - A&A, 2007, 462, pp.801-810. 10.1051/0004-6361:20054519. hal-00127335

HAL Id: hal-00127335

https://hal.science/hal-00127335v1

Submitted on 1 Dec 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# The ALMA correlator

R. P. Escoffier<fnref n="1" />, G. Comoretto<fnref n="2" />, J. C. Webber<fnref n="1" />, A. Baudry<fnref n="3" />, C. M. Broadwell<fnref n="1" />, J. H. Greenberg<fnref n="1" />, R. R. Treacy<fnref n="1" />, P. Cais<fnref n="3" />, B. Quertier<fnref n="3" />, P. Camino<fnref n="3" />, A. Bos<fnref n="4" />, and A. W. Gunst<fnref n="4" />

Received 15 November 2005 / Accepted 18 September 2006

## ABSTRACT

**Aims.** The Atacama Large Millimeter Array (ALMA) is an international astronomy facility to be used for detecting and imaging all types of astronomical sources at millimeter and submillimeter wavelengths at a 5000-m elevation site in the Atacama Desert of Chile. Our main aims are: describe the correlator sub-system which is that part of the ALMA system that combines the signal from up to 64 remote individual radio antennas and forms them into a single instrument; emphasize the high spectral resolution and the configuration flexibility available with the ALMA correlator.

**Methods.** The main digital signal processing features and a block diagram of the correlator being constructed for the ALMA radio astronomy observatory are presented. Tables of observing modes and spectral resolutions offered by the correlator system are given together with some examples of multi-resolution spectral modes.

**Results.** The correlator is delivered by quadrants and the first quadrant is being tested while most of the other printed circuit cards required by the system have been produced. In its final version the ALMA correlator will process the outputs of up to 64 antennas using an instantaneous bandwidth of 8 GHz in each of two polarizations per antenna. In the frequency division mode, unrivalled spectral flexibility together with very high resolution (3.8 kHz) and up to 8192 spectral points are achieved. In the time division mode high time resolution is available with minimum data dump rates of 16 ms for all cross-products.

**Key words.** techniques: interferometric – techniques: spectroscopic – instrumentation: interferometers – instrumentation: spectrographs

## 1. Introduction

The Atacama Large Millimeter Array (ALMA) is an international astronomy facility. ALMA is a partnership between Europe, North America and Japan, in cooperation with the Republic of Chile (see Alma Information and note at the end of this paper). ALMA will consist of a main array of up to 64 12-m diameter antennas supplied by Europe (ESO) and North America (NRAO/AUI) and a compact array of twelve 7-m plus four 12-m antennas supplied by Japan (NAOJ). The main goal of this paper is to describe the essential features of the ALMA correlator being constructed for the main array (64 antennas) and to emphasize its huge versatility making this correlator a key element of all future astronomy programs led with ALMA. The ALMA correlator is a very large digital system that combines the outputs of up to 64 array elements using an instantaneous bandwidth of 8 GHz in each of two polarizations per antenna, and produces a single astronomical telescope from the 64 movable antennas distributed within a diameter of about 150 m to an expanded configuration of maximum antenna separation 18.5 km.

A digital correlator system is the heart of any modern radio astronomy system in which the signal power is detected by measuring (a) the cross-correlation of all antenna pairs in an array of antennas, and/or (b) the auto-correlation of one (or all) single antenna(s). Digital lags can easily be implemented in correlators thus allowing flexible analysis of astronomical spectral features. There are two different correlator architectures (see e.g. Chap. 8 in Thompson et al. 2001). The most commonly

adopted architecture is the lag correlator (XF architecture) in which the cross-correlated data (X-part of the system in the lag domain) are produced first and the Fourier transform (F-part of the system) is applied at a later stage for transformation to the frequency domain. For example, the VLA (Very Large Array), or IRAM (Institut de Radio Astronomie Millimétrique) interferometers use XF-type correlators. The other type of correlators is named FX because the Fourier transform to the frequency domain is performed first. The Nobeyama interferometer, the NRAO VLBA Correlator, and the ALMA Compact Array (16 antennas in addition to the ALMA main array) have adopted the XF architecture. Comparison of the relative merits of the XF and FX schemes is complicated by the details of practical implementation of the various “X” and “F” calculations, but we note that beyond around 20 antennas all interferometers have adopted the XF architecture.

The case of the ALMA project is particular because we need to process simultaneously many antennas and large bandwidths with high spectral versatility and a resolution as high as a few kHz. Such an ambitious project required innovative designs, in particular for measuring the correlation coefficients (with a new application specific integrated circuit) and for digital filtering (with a versatile new digital filter design). The ALMA correlator originally used a pure lag architecture (Escoffier 1997) incorporating simple Finite Impulse Response (FIR) digital cards (Escoffier & Webber 1998; Escoffier et al. 1999). The initial XF architecture has recently evolved into a digital hybrid

<footnote n="1">¹ National Radio Astronomy Observatory, Charlottesville, 22903 VA, USA</footnote>

<footnote n="2">² Osservatorio Astrofisico di Arcetri, Largo Fermi 5, 50125 Firenze, Italy</footnote>

<footnote n="3">³ Observatoire Aquitain des Sciences de l'Univers, UMR CNRS 5804, BP 89, 33270 Floirac, France
e-mail: baudry@obs.u-bordeaux1.fr</footnote>

<footnote n="4">⁴ ASTRON, PO Box 2, 7990 AA Dwingeloo, The Netherlands</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. ALMA Correlator under construction (first quadrant).</figcaption></figure>

<table><caption>**Table 1.** ALMA correlator specifications.</caption><thead><tr><td>Item</td><td>Specification</td></tr></thead><tbody><tr><td>Number of antennas</td><td>64</td></tr><tr><td>Number of baseband channel inputs per antenna</td><td>8</td></tr><tr><td>Input sample format</td><td>3 bit, 8 level at 4 GSample/s per baseband channel</td></tr><tr><td>Correlation sample format</td><td>2 bit, 4 level and 4 bit, 16 level; Nyquist and twice Nyquist</td></tr><tr><td>Maximum baseline delay range</td><td>30 km</td></tr><tr><td>Hardware cross-correlators per baseline*</td><td>32,768 leads + 32 768 lags</td></tr><tr><td>Hardware autocorrelations per antenna*</td><td>32,768</td></tr><tr><td>Typical performance in digital hybrid modes</td><td>8192 spectral points provided for each pair of baseband inputs**</td></tr><tr><td>Product pairs possible for polarization</td><td>HH, VV, HV, VH (for orthogonal H and V)</td></tr></tbody></table>

design that yields a 32-fold improvement in frequency resolution at the widest bandwidth. This performance upgrade required the design of a digital Tunable Filter Bank (TFB) which performs a frequency division of the digitized input wideband (2 GHz) into 32 sub-channels 62.5 MHz (or 31.25 MHz) wide. Rather than the conventional XF architecture, the ALMA design is now FXF with tunable anti-aliasing digital filters making a first coarse spectral division of each 2 GHz wide input channel. The FXF scheme was initially proposed for the Second Generation ALMA Correlator (Baudry & de Vos 2000; Bos 2000; Quertier et al. 2003).

The ALMA correlator is organized by quadrants including a total of 32 main racks (not counting computer or power supply racks). When all correlator cards are used at the nominal clock rate of 125 MHz the correlation calculation rate goes up to $1.7 \times 10^{16}$ multiply-and-add operations per second. The correlator system is still under construction with testing and acceptance of individual correlator and filter cards being performed at NRAO, Charlottesville and Université de Bordeaux. The first quadrant of the final system is being tested at the new NRAO Technology Center in Charlottesville, Virginia (Fig. 1).

It is interesting to note that in the future when the full ALMA system is complete and delivered to Chile any combination of the ALMA main array antennas with the ALMA compact array antennas could be processed by the ALMA correlator provided that the maximum number of antennas does not exceed 64. Parallel processing with the two correlator systems at the Atacama site of the data captured by both the main and compact arrays will be common practice in the future.

In this paper we present the main features of the ALMA main array correlator. System specifications, block diagram and main system functions are given in Sect. 2. Time and frequency division operation are outlined in Sect. 3 together with a brief description of the Tunable Filter Bank card sub-system. Other antenna-based electronics (Station cards) and the correlator and

long term accumulator cards are described in Sects. 4 and 5. The observing modes and some examples showing the correlator system flexibility are discussed in Sect. 6. More technical details concerning the control cards, the system test strategy and components selected for fabrication of the correlator system are given in Appendices A and B.

## 2. System specifications, block diagram and main functions

The ALMA correlator measures cross-correlation coefficients of all 2016 independent pairings of the 64 antennas, taken two at a time. Mathematical processing of the cross-correlation coefficients will produce millimeter- and submillimeter-wave intensity maps of the celestial emitting sources. The maps may be wide bandwidth continuum or multiple spectral line maps, each over an incremental bandwidth, for a subset of or all the 4 polarization cross-products.

The system specifications for the ALMA correlator can be seen in Table 1. The 64 antenna specification with a total 16 GHz bandwidth per antenna drives the correlator input specifications (see Escoffier et al. 2005). The receiver intermediate frequency outputs are down-converted into 4 basebands 2 GHz wide for each of 2 different senses of polarization. The input to the correlator is thus split into 8 baseband signals of 2 GHz bandwidth each.

Signals from each 2 GHz baseband output drive into the correlator as three digital input data streams, each with an equivalent clock rate of 4 GHz reflecting the ALMA standard of the 3-bit, 4 Gsample/s digitization of the baseband analog outputs at the antennas. Signal transmission from the remote antennas to the correlator is done by the ALMA backend system over fiber optic cables (McCool 2001). Recovery of the sample streams from the optic input signal is done in Data Transmission System (DTS) receiver printed circuit cards which are physically

<footnote>* 62.5 MHz correlators (125 MHz clock rate), divide by 32 to get number of equivalent 2 GHz correlators.</footnote>

<footnote>** Resulting in 8192, 4096 or 2048 spectral points across the baseband spectrum, depending on polarization mode.</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 2. ALMA Correlator Block Diagram.</figcaption></figure>

installed in the correlator racks but are considered in the ALMA project to be part of the backend system, not the correlator.

A block diagram of the ALMA correlator can be seen in Fig. 2. This figure also includes some details of the ALMA digitizers, the DTS transmission system, and the real time computer. Although these components of ALMA are not formally part of the correlator, they are shown here for clarity. The block diagram shows antenna- or station-based electronics (upper part of Fig. 2) and antenna baseline electronics (lower part of Fig. 2).

Antenna-based electronics in the correlator system include fiber optic receiver cards to recover digitized samples from the remote telescopes, tunable digital filter bank cards for bandwidth selection, bulk RAM delay lines to compensate for geometric delays in signal paths, and signal conditioning logic to packetize the output of the high-speed digitizers (operated at 4 Gsample/s) in order to drive lower-speed correlator circuits (125 MHz) in the widest bandwidth application.

The input stage of the correlator is the Tunable Filter Bank (TFB) card. This card is driven by the output of the ALMA DTS receiver card, which recovers the 3-bit samples generated by the ALMA digitizer and sent *via* optical fiber from the remote antennas over the ALMA DTS. The filter bank card can be configured as 32 digital filters with either 62.5 or 31.25 MHz bandwidth. The center frequency of each filter is independently tunable.

The station card has several functions. The main functions are: (i) the implementation of bulk delay for geometric delay compensation and (ii) providing for signal routing between the filter card outputs and the correlator sub-system inputs (RAMs for geometric compensation are also provided in the tunable filter bank card). Programming in the station cards determines how the 32 filter outputs of the filter card are processed in the correlator system.

The correlator portion of the system in each of the 4 quadrants (baseline electronics in the system) consists of 32 correlator

“planes” with each plane implementing a 64-by-64 correlation matrix. When samples from all 64 antennas in the ALMA main array drive this matrix (both vertical and horizontal axes), autocorrelations are produced on the matrix diagonal and cross-correlations are produced elsewhere, with correlation “leads” on one side of the diagonal and “lags” on the other. “Leads” and “lags” are correlator nomenclature for cross-correlation in which one signal is multiplied by time-displaced versions of another signal. Leads involve time displacement in one direction and lags involve time displacement in the other direction. The term lag is also frequently used generically to refer to either a lead or a lag.

The LTA (Long Term Accumulator) sub-system seen in the lower part of Fig. 2 takes short 1 ms or 16 ms integrations from the ALMA custom correlator chips and provides longer term integration. The LTA also provides the interface into the real time computer system for both control and data extraction.

The ALMA correlator is designed in 4 essentially identical quadrants. Each quadrant processes the output of two of the eight 2 GHz baseband channel outputs of the array. For polarization observations, the two baseband channel outputs processed in a quadrant are usually of orthogonal polarizations.

The ALMA correlator is a very large digital system. Its 32 main racks contain around 3000 printed circuit cards employing more than 135 000 complex integrated circuits. The overall system dissipation is around 170 kW. For many years, the correlator for the VLA observatory was the largest correlator for a 27-element radio telescope array in the world. For ALMA, the increase in the number of antennas (64 antennas require to process 2016 independent baselines instead of 351 for the VLA), the much wider bandwidth per antenna (16 GHz instead of 200 MHz for the VLA), and the much higher spectral resolution requirements (up to 8192 channels available for each pair of baseband inputs) have resulted in the ALMA correlator being a factor of over 15 000 larger than the VLA correlator. To insure that an
<figure><img src="image_4.png" /><figcaption>Fig. 3. Time division operation versus frequency division operation.</figcaption></figure>

economical and reliable system would result it was decided to use as often as possible in the design commercially available highly integrated circuits and Field Programmable Gate Arrays (see end of Sect. 3 and Appendix B).

### 3. Time and frequency division operation and Tunable Filter Bank card

In the original time division strategy adopted for the ALMA correlator design, a 1 ms packet of 4 GHz clock rate samples from a digitizer output is split into 32 sub-packets, each with a 125 MHz clock rate (input: one millisecond packet of samples at 4 GHz; output: 32 one millisecond packets at 125 MHz). Each of the output time packets contains 125 000 time-contiguous samples.

Cross-correlation coefficients between each of the 125 MHz packets and samples from an identical 1/32 ms time interval from the other 63 antennas in the ALMA array are calculated in the correlator. In order to get full efficiency, measured correlation coefficients from all 32 low speed time packets must be summed since each processes only a 1/32 part of the input signal (a small inefficiency is incurred because samples on either side of a packet boundary are never correlated against one another).

With the use of the filter bank, correlator operation can be considered to use a frequency division strategy and the architecture is now FXF rather than the conventional XF. The digitizer wideband (2 GHz) input to the correlator is split into up to 32 sub-bands or subchannels 62.5 MHz wide (or 31.25 MHz wide in some modes) with each subchannel being independently processed by the digital correlators.

The operation of the frequency division strategy can be seen in Fig. 3, below. The filter bank card can split the 2 GHz input frequency band into as many as thirty-two 62.5 MHz bands. Cross-correlation coefficients between same-band signals from all 64 antennas are measured. But now, since each frequency band is independent, the 32 resulting correlation functions are not summed, but the spectra that result from each may be stitched end-to-end and a factor of 32 improvement in frequency resolution across the original 2 GHz band is realized.

Operation with fewer than all 32 subchannels allows the station card to distribute the available correlator capacity among the active filters for optimum spectral resolution performance. As the number of active filters in the filter bank decreases, the

overall performance on the system in frequency division operation approaches the original system performance in time division mode. When only a single filter output is active, there is no difference in spectral resolution performance between the two strategies.

The independent tuneability of the filter bank card also allows versatility in the use of the correlator. For example, if only a few filter subchannels or a few groups of subchannels are used in a given observation, they need not cover contiguous frequencies. They may, if desired, each be centered on different molecular emission lines in the astronomical source within the overall 2-GHz input band.

We now briefly describe the essential elements of the Tunable Filter Bank card and how it is interfaced to the correlator system. Each 2 GHz IF channel input signal to a filter card consists of 32 time-demultiplexed 3-bit samples (total of 96 signal wires) at the system 125 MHz clock rate (total sample rate, 4 GS/s). The basic function of the digital filter card is to develop 32 subchannels from this 2 GHz input as follows. First, the input signal is processed by a complex digital mixer driven by a digital local oscillator to translate each subchannel frequency center to zero frequency. Second, each complex mixer output is sent to 2 parallel filters processing the real and imaginary data streams; each of these 2 parallel filters is implemented in a 2-stage architecture subsystem. The first stage is a low-pass decimation filter (decimation by 32, 128-tap weights). The second stage (64-tap weights) is a high performance low-pass filter synthesizing the selected subchannel band shape. The global 2-stage filter performance is thus equivalent to that of a single 32 × 64 = 2048 tap weight filter. After decimation and complex to real conversion the signal is either 2- or 4-bit requantized, as specified in the observation schedule. Each filter then produces streams of 2- or 4-bit samples at 125 MHz; in the 4-bit stream case there are some restrictions depending on the observing modes (see Tables 2 to 4).

The 32-filter array allows the selection of subchannels with a maximum bandwidth of 62.5 MHz that can be positioned anywhere within the 2 GHz IF channel with a precision of 30.5 kHz. The twice Nyquist sampling option, resulting in increased signal-to-noise ratio at the expense of half the number of spectral points (see section on Observing modes and Tables 2 to 4), is available with both the nominal 62.5 MHz bandwidth
<table><caption>**Table 2.** Mode chart with one baseband channel being processed.</caption><thead><tr><th>Number of active filters</th><th>Total bandwidth</th><th>Number of spectral points</th><th>Spectral resolution</th><th>Velocity resolution at 230 GHz</th><th>Correlation*</th></tr></thead><tbody><tr><td>32</td><td>2 GHz</td><td>8192/4096/2048</td><td>244/488/976 kHz</td><td>0.32/0.64/1.28 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N</td></tr><tr><td>16</td><td>1 GHz</td><td>8192/4096/2048/1024</td><td>122/244/488/976 kHz</td><td>0.16/0.32/0.64/1.28 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>8</td><td>500 MHz</td><td>8192/4096/2048/1024</td><td>61/122/244/488 kHz</td><td>0.08/0.16/0.32/0.64 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>4</td><td>250 MHz</td><td>8192/4096/2048/1024</td><td>30/61/122/244 kHz</td><td>0.04/0.08/0.16/0.32 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>2</td><td>125 MHz</td><td>8192/4096/2048/1024</td><td>15/30/61/122 kHz</td><td>0.02/0.04/0.08/0.16 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>1</td><td>62.5 MHz</td><td>8192/4096/2048/1024</td><td>7.6/15/30/61 kHz</td><td>0.01/0.02/0.04/0.08 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>1</td><td>31.25 MHz</td><td>8192/2048</td><td>3.8/7.6 kHz</td><td>0.005/0.01 km s<sup>-1</sup></td><td>2B-2N/4B-2N</td></tr><tr><td>Time Division Mode</td><td>2 GHz</td><td>64</td><td>31.25 MHz</td><td>40.8 km s<sup>-1</sup></td><td>Full 3-bit x 3-bit, Nyquist sampling</td></tr></tbody></table>

<table><caption>**Table 3.** Mode chart with two baseband channels per quadrant processed with polarization cross products.</caption><thead><tr><th>Number of active filters</th><th>Total bandwidth</th><th>Number of spectral points</th><th>Spectral resolution</th><th>Velocity resolution at 230 GHz</th><th>Correlation*</th></tr></thead><tbody><tr><td>32</td><td>2 GHz</td><td>4096</td><td>488 kHz</td><td>0.64 ms</td><td>2B-N</td></tr><tr><td>16</td><td>1 GHz</td><td>4096/2048/1024</td><td>244/488/976 kHz</td><td>0.32/0.64/1.28 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N</td></tr><tr><td>8</td><td>500 MHz</td><td>4096/2048/1024/512</td><td>122/244/488/976 kHz</td><td>0.16/0.32/0.64/1.28 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>4</td><td>250 MHz</td><td>4096/2048/1024/512</td><td>61/122/244/488 kHz</td><td>0.04/0.08/0.16/0.32 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>2</td><td>125 MHz</td><td>4096/2048/1024/512</td><td>30/61/122/244 kHz</td><td>0.04/0.08/0.16/0.32 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>1</td><td>62.5 MHz</td><td>4096/2048/1024/512</td><td>15/30/61/122 kHz</td><td>0.02/0.04/0.08/0.16 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>1</td><td>31.25 MHz</td><td>4096/1024</td><td>7.6/30 kHz</td><td>0.01/0.04 km s<sup>-1</sup></td><td>2B-2N/4B-2N</td></tr><tr><td>Time Division Mode</td><td>2 GHz</td><td>128</td><td>15.6 MHz</td><td>20.4 km s<sup>-1</sup></td><td>2B-N</td></tr></tbody></table>

<table><caption>**Table 4.** Mode chart with two baseband channels per quadrant processed with polarization cross products.</caption><thead><tr><th>Number of active filters</th><th>Total bandwidth</th><th>Number of spectral points</th><th>Spectral resolution</th><th>Velocity resolution at 230 GHz</th><th>Correlation*</th></tr></thead><tbody><tr><td>32</td><td>2 GHz</td><td>2048</td><td>976 kHz</td><td>1.28 km s<sup>-1</sup></td><td>2B-N</td></tr><tr><td>16</td><td>1 GHz</td><td>2048/1024</td><td>488/976 kHz</td><td>0.64/1.28 km s<sup>-1</sup></td><td>2B-N/2B-2N</td></tr><tr><td>8</td><td>500 MHz</td><td>2048/1024</td><td>244/488 kHz</td><td>0.32/0.64 km s<sup>-1</sup></td><td>2B-N/2B-2N</td></tr><tr><td>4</td><td>250 MHz</td><td>2048/1024</td><td>122/244 kHz</td><td>0.16/0.32 km s<sup>-1</sup></td><td>2B-N/2B-2N</td></tr><tr><td>2</td><td>125 MHz</td><td>2048/1024/512</td><td>61/122/244 kHz</td><td>0.08/0.16/0.32 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N</td></tr><tr><td>1</td><td>62.5 MHz</td><td>2048/1024/512/256</td><td>30/61/122/244/ kHz</td><td>0.04/0.08/0.16/0.32 km s<sup>-1</sup></td><td>2B-N/2B-2N/4B-N/4B-2N</td></tr><tr><td>1</td><td>31.25 MHz</td><td>2048/512</td><td>15/61 kHz</td><td>0.02/0.08 km s<sup>-1</sup></td><td>2B-2N/4B-2N</td></tr><tr><td>Time Division Mode</td><td>2 GHz</td><td>64</td><td>31.25 MHz</td><td>40.8 km s<sup>-1</sup></td><td>2B-N</td></tr></tbody></table>

and the 31.25 MHz half-bandwidth. A factor of 2 loss of resolution is not incurred for 31.25 MHz twice Nyquist modes.

The digital filter functions described above are implemented in programmable gate logic arrays offering large computational power. These devices, named Field Programmable Gate Arrays (FPGA's), contain different types of RAM blocks, configurable digital signal processor blocks and a huge number of logic cells. The FPGA's are commercially available with new nanometer technology implying FPGA core low voltage and thus less power requirement. Trading cost against available logic cells and overall filter performance, we have implemented two digital filters in one fast FPGA. Sixteen FPGAs are assembled on one Tunable Filter Bank card. The entire filter bank consists of 64 × 2 cards per correlator quadrant and there are 4 × 2 cards for 4 baseband pairs per antenna. There are 512 filter cards in the complete correlator system.

Each filter card is interfaced to the correlator subsystem via the station bin backplane. Each filter communicates with the station bin control computer via a standard CPLD interface chip; this interface allows the system to monitor the TFB processing in the FPGAs and to download the FPGA personalities; bandwidths and filter shapes are selected using pre-calculated filter tap weights. There is an additional CPLD on each filter card to monitor the integrity of the FPGA filter personalities.

In addition to the 16-FPGA array and to the CPLD interface, there are 3 fine-delay chips implemented on each Tunable Filter Bank card; each one of these 3 chips processes one bit of the 3-bit digitized input data stream. The fine delay increments are operated in synchronism with the bulk delay implemented in the Station Cards. (Ultra-fine delay tracking is implemented in the

digitizer clock module.) The fine delay chips are also used as receiver buffers which distribute the input signal data through the filter FPGA array.

## 4. Station card and correlator data interface

Each Station card provides two 4 ms RAM buffers, one for each of two filter card outputs; that is, each buffer holds approximately 4 ms of samples at the full 4 GHz sample rate of a single baseband channel output. A station card provides several basic functions in the correlator:

* bulk storage of the geometric delay compensation;

* a versatile crossbar function between the filter card output and the correlators;

* lag generation for high frequency-resolution modes;

* 1 ms time packets of samples in time division operation at 125 MHz rate.

The two basic modes of the station card reflect the two operation modes of the ALMA correlator. Most spectral observations will use the high frequency resolution digital hybrid configuration, or frequency division, mode. In these cases, the input to the station card from a filter card consists of from 1 to 32 separate bands each with a 125 MHz clock rate, either Nyquist or twice-Nyquist sampled.

The other operating mode of the correlator is used in high time resolution observations, or typically to reduce the amount of archived data in continuum mode. In this mode, the filter cards provide a pass-through function and the station card input is the

<footnote>* 2B-N, 2-bit by 2-bit correlation, Nyquist sampled.</footnote>

<footnote>* 2B-2N, 2-bit by 2-bit correlation, twice Nyquist sampled.</footnote>

<footnote>* 4B-N, 4-bit by 4-bit correlation, Nyquist sampled.</footnote>

<footnote>* 4B-2N, 4-bit by 4-bit correlation, twice Nyquist sampled.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 4. Quadrature signal drive into rack-to-rack cable system.</figcaption></figure>

full 4 GHz clock rate output of the ALMA digitizers with samples limited to 2 bits. In this mode, each one millisecond of the full bandwidth signal at the station card input is split into 32 one millisecond packets, each with a 125 MHz clock rate, in the station card RAM buffer (this function is also called a corner-turning operation). The 32 packets are routed to 32 correlator planes where cross correlation occurs (see Fig. 2). For full efficiency, the results obtained for the 32 planes must be summed.

The data interface between the station-based and the baseline-based electronics in the ALMA correlator probably represents the greatest design challenge in the system. The system has 64 antennas times 8 baseband signals per antenna times 32 demultiplexed 125 MHz signals per 2 GHz baseband signal times 2 bits per sample (correlated) or 32 768 rack-to-rack digital interfaces with a 125 MHz clock rate. Card-to-card 125 MHz data runs over bin mother boards exceed 125 000.

The first design goal set for the ALMA correlator was to find a system architecture in which every 125 MHz data stream between the station electronics and the correlator racks had one and only one destination. Achieving this goal would mean that the ratio between the 32 768 signals and the number of rack-to-rack signal cables would be 1. A factor of 2 would mean that the 32 768 signals would require 65 536 cables (at 125 MHz data rate).

The rack-to-rack interface standard selected for the ALMA correlator is 250 MHz LVDS. The two (125 MHz) sample bits of a filter card output are multiplexed to obtain the 250 MHz data rate. Standard 2 mm hard metric shielded twisted pair cables are the interface medium. A total of 16 384 signal cables are thus required in the full system.

In order to remove tight length specifications on the cables, the 250 MHz data signals driving into the cable system can be individually selected for drive on 0, 90, 180, or 270 deg phase-shifted versions of the 250 MHz interface clock. With programmable individual control on every rack-to-rack 250 MHz interface signal, the system may be put through a training session in which all possible interface phases are evaluated with known test data being carried over the cables. Error measurements can be made and error-free transmission phases recorded

for each signal path. The system can then be programmed to of the appropriate drive phases for error-free operation across all outputs.

Thus, instead of working with fixed phase transmission and capture clocks and trimming cable lengths for error-free capture, the output phase of each cable driving signal is adjusted for error-free capture. Also, each cable may be cut to a convenient physical length to fit its application, avoiding miles of slack cable lengths to be accommodated if all cables had to be the same length.

Figure 4 shows a single 250 MHz output cable signal cycling through all 4 drive phases. The horizontal scale in this figure is 4 nanoseconds per division.

## 5. Correlator section and Long Term Accumulator

The basic building block of the correlator system is a matrix of 64-by-64 correlator blocks referred to as a correlator "plane". A correlator plane provides a 256 lag correlator circuit at each of the intersections of the matrix and a correlator plane thus has 64 × 64 × 256 total correlator lags. The correlator system places the 64 × 64 matrix (one plane) in 4 printed circuit cards, each one with 64 application specific integrated circuits (ASIC). This ASIC has 4096 multipliers (lags) including 20 bits of integration and 16 bits of secondary storage for the results readout of the cross- and auto-correlation coefficients. Each 256 lag circuit in a correlator plane is driven by two different polarization signals (two baseband signals) from each of two antennas and can be programmed to be a single 256-lag block, to be two 128-lag blocks, or to be four 64-lag blocks (to support the various polarization options possible with the two baseband inputs). For 256 lag blocks on the diagonal of the matrix, the two signals are from the same antenna, and these yield auto-correlation results.

The next building block of the correlator system is the correlator "array" which consists of 32 correlator planes. There is one correlator array in each correlator quadrant and this array can process the full 2 GHz bandwidth outputs of all 64 antennas for 2 baseband channels.
In the widest bandwidth digital hybrid mode, each of the 32 filter card outputs is processed in one of the 32 correlator planes of a correlator array; each filter card output will typically be a 62.5 MHz bandwidth, Nyquist sampled, with a 125 MHz clock rate. For narrower bandwidth modes, some filter card outputs are not processed and the freed-up correlator planes are used to develop additional lags from the active filter card outputs. Lag generation for these modes is done on the station card by offset RAM addressing.

In time division observations, the 32 correlator planes in a quadrant process the 32 time bins developed by the station card. Thus, each plane handles 1/32 of the samples taken by an antenna digitizer since it takes all 32 planes correlating station card 125 MHz time packet outputs to keep up with the original 4 GHz sample rate. In this mode, the outputs of the 32 planes are summed, reducing the total number of effective correlators in the system by a factor of 32.

The Long Term Accumulator (LTA) cards provide buffers, or longer term integration, for correlator ASIC results. For correlator ASICs operating in 1 ms Correlator Accumulation Mode (CAM), results transfer from correlator chip to LTA every 1 ms. The LTA stores 16 consecutive sets of 1 ms results in separate buffers and transfers these results to the Correlator Data Processing computer (CDP) every 16 ms. For correlators operating in 16 ms CAM, results transfer from correlator chip to LTA every 16 ms. The LTA provides integration of these results for integer multiples of 16 ms.

Further details on the LTA and correlator control cards are given in Appendix A.

## 6. Observing modes

The ALMA correlator offers a huge variety of observing modes.

(i) *Firstly*, a broad range of spectral resolutions across various total bandwidths is available together with different polarization modes. Table 2 gives the performance of a single quadrant of the correlator in modes in which only one baseband channel of a baseband pair is processed. A maximum resolution of 3.8 kHz is achieved. Tables 3 and 4 give system performance with 2 basebands per quadrant processed with and without polarization cross products. In the latter case (Table 4) a full Stokes analysis of the incoming waves is possible and thus allows the study of the magnetic filed in the gas and dust components of the interstellar medium.

(ii) *Secondly*, several sensitivity options can be selected when less spectral resolution across a given bandwidth is sufficient. This is achieved with double Nyquist sampling and 4-bit × 4-bit correlation. The theoretical 0.88 correlation efficiency of the basic 2-bit × 2-bit correlation (with respect to a correlator which would have an infinite number of quantized levels) then increases to 0.94 (double Nyquist sampling) and 0.99 (4-bit correlation). Double Nyquist sampling and 4-bit correlation is also available; it does not significantly improve the 0.99 efficiency but with half the spectral resolution it allows speeding up by a factor of two the data dump time. It is worth mentioning that for spectra with same spectral resolution increasing the efficiency to 0.94 and 0.99 is equivalent to diminishing by 14% and 27% the observing time required to achieve a given signal to noise ratio.

(iii) *Thirdly*, the time division mode for which the correlator system behaves as a pure XF system is always available and allows fast dump times if that would be desired. All cross-correlation products are transferred to the Correlator Data Processor in 16 ms for the time division mode whereas 64 to

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Synthetic spectra generated with a multi-gauss routine to mimic the forest of lines observed around 348 GHz in Orion by Beuther et al. (2005). They illustrate the crowding of lines and the need for multi-resolution analysis (see text and flexible ALMA correlator modes).</figcaption></figure>

512 ms dump times are possible in frequency division operation depending on the selected modes. An even faster dump time of 1 ms is achievable for auto-correlation spectra.

(iv) *Finally*, because of the high flexibility of the frequency setting in the digital filter cards and because the correlator card resources available per quadrant can be redistributed in different observing modes, it is possible to set up modes to observe simultaneously several spectral windows or a given bandwidth with different resolutions. A combination of multi-windows and multi-resolution modes is possible. Such modes are implemented by means of spectral multi-resolution programming commands in the station cards of the correlator system.

It is impossible to describe in short the huge flexibility of the ALMA correlator but it will allow, for example, performing in star-forming regions multi-resolution spectral scans together with multi-broad band observations to analyze simultaneously the gas and dust properties of these regions. Similar studies will also be feasible in highly redshifted objects. In an attempt to illustrate the potential of the correlator, we have first mimicked (Fig. 5) one particular spectral region of the highly complex sub-millimeter line spectrum of the Orion-KL source. The upper part of the figure shows the overall distribution over 2 GHz (one ALMA baseband). These features mimic (but do not reproduce exactly) the original data and line identification obtained by Beuther et al. (2005) with the SMA interferometer and about 1 MHz resolution. The lower part of Fig. 5 shows how complex the observed features may be across 125 MHz when the resolution is improved. Such a rich forest of lines could be aged with, for example, the following *simultaneous* combination of ALMA spectral modes: 244 kHz resolution in 500 MHz placed in any spectral region of interest within 2 GHz; 30 kHz
<figure><img src="image_7.png" /><figcaption>**Fig. A.1.** Block Diagram of Station Card showing self-testing capability.</figcaption></figure>

resolution in 62.5 MHz bandwidth to zoom, for example, on a particular group of lines within the initial 500 MHz; obtain even higher spectral resolution, 7.6 kHz, again within the above 62.5 MHz band or, if that would be desirable, anywhere else within 2 GHz. These options could be slightly different in terms of spectral resolution if polarization were analyzed.

It is worth noting that the spectral resolution given in Tables 2 to 4 is in fact the separation between spectral points and this value must be used for signal to noise ratio calculations. The actual spectral resolution, that is to say the spectral response to an “infinitely” narrow line observed by ALMA, depends on the observing mode. In the time division mode for which the system is a pure XF correlator, the Fourier transform of the time-limited correlation function is the direct product of the true correlation function with a sinc function corresponding to the maximum lag window used during the integration. The effective resolution is thus 1.2 times the spectral point separation. In the frequency division mode the correlator behaves as a FXF system. It would be a pure FX system if there were a large number of subchannels dividing up the 2 GHz wide input channel. The effective resolution would then be the spectral channel separation weighted by a (sinc)² function but with some amplitude loss as formally described in Thompson et al. (2001). Because the ALMA correlator is a FXF system with 32 sub-bands when the full tunable filter bank is active the 1.2 factor degradation mentioned above must be seen as an upper limit and the effective spectral resolution must be determined in practice.

## 7. Conclusion

The correlator for the ALMA radio astronomy observatory will process up to 64 antennas in a highly flexible manner and with a broad variety of spectral and time resolutions using an instantaneous bandwidth of up to 8 GHz in each of two polarizations. The system supports frequency resolutions ranging from a single bandwidth 2 GHz wide (continuum observations) to 8192 increments, each 3.8 kHz wide, spanning a 31.25 MHz band (spectral line observations).

The correlator should be delivered to the array site in the Atacama desert of Chile starting in 2007. The entire system should be at site by 2008 or 2009. Depending on the procurement of the antennas, astronomical test observations could begin in 2008 with one correlator quadrant accommodating up to 16 antennas and all cross-correlation products. Completion of the entire observatory is currently set for 2012.

**Acknowledgements.** The ALMA Correlator Integrated Product Team acknowledges strong support during this work from their respective institutes: NRAO, Charlottesville for the North America group and, for the European group, the European Southern Observatory (ESO), Université de Bordeaux, OASU/L3AB, France, Osservatorio Astrofisico di Arcetri, Italy and ASTRON, The Netherlands. We thank M. Walmsley who suggested a number of improvements to this paper. We particularly thank J. Pisano, J. Perez, and R. Amestica of the ALMA Computing group for their work in software development which has made possible the integrated testing of the hardware and firmware described here.

The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc.

**Note on ALMA facility.** The Atacama Large Millimeter/submillimeter Array (ALMA), an international astronomy facility, is a partnership between Europe, Japan and North America in cooperation with the Republic of Chile. ALMA is funded in Europe by the European Organisation for Astronomical Research in the Southern Observatory (ESO), in Japan by the National Institutes of Natural Sciences (NINS) in cooperation with the Academia Sinica in Taiwan and in North America by the US National Science Foundation (NSF) in cooperation with the National Research Council of Canada (NRC). ALMA construction and operations are led on behalf of Europe by ESO, on behalf of Japan by the National Astronomical Observatory of Japan (NAOJ) and on behalf of North America by the National Radio Astronomy Observatory (NRAO), which is managed by Associated Universities, Inc. (AUI).
Alma Information, http://www.alma.info.

## Appendix A: Long Term Accumulator, Control cards, System test strategy

The Long Term Accumulator (LTA, see Sect. 5) provides:

- The first two stages of an adder tree for modes in which multiple correlator planes must be added together (that is, in time division mode). The final stage of the adder tree is provided by the Final Adder cards. The interface to the CDP is provided by a Data Port Interface card that is driven by the Final Adder cards.

- The necessary multiplication (shifting) of correlator results in planes that require this operation for support of 4-bit (16 level) correlation.

- Control card functions for the correlator cards.

The correlator system control cards and system test strategy are briefly described below. There are three types of microprocessor control cards in the correlator system:

- station control card (SCC);

- LTA/Correlator control card (LTA/CCC);

- quadrant control card (QCC).
<figure><img src="image_8.png" /><figcaption>**Fig. A.2.** Interface cards showing FPGA's with self-test capability.</figcaption></figure>

Each correlator system control card has a 16-bit Infineon C167 microprocessor configured in a standard correlator design that communicates with the correlator control computer over a Controller Area Network (CAN) serial communication bus. The SCC and LTA/CCC control cards each support an 8-bit bidirectional communication bus over the bin motherboard to individual logic cards in the system. This bus is used for control and monitoring functions such as initial FPGA personality loading, mode programming, active observation support, and various monitoring responsibilities. The QCC monitors voltages and temperatures in the system and can force a power-down when a danger is detected. Software for the ALMA correlator control cards is written in C using a Keil software development system.

Since the ALMA correlator is such a large system (32 racks, >135 000 integrated circuits) there was a need to design it with the ability to test for error free performance easily and to include the capacity to localize hardware failures quickly. The fact that the system was to be located at a 5000-m altitude, minimally staffed, site also made the ability to diagnose problems to the card level and return the correlator to service quickly with a simple card change important. Once located, a defective card can be brought down to a lower altitude service facility for repair.

Logic cards in the system that carry a sample stream have the ability to replace the input or output data with the output of a correlator standard 35-bit Pseudo random Number (PN) generator and also, the ability to test the PN sequence for errors at the input and output of the cards. Figure A.1 illustrates the self-testing ability of the Station card. Figure A.2 shows the testing capability built into the rack-to-rack interface. This test facility is used

in the initial signal training to set error-free data transmission, in testing the operational system, and in troubleshooting the system to localize hardware failures.

## Appendix B: Main components in correlator system

The finally adopted correlator system design includes many highly integrated circuits (IC) and various types of Field Programmable Gate Arrays (FPGA). The Stratix II FPGA's have been adopted for the Tunable Filter Bank cards. The main system components are listed below.

* Xilinx VIRTEX-E FPGA family.

* 1.8 VDC and 3.3 VDC system power supplies.

* 250 MHz LVDS rack-to-rack data transmission.

* Quad-flat pack IC packaging.

* ORCAD schematic capture design tools.

* Xilinx Foundations schematic capture FPGA design tools.

* In-house PCB layout using ORCAD PCB design tools.

* FR-4 fiber glass multi-layer printed circuit card fabrication.

* 2 mm Hard Metric shielded twisted-pair transmission cables.

* 2 mm Hard Metric printed circuit card interface.

* Distribution of 48 VDC power through system with dispersed DC-DC converters.

* 16-bit Infineon C167 imbedded microprocessors programmed in C.

* Altera Stratix II FPGA family, 90 nm technology.

* 1.2 VDC on-board conversion of 48 VDC power.
References

Baudry, A., & de Vos M. 2000, Proceedings of the ALMA BackEnd & Correlator Subsystems Meeting on Future Correlator Concept and Digital Filtering, Bordeaux, Internal Report

Beuther, H., Zhang, Q., Greenhill, L. J., et al. 2005, ApJ, 632, 355

Bos, A. 2000, A System Design Study of the ALMA Second Generation Correlator, Internal Report-001 Astron-28000-R1

Escoffier, R. 1997, The MMA correlator, ALMA Memo No. 166, http://www.alma.nrao.edu/memos/html-memos/alma166/memo166.html

Escoffier, R., & Webber, J. 1998, Digital filtering in the MMA, ALMA Memo No. 204, http://www.alma.nrao.edu/memos/html-memos/alma204/memo204.html

Escoffier, R., d'Addario, L., Broadwell, C., Webber, J., & Schwab, F. 1999, Computer simulation of a FIR filter, ALMA Memo No. 248, http://www.alma.nrao.edu/memos/html-memos/alma248/memo248.pdf

Escoffier, R., Webber, J., & Baudry, A. 2005, 64 Antenna Correlator Specifications and Requirements, ALMA system document ALMA-60.00.00.00-001-B-SPE, http://edm.alma.cl/forums/alma/dispatch.cgi/documents/showFile/100591/d20050708085722/No/ALMA-60.00.00.00-001-B-SPE.pdf

McCool, R. 2001, Fibre-optic link design of the Atacama IF data transfer system, ALMA Memo No. 349, http://www.alma.nrao.edu/memos/html-memos/alma349/memo349.pdf

Quertier, B., Comoretto, G., Baudry, A., Gunst, A., & Bos, A. 2003, Enhancing the baseline Alma correlator performances with the Second Generation Correlator digital filter system, ALMA Memo No. 476, http://www.alma.nrao.edu/memos/html-memos/alma476/memo476.pdf

Thompson, A. R., Moran, J. M., & Swenson, G. W. 2001, Interferometry and Synthesis in Radio Astronomy, Chap. 8 (John Wiley, & Sons, Inc.)