A Real Time Processing System for Big Data in Astronomy: Applications to HERA
Creators: name: La Plante, Paul
name: Williams, Peter K. G.
name: Kolopanis, Matthew
name: Dillon, Joshua S.
name: Beardsley, Adam P.
name: Kern, Nicholas S.
name: Wilensky, Michael
name: Ali, Zaki S.
name: Abdurashidova, Zara
name: Aguirre, James E.
name: Alexander, Paul
name: Balfour, Yanga
name: Bernardi, Gianni
name: Billings, Tashalee S.
name: Bowman, Judd D.
name: Bradley, Richard F.
name: Bull, Phil
name: Burba, Jacob
name: Carey, Steve
name: Carilli, Chris L.
name: Cheng, Carina
name: DeBoer, David R.
name: Dexter, Matt
name: Acedo, Eloy de Lera
name: Ely, John
name: Ewall-Wice, Aaron
name: Fagnoni, Nicolas
name: Fritz, Randall
name: Furlanetto, Steven R.
name: Gale-Sides, Kingsley
name: Glendenning, Brian
name: Gorthi, Deepthi
name: Greig, Bradley
name: Grobbelaar, Jaspar
name: Halday, Ziyaad
name: Hazelton, Bryna J.
name: Hewitt, Jacqueline N.
name: Hickish, Jack
name: Jacobs, Daniel C.
name: Julius, Austin
name: Kerrigan, Joshua
name: Kittiwisit, Piyanat
name: Kohn, Saul A.
name: Lanman, Adam
name: Lekalake, Telalo
name: Lewis, David
name: Liu, Adrian
name: MacMahon, David
name: Malan, Lourence
name: Malgas, Cresshim
name: Maree, Matthys
name: Martinot, Zachary E.
name: Matsetela, Eunice
name: Mesinger, Andrei
name: Molewa, Mathakane
name: Morales, Miguel F.
name: Mosiane, Tshegofalang
name: Murray, Steven
name: Neben, Abraham R.
name: Nikolic, Bojan
name: Parsons, Aaron R.
name: Pascua, Robert
name: Patra, Nipanjana
name: Pieterse, Samantha
name: Pober, Jonathan C.
name: Razavi-Ghods, Nima
name: Ringuette, Jon
name: Robnett, James
name: Rosie, Kathryn
name: Santos, Mario G.
name: Sims, Peter
name: Smith, Craig
name: Syce, Angelo
name: Thyagarajan, Nithyanandan
name: Zheng, Haoxuan
Publisher: name: arXiv
Year: 2021
Subjects: subject: Instrumentation and Methods for Astrophysics (astro-ph.IM)
subject: FOS: Physical sciences
subject: FOS: Physical sciences

description: As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.
description: 15 pages, 3 figures, published in Astronomy and Computing