ZCal: Machine learning methods for calibrating radio interferometric data 
Authors: Simphiwe Zitha
Arun aniyan
Oleg Smirnov
Risuna Nkolele
Venue:  (2021)
Keywords: Radio astronomy
Calibration
Radio interferometry
ska
kat-7
MeerKat

Abstract: Calibration is the most critical data processing step needed for generating images of high dynamic range \citep{editioncasa}. With ever-increasing data volumes produced by modern radio telescopes \cite{aniyan2017classifying}, astronomers are overwhelmed by the amount of data that needs to be manually processed and analyzed using limited computational resources \citep{yatawatta2020stochastic}. Therefore, intelligent and automated systems are required to overcome these challenges. Traditionally, astronomers use a package such as Common Astronomy Software Applications (CASA) to compute the gain solutions based on regular observations of a known calibrator source \citep{thompson2017interferometry} \citep{abebe2015study} \citep{grobler2016calibration} \citep{editioncasa}. The traditional approach to calibration is iterative and time-consuming \citep{jajarmizadeh2017optimal}, thus, the proposal of machine learning techniques. The applications of machine learning have created an opportunity to deal with complex problems currently encountered in radio astronomy data processing \citep{aniyan2017classifying}. In this work, we propose the use of supervised machine learning models to first generation calibration (1GC), using the KAT-7 telescope environmental and pointing sensor data recorded during observations. Applying machine learning to 1GC, as opposed to calculating the gain solutions in CASA, has shown evidence of reducing computation, as well as accurately predicting the 1GC gain solutions and antenna behaviour. These methods are computationally less expensive, however they have not fully learned to generalise in predicting accurate 1GC solutions by looking at environmental and pointing sensors. We use an ensemble multi-output regression models based on random forest, decision trees, extremely randomized trees and K-nearest neighbor algorithms. The average prediction error obtained during the testing of our models on testing data is $ \approx 0.01 < rmse < 0.09$ for gain amplitude per antenna, and $0.2 rad < rmse <0.5 rad$ for gain phase. This shows that the instrumental parameters used to train our model strongly correlate with gain amplitude effects than a phase.


Under review as a conference paper at ICLR 2021
ZCAL: CALIBRATING RADIO INTERFEROMETRIC DATA WITH MACHINE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Calibration is the most critical data processing step needed for generating images high dynamic range (CASA cookbook] 2009). With ever-increasing data vol- umes produced by modern radio telescopes (Aniyan & Thorai 2017), astronomers are overwhelmed by the amount of data that needs to be manually processed and analyzed using limited computational resources (Ytaaaatta) 2020). Therefore, in- telligent and automated systems are required to overcome these challenges. Tra- ditionally, astronomers
package such as Common Astronomy Software Ap- plications (CASA) to compute the gain solutions based on regular observations of known calibrator source (Thompson et al. (017)) (Abebe) (2055) (Grobler) let al. 2016) (CASA cookbook]2009) The traditional approach to calibration is iterative and ume-consuming Uajarmizadeh et al. 2017). thus, the proposal of ma- chine learning techniques. The applications of machine learning have created an opportunity to deal with complex problems currently encountered radio astron- omy data processing (Aniyan & Thorat 2017). In this work, we propose the use supervised machine learning models to first generation calibration (IGC), us- ing the KAT-7 telescope environmental and pointing sensor data recorded during observations. Applying machine learning to IGC, as opposed to calculating the gain solutions in CASA, has shown evidence of reducing computation, well accurately predicting the 1GC gain solutions and antenna behaviour. These meth- ods computationally less expensive, however they have not fully learned generalise in predicting accurate 1GC solutions by looking at environmental and pointing sensors. We use an ensemble multi-output regression models based on random forest, decision trees, extremely randomized trees and K-nearest neigh- bor algorithms. The average prediction error obtained during the testing of our models on testing data is = 0.01 < rmse < 0.09 for gain amplitude per antenna, and 0.2rad < rmsse < 0.5rad for gain phase. This shows that the instrumental parameters used to train our model strongly correlate with gain amplitude effects than a phase.
INTRODUCTION
Modern-day astronomy is at an unprecedented stage, with a deluge of data from different telescopes. In contrast to conventional methods, today astronomical discoveries data-driven. The upcoming Square Kilometer Array (SKA) is expected to produce terabytes of data every hour (The SKA tele- scope). With this exponential growth of data, challenges for data calibration, reduction, and analysis also increase (Aniyan & Thorat 2017), making it difficult for astronomers to manually process and analyse the data (Yatawatta) (020). Therefore, intelligent and automated systems are required to overcome these challenges. One of the main issues in radio astronomy is determining the quality of observational data. Astronomical signals are very weak by the time they reach the Earth's surface. They are easily corrupted by atmospheric interferences, incorrect observational parameters (e.g. telescope locations or telescope pointing parameters), malfunctioning signal receivers, interference from terrestrial man-made radio sources and tracking inaccuracies (Taylor al. 1999). Therefore, is required to do proper corrections to the observational data before processing the data. Radio astronomers spend considerable amount of time performing series of preprocessing steps called calibration, which involves the determination of a set of parameters to correct the received data. These generally include instrumental as well as astronomical parameters. The general strategy for

---

Under review as a conference paper at ICLR 2021
doing these corrections makes use of calibrator source. Calibrator sources well suited for deter- mining astronomical parameters for data corrections because they have known characteristics such as the brightness. shape, and frequency spectrum (Taylor et al., 1999). This process of calibration is iterative and time-consuming. During scientific observations, different external parameters such as atmospheric pressure, temperature wind conditions, and relative humidity are collected through thousands of sensors attached to the telescopes and its adjoining instrumentation. The data com- ing from different sensors may provide information about the external conditions that may have corrupted the observed data. This piece of information is not always included in the conventional calibration steps. We propose to use machine learning methods predict the calibration solutions, looking at pointing and environmental sensor data. This is mainly motivated by the fact that cali- bration steps make corrections to data that has been corrupted by environmental parameters. study, we make use of data from the Karoo Array Telescope (KAT-7), an array consisting of seven telescopes, which is a precursor to the MeerKAT radio telescope The SKA telescope) We look at eight types of sensor data recorded during observations, with a calibrator source PKS1613-588 to generate the training and testing dataset. The overall generated dataset contains sensor data per telescope and calibration solutions for the signal received by each telescope in horizontal polariza- tion (H-pol) and vertical polarization (V-pol). These calibrator solutions are calculated using the astronomy software called Common Astronomy Software Applications.
this
2 CALIBRATION
In radio astronomy, ideally one might think that after obtaining the observed visibilities the next step would be to directly retrieve the actual visibilities of the target source and perform imaging. However, the measured visibilities Vobs are different from the actual visibilities Trree and this is due to instrumental and environmental effects (Richard Thompson et al., 2017). An example of these effects on the signal measured by radio interferometry include antenna gains (slowly and fast time-varying instrumental part), atmospheric effects, pointing errors (tracking inaccuracies) and incorrect observation parameters (antenna pointing parameters). Signal effects are classified into two types, ierectionniddeendeenn effects (affecting the signal from all directions equally) and direction- dependent effects (which vary based the sky position of the signal) (Taylor et al. 1999). These effects can be corrected by estimating the errors associated with the measured visibilities, thereby recovering the visibilities. This process is called calibration. In its simplest form, calibration minimizes the error between observed and predicted (model) visibilities by estimating the correct complex instrumental gain response (Grobler et al., 2016). Suppose for baseline pair (i.j), the observed visibility is Velue (t) and the true visibility IS VT at observation time t. The basic calibration formula is written as,
Viobe Gi, V..VV.yuee + (()
(1)
where, Gi,jte) denotes the complex antenna gains for baseline (i.j) as result of unwanted effects and may vary with time (Thompson et al., 2001). The extra term ei, (t) is a stochastic complex noise (Taylor al.) 1999). Most of the corruptions in data occur before the signal is correlated and the response associated with antenna does not depend on the response of antenna j. Note that the sources that are the subject of astronomical investigation will be referred to as "target sources" to distinguish them from calibrator sources (Thompson et al. 2001)
3 KAT-7 TELESCOPE
The KAT-7 is a seven-dish interferometry that was built as an engineering prototype for techniques and technologies in preparation for the 64-dish Karoo Array Telescope (MeerKAT) (Foley et al.) 2016). These instruments are located in the Northern Cape Karoo desert region and are operated remotely from Cape Town. The construction of KAT-7 began in 2008 with the writing of the tele- scope requirements specification and was completed in 2010. It was then operated in engineering (commissioning) mode until its shut-down in 2016 (Foley et al. 2016).
3.1 SENSOR DATA
During science observations, different external parameters like atmospheric pressure, temperature wind conditions, and relative humidity are also collected through thousands of sensors attached
2

---

Under review as a conference paper at ICLR 2021
to the telescopes and its adjoining instrumentation. The data coming from different sensors may provide information about the external conditions that may have corrupted the observed data. This piece of information not always included in the conventional calibration steps. We propose to machine learning methods to predict the calibration solutions looking at pointing and environmental sensor data. This is mainly motivated by the fact that calibration steps do corrections to data that is corrupted by environmental parameters, In this study, we make use of the data from the Karoo Array Telescope (KAT-7). We look at pointing azimuth, elevation, scan, offset, temperature, wind speed, air pressure. relative humidity sensor data recorded during observations with a calibrator source PKS1613 586 to generate the training and testing dataset. The overall generated dataset contains sensor data per telescope and calibration solutions for correcting the signal received by each telescope horizontal polarization (h-pol) and vertical polarization (V-pol). These calibrator solutions are calculated using one of the traditional astronomy software called CASA which is used for data calibration and imaging in radio astronomy.
3.2 PREPARATION OF TRAINING DATA
The objective of this study is to find correlations between calibration solutions and sensor informa- tion on the telescope. Therefore, the main dataset for the study is the time-based sensor information of each antenna. The process of data collection encompasses all of the steps required obtain the desired data in digital format. Methods of data collection include acquiring and archiving new ob- servations, querying existing databases according to the science problem hand, and performing as necessary any cross- mooossna or data combining (Ball & Brunner 2010).
In every observation, the collected data stored by the data capturing system in the Hierarchical Data Format (HDFS), which is a set of file formats designed to store and organize large amounts of data. The HDF5 file consists of two parts meta-data and observed visibilities. In meta-data one finds static information of the data set, including observer, dump rate and all the available subarrays and spectral windows in the data selection criteria (antennas, channel frequencies, targets, scan infor- mation) and sensor data of interest as function of time. The data observed by the radio telescope the form of complex numbers referred to visibilities. Each source observed contains its own visibilities function of time along with sensor data, which keep record of the telescope's activity and behaviour as these are observed.
0.08
10'
20'
0.06
J20M Derlination
30'
0.04
40'
0.02
Beee 4
50'
-900
- 0.02
10'
20'
0.04
16"
21
19
17
15th
13
-0.08
12000 Right. Asvension
-0.08
Figure I: An image of the phase calibrator PKS1613-586 located at RA, Dec (J2000) using CLEAN task in CASA.
In preparation for the training and testing dataset, we look at environmental sensors and instrumental sensors recorded during observations with flux calibrator and a phase calibrator source PKS1613- 586 Figure I The chosen sensors of interest from each observation are: air temperature, wind speed, wind direction, air pressure, relative humidity, actual refraction elevation, actual refraction azimuth, actual scan elevation, actual scan azimuth, actual pointing elevation and actual pointing azimuth.
3

---

Under review as a conference paper at ICLR 2021
4 PROPOSED METHOD
Different calibration techniques have been developed with the enhancement of the dynamics of the modern radio astronomy instruments address these challenges raised by the new instruments, providing precise calibration performance. These techniques are loosely classified into first genera- tion calibration (IGC), second generation calibration (2GC) and third generation calibration (3GC) (Noordam & Smirnov 2010). In this study, we concentrate on generating IGC calibration with the help of machine learning techniques. Our aim is to provide a machine learning model that predicts calibration solutions from sensor data from the telescope. This approach would help to speed up the calibration processes and decrease the time period of the calibrator monitoring, thus improving the time duration duration for tracking the target source observed as shown in
Tarpet
SaMECe
Caktuate sclutions from sensors
Observation procedure and calibration using
(b) Observation procedure using machine learning
CASA
Figure 2: Observation procedures
Several different approaches are employed machine learning regression. These approaches learn the relationship between the input and output by fitting model directly from the data. In this study, we consider tree-based approaches: Decision tree, random forest, extremely randomized tree, and the neighborhood search approach: (K-nearest neighbor) to tackle problem. We call our approach ZCal model which is multi-output regression. We formulate our regression estimation problem follows:
Suppose we have a feature matrix of sensor data
T12
273
T Iin
221
T22
T23
F2n
X,
(#i.)) E Rdxx ,iE (1,2,...., (1,2,...,n)
[dd1
222
Ødn
and corresponding complex target variables to learn and predict on,
V11
V12
V13
V1m
T21
1/2
y23
92m
Y,
= (2k,1) ddm ke (1, (1,2,..., m,]
Dan
Mdz
Mas
ydm
where each column represents vector of length d, containing unique calibration solutions func- tion of time per observation represented as complex variable A = A(coss 6 - iin Φ) for each polarization &&V (Thompson al. (017].
Due to different physical causes on the received signal, we therefore choose to treat the antenna phases and amplitudes separately by splitting equation the complex variable into gain amplitude solutions Ae and gain phase solutions ø.
We construct learning machine, M X, + Y,, which when given validation of sensor examples, X minimises some measure of discrepancy between its prediction MX;) z Y., and the value of Y., where M represents the predictor. We measure the discrepancy using four com- monly used statistical measures regression (Borchani et al. 2015); coefficient determination, explained variance, root mean squared error (RMSE) and root mean absolute error (RMAE).
The aim of this regression exercise is predict multiple target variables Y, hence it is referred to multi-output regression. The learned model will then be used predict multi-output values Y

---

Under review as a conference paper at ICLR 2021
of all target variables of the new incoming unlabelled instances X It has been proven that multi- output regression methods provide means to model the multi-output datasets effectively and produce better predictive results (Borchani al. 2015). This method does not only consider the underlying relationships between the features and the corresponding targets but also the relationships between the targets themselves, thereby producing simpler models with better computational efficiency (Bor- chani et al., 22015), Borchani et al. (2015) discuss several applications of multi-output regression including the challenges such missing data, i.e., when some features or target variables not observed.
RESULTS AND DISCUSSION
Antenna
Extremely randomized phase
Rmac
R2score
Explained
Unifamm average
0.337
0.336
0.936
0.936
Uniform average-V
0.235
0,301
0.956
0. 957
Extremely randomized free amplitude
Unifarm average-H
0.027
0.083
0.925
0.925
Uniform average-V
0.027
0.083
0.923
0.923
The table shows the performance of the extremely randomized tree algorithm as the best method in predicting the amplitude and phase gain solutions for both H and V polarization as shown in figures 3] The values shown represents the uniform average of all KAT-7 antennas, i.e., all output measures are averaged with uniform weight. Most of the predictions stay near the ideal truth values with RMSE and RMAE << 0.5, Rº and explained variance V are converging to
Table 1:
0.
H-poll phase learning algorithm
H-pol amplitude mas per learning algorithm
3.7
on
0.34
0.32
0.5
0.28
0.06
52
0.2
0.
antemnal
entoonal
0.20
antoovat
anteondd
artsonal
V-pol phase mos per learning algorithm
V-pol amplitude rmse learning algorithm
0.55
0.6
0.00
0.08
004
20
antnnnt
antonnl
antoonal
Antertzai
Figure 3: Root mean square error for each learning algorithm in predicting the H and V polarization phase and amplitude gain solutions for each antenna. The blue, green, red and purple lines represent the different learning algorithms used in the experiment.
From Figure S we observes that the learning algorithms have learned this behaviour pattern with rms accuracy < 0.5 Radians during test time, resulting in predicting zero phase gain solutions for

---

Under review as a conference paper at ICLR 2021
antenna 5 for both polarizations. Similarly for amplitude gain, we have achieved an error of less than 0.02 during training time. We validated our models looking at one observational data for a calibrator source PKS 1613-586. We provide models with sensor data during tracking of the calibrator source as input to our trained model. The phase and amplitude output predictions Figures and 5 shows that the models have learned generalize for the reference antenna 5, by predicting phases for antenna 5 H and V polarization, assuming that the antenna was stable without any drop- outs during the period of the observation with antenna 6 and antenna 7 being a bit nosier. In Figure 3 though the models were supposed to perform differently because of their parameter settings, one notices that the random forest, K-nearest neighbor and the extremely randomised trees methods are very close to one another in rms error as function of antenna IH, 2H, 5H, 6H and 7H, whereas there are large large rms error bars in each model per antenna 3H and 4H. Such behaviour gives us an idea about the instability of these two antennas. This is valuable information that will make contribution towards flagging corrupted data due to unstable antennas.
These models managed to learn the most critical part, i.e., the sinusoidal variation of the gain am- plitude solutions over time as shown in Figures 5 However, We observe that the machine learning amplitude is performing lower than CASA solutions with factor of 33.33%.
In Figure 5 The model predicts a drop in amplitude as a function time in the middle of the ob- servation from 0.12 to 0.09 level of CASA, while as the true amplitude have not dropped. From this unique behaviour observed, we can conclude that it is either due to prediction failure as the models have been trained on limited amount of observations and therefore failing on different observation settings, or it could be a behaviour that CASA did not detect as it does not include sensor data when calculating its calibration solutions.
Antl-phase gain solutions
Ant2-phase gain solutions
Phaze (Dog)
Phase
Time index
Time index
Ant3-phase gain solutions
Ant4-phase gain solutions
Phoze (Dog)
Phose (Deg)
Time index
Time index
AntS-phase gain solutions
Ant6-phase gain solutions
6415
0414
Phosee 10
0.03
phoze (Deg)
Time
Ant7-phase gain solutions
Time index
Figure 4: This figure shows the response of the extremely randomized tree learning algorithm in predicting the phase gain solutions for the calibrator PKS1613-586 from the validation data. In each subplot we compare the predicted phase gain solutions (where H polarization is represented by red and V-polarization represented by yellow) with the CASA generated phase gain solutions (where H polarization is represented by blue and V-polarization is represented by green). The ex- tremely randomized tree algorithm is failing to predict the gain phase solutions for some antennas and performing well in others with few outliers.

---

Under review as a conference paper at ICLR 2021
-ampliuie gain solutions
Ant2-amplitude gain solutions
612 6130
012
11x
61100
4.18
100
130
139
Index
Time
Amp3-ame gain solutions
Ant4-amplitude gain solutions
Anplitude
30
100
Time index
Time index
Ant5-amplitude gain solutions
Ant6-amplitude gain solutions
Time index
Time index
Ant7-amplitude gain solutions
2014
Time Index
Figure 5: This figure shows the response of the extremely randomized tree learning algorithm in predicting the amplitude gain solutions for the calibrator PKS1613-586 from the validation data. In each subplot we compare the predicted amplitude gain solutions (where polarization is rep- resented by red and V-polarization is represented by yellow) with the CASA generated phase gain solutions (where H polarization is represented by blue and V-polarization represented by green). The extremely randomized tree algorithm is failing the antennas, for solutions at time index < 70. We further observe that at time index 80 the algo- rithm performing better in predicting the amplitude solutions, slightly close to CASA the new observation dataset.
predict the gain amplitude solutions in all
CONCLUSION
We have shown that the application of machine learning techniques to telescope sensor data opens avenue in the development of calibration methods. We used the telescope sensor data to leam the variability of the complex gain calibration solutions for the calibrator PKS1613-586 as a function of time. The implementation of the ZCal algorithm is based regression machine learn- ing algorithms, to predict the calibration solutions and study each antenna's 's behaviour. Using the IGC calibration solutions obtained with CASA, we constructed a matrix of training sample ny and testing sample Tyr to train the machine learning algorithms (decision tree, random forest, extremely randomised trees, and K-nearest neighbor) to be able to discern the patterns that relate complex gain solutions to external parameters. Since gain solutions are complex, we have implemented ZCal to learn on phase and amplitude separately. Each learning algorithm ran on the learning sample N times and its error was estimated on the test sample. We presented a statistical framework to mea- sure the accuracy of each multi-output regression mode and our results are encouraging with an rms eтrоr of ss 0.5 rad during testing of our models using the testing data for gain amplitude and phase. Comparing the performance of these algorithms, the random forest, extremely randomized tree and K-ncarest neighbours environmental and the pointing sensor readings were more strongly correlated the amplitude than phase. Consequently, the ability predict gain-phase was overall poor; gain-amplitude prediction
shown to be the best for purpose. We observed that the

---

Under review as a conference paper at ICLR 2021
was accurate in some cases (capturing non-trivial behaviour such oscillations), and completely failed in others. The purpose the study was to show that machine learning techniques can make available connections "blindly" without access to physical intuition. The accurate prediction of gain-amplitudes, in some cases, suggests that this is indeed feasible. It is not clear what caused the failed predictions, although that our sensors were not sensitive to. We can therefore expect that with access to a larger array of sensors, the ZCal approach will be able make better gain predictions.
always speculate
physical differences between observations
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ermias Abebe. A study of potential calibrators using the kat-7 radio telescope. 2015.
AK Aniyan and Kshitij Thorat. Classifying radio galaxies with the convolutional neural network. The Astrophysical Journal Supplement Series, 230(2):20, 2017.
Nicholas M Ball and Robert Brunner. Data mining and machine learning in astronomy. Interna- 106, 2010.
tional Journal of Modern Physics D,
Hanen Borchani, Gherardo Varando, Concha Bielza, and Pedro Larrañaga. A survey multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(5):216
233, 2015.
Public Release CASA cookbook. Casa user reference cookbook. 2009.
AR Foley, Alberts, RP Armstrong, Barta, EF Bauermeister, Bester, S Blose, Booth, DH Botha, SJ Buchner, et al. Engineering and science highlights of the kat-7 radio telescope. Monthly Notices of the Royal Astronomical Society, 460(2): 1664-169,67 2016.
TL Grobler, AJ Stewart, SJ Wijnholds, JS Kenyon, and OM Smirnov. Calibration artefacts in radio interferometry-iii. phase-only calibration and primary beam correction. Monthly Notices of the Royal Astronomical Society, 461(3):2955-2992 2016.
Milad Jajarmizadeh, Lariyah Mohd Sidek, Sobri Harun, and Mohsen Salarpour. Optimal calibra- tion and uncertainty analysis of swat for an arid climate. Air, Soil and Water Research, 10: 1178621117311992, 2017.
Jan E Noordam and Oleg M Smirnov. The meqtrees software system and its for third generation calibration radio interferometers. Astronomy & Astrophysics, 524:A61, 2010.
A Richard Thompson, James M Moran, and George W Swenson Jr. Interferometry and synthesis in radio astronomy. Springer Nature, 2017.
Greg B Taylor, Chris Luke Carilli, and Richard Perley. Synthesis imaging in radio astronomy ii. ASPC, 180, 1999.
The SKA telescope. The ska telescope. http:://www..stteeeespppp 2017.
Accessed May
A Richard Thompson, James M Moran, and George W Swenson Jr. Interferometry and synthesis in radio astronomy. In isra, 2001.
Anthony Richard Thompson, James M Moran, George Warner Swenson, al. Interferometry and synthesis in radio astronomy. Springer, 2017.
Sarod Yatawatta. Stochastic calibration of radio interferometers. Monthly Notices of the Royal Astronomical Society, 493(4))6071--6078 2020.


Reviews:
[{"review_id": "wtu6lSwWd7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThe paper presents a study of using machine learning methods to calibrate a radio telescope using information from sensor data on, e.g., atmospheric conditions. The authors consider tree- and neighbourhood-based methods for predicting amplitudes and phases for seven antennas. The results show that the methods perform quite well in terms of RMSE and explained variance.\n\n# Evaluation\n## Strong points\nIt is a timely and interesting application, and it is clear that the field of astronomy holds lots of potential for machine learning applications.\n\n## Weak points\n* From a machine learning point of view, the methods are very simple.\n* The evaluation overall is confusing and somewhat weak. For instance, it is unclear to me whether the competing method, CASA, can be considered as the ground-truth or as a method to outperform.\n* The evaluation based on figures 4 and 5 seem to be done qualitatively.\n\n## Recommendation\nRejection.\n\nThe main reason for my recommendation is that ICLR is not the right venue for this paper. It does not deal with representation learning, and the methods are too basic for a top-tier machine learning conference in general.\nIn addition, the experimental evaluation, which is the main body of the paper, is quite weak and will need significant improvements.\n\n## Detailed feedback\nThe methods considered in the paper (k-NN and tree-based methods) all base their predictions directly on the input features; thus, no representation learning is happening. Thus ICLR is not the right venue, and I doubt the paper would be accepted to a top-tier machine learning conference since it only considers (quite simple) off-the-shelf methods. KDD might be a better fit or an astronomical journal such as PASP.\n\nFurthermore, there seems to be some confusion with regards to what a multi-output regression model is. As described in Borchani et al. (2015), multi-output regression methods also make use of correlations between outputs. It seems the authors are aware of this, as they cite the models' ability to make use of the 'relationships between the targets', which I assume refers to modelling correlations between the output dimensions. However, this is not how the methods used in the paper work. For both k-NN and tree-based methods, there is a clear separation between inputs and outputs, so the paper does not deal with multi-output regression either. This is not a problem per se, however, as it is not clear that a multi-output model would be better suited for the task considered here.\n\nThe paper completely lacks a section on prior work - both regarding related uses of machine learning and different ways to perform calibration.\n\nRegarding the methods, and given that the predictions seem to be part of a time-series, I am not convinced the chosen ones are the most suited for the tasks. It would make sense to test methods from signal processing as well, for instance Gaussian processes or a recurrent neural network like an LSTM.\nThere are also no specific details on how the methods in the paper are constructed. Do they only use the instrumental and environmental sensor data as inputs, or do they use the temporal dimension in some way? Which hyperparameters were adjusted, and how were they selected?\n\nThere is very little information regarding the experimental setup. How large were the training and test sets? What are the dimensionalities of the inputs and outputs? It seems like only one source was considered; why is that? Only testing one source seems brittle. Similarly, was cross-validation used? If not, why? If it was, we need details on how it was set up and what it was used for (hyperparameter tuning, experimental evaluation, etc.).\n\nFor the experimental evaluation, I am confused about whether CASA is considered the ground-truth to compare with, or whether it is a method to outperform. Regardless, would it be possible to evaluate both CASA and your method on simulations to get an idea about how well each performs?\n\nIt also confuses me that antenna 5 is referred to as 'the reference antenna'. In what way was it used as a reference?\n\nFurther to the experimental evaluation, it would make sense to include the performance of a baseline or two, such as linear regression. I would also like to see error bars in figure 3, e.g., from cross-validation.\n\nSince the paper deals with quite simple methods, I would expect some analysis of why they work and when they might fail. In figure 3, especially for the amplitude, some antennas have dramatically larger errors. What happens here? Also, which of inputs were most informative for predicting the phase and amplitude? You could get a feeling for the importances by, e.g., training the models on only one feature, or all features expect one, and repeating this for all features in turn.\n\n# Questions\n1. Why did you choose these particular machine learning methods?\n2. How did you determine which method was the best? From the plots in figure 3, it looks like k-NN is slightly better overall.\n3. Did you perform any cross-validation? If so, where are the error bars in figure 3? If not, why not?\n  1. You mention 'large rms error bars in each model per antenna 3H and 4H'. Which error bars are you referring to?\n4. Are the RMSEs in Figure 3 wrt. the CASA estimations?\n  1. Also, to me (with no background in radio astronomy, though), it seems like a phase error of 0.3-0.4 is quite large (assuming one phase is 2\\pi). How large errors can we expect from CASA?\n\n# Suggestions for improvement\n* Evaluate more than one calibration source. Without more sources, we can't be sure if your method performs consistently well.\n* Figures 4 and 5: it might be useful to see instead residuals between your method and what you consider ground-truth.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VrrNtIv9v4n", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY \n\nThis work proposes to use an ensemble of very well-known machine learning models (mainly tree-based methods) to calibrate radio interferometric data from the KAT-7 telescope. This provides a more efficient alternative to the traiditional approach, which is based on the astrophysicists workforce.\n\nREASONS FOR SCORE\n\nIn my humble opinion, this work does not present a machine learning contribution of interest for the ICLR community. It applies standard and well-known approaches to an specific remote sensing problem.  I would point the authors to different venues, such as the IEEE International Geoscience and Remote Sensing Symposium. \n\nMoreover, I would recommend the authors to compare their approach to some others methodologies used in the field. Right now, the experimental section only analyzes the results of the proposed approach, with no baselines.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HwVDoKQKMY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the manuscript, \"ZCal: Calibrating radio interferometric data with machine learning\", the authors consider the problem of calibrating radio telescope data against a source of known brightness, which they (correctly) identify as a multi-variate, multi-output regression problem.  Compared to a typical accepted ICLR paper the application area does not appear to be particularly novel with regard to methods deployed (off-the-shelf KNN, decision tree, random forests etc) or the size/structure of the dataset; and no new insights into these methods (such as might generalise to help solve other problems) are recovered.  For this reason I would be surprised if a revised draft could encourage me to change my recommendation to accept.  For what it's worth, I would have been interested to see a description of the CASA routine for calibration to understand the difference between this and the ML algorithms.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As discussed by several reviewers the paper is an application of classical ML approaches for a very relevant problem of calibration of radio interferometers. The application is interesting but lacks novelty in terms of ML methodology and the experiments do not provide a meaningful comparison between the state of the art and the proposed approach or justification for the choice of predictors and their parameters. This paper in clearly not a good fit for a general ML conference.
Decision: Reject