## 2 Materials and Methods  
In this study, a total of 33 physicians (internal medicine \(n=16\); intensivist \(n=4\); family physician \(n=5\); pneumology \(=1\); pediatrics \(=1\); and emergency medicine, \(n=6\)) with advanced experience in performing and interpreting LUS, from 29 different healthcare centers in Spain, independently evaluated previously acquired ultrasound videos of 20 patients. All had more than 3 years' experience performing and interpreting LUS. The acquisitions corresponded to patients with COVID-19 diagnosed by nasopharyngeal RT-PCR for SARS-CoV-2 obtained in the internal medicine service of two different hospitals in Madrid collected during the summer of 2021 [8]. All studies were collected by two physicians (YT-C, AT-V), who followed a 12-areas LUS protocol and a 0-3 point per finding score system [18] (Figure 1). Specifically, each area was scored from 0 to 3 according to the observed patterns (Figure 2). Score 0 is associated with the physiological horizontal artifacts, A-lines. Score 1 is assigned when isolated vertical artifacts appear (B-lines). Score 2 represents confluent B-lines in less than 50% of the pleural line. Score 3 is associated with confluent B-lines extending more than 50% of the pleural line, as well as subpleural or lobar consolidation or pleural effusion. In each zone, the ultrasound probe was used in longitudinal and transverse positions, and a 3 s video of 20 fps was recorded. In total,  
Figure 1: Ultrasound 12 scanning locations (**top row**) and the two orientations of the transducer considered (**bottom row**). “R” stands for rib. 24 videos of 3 s each were acquired per patient. No patient had more than one scan in the database. Each physician assigned the highest score to the 3 s video. In all cases, the data were obtained with a ULTRACOV ultrasound scanner prototype using a 3.5 MHz convex probe with 128-channel ultrasound electronics [8]. This resulted in a total of 480 videos (28,800 frames). The imaging depth in all cases was set to 13 cm. More details of the data acquisition can be found in a recently published work, which focused on the automatic calculation of LUS score [8]. The data were acquired from a study conducted at a tertiary academic hospital and an emergency field hospital that investigated a reduction in the exploration time per patient when using an ultrasound system developed specifically for LUS. The study was reviewed by our institutional review board (IRB) and approved at both participating sites. Informed consent was obtained for each patient. Several LUS protocols have been proposed for the lung assessment of COVID-19 patients based on the number of areas or points to explore. We adopted a 12-zone scanning protocol, which was previously validated and shown to be consistent with higher ICC and a higher degree of concordance with CT [18]. The selected patients for this study (\(n=20\)) were selected from the total acquired dataset from that study (\(n=28\)) so that half of the cases (\(n=10\)) corresponded to patients in relatively good condition (with a total score between 1 and 7 based on the in situ assessment of the LUS expert), while the other half (\(n=10\)) corresponded to patients who had a moderate condition (with a score between 8 and 18 based on the in situ assessment of the LUS expert). As the LUS device was not located within the ICU, no severe cases were present in the database. The physicians had no information about the patients in the survey and were blinded to their history and clinical information. They also did not have access to the characteristics of the scanner and the evaluations performed by the other physicians. All the videos from the same patient were supplied together before moving on to the next patient, as would happen in a regular patient examination. The sonographer expert who collected the videos also conducted the survey, so that a comparison between the findings obtained during the examination and those observed in the surveyed videos was also undertaken.