**_Types and formats of data_ **

Within the RADIOFOREGROUNDS project, we consider three types of data products.

  1. **Maps** . Most of the data generated during this project will correspond to either sky maps or component separated maps. The standard format to be used within this project is the same one adopted by the CMB community at large: the HEALPIX ( _http://healpix.jpl.nasa.gov_ ) pixelization scheme. HEALPix is an acronym for Hierarchical Equal Area isoLatitude Pixelization of a sphere. As suggested in the name, this pixelization produces a subdivision of a spherical surface in which each pixel covers the same surface area as every other pixel. All Planck and QUIJOTE maps are provided as FITS files under this scheme. This is the same scheme in which all the Planck maps appear inside the PLA, or the scheme in which the WMAP data are provided in NASA’s LAMBDA ( _http://lambda.gsfc.nasa.gov_ ) archive. 

  

  2. **Catalogues** . The project will also provide complementary information to the PCCS (Planck Catalogue of Compact Sources), including the flux densities, polarization fractions and angles of specific radiosources at the QUIJOTE frequencies. Here, we will provide the results as extensions in binary fits tables, which could be added to the ESA database. 

  

  3. **Models** . We will also provide specific physical models of the different radio foreground components. Although these models will be described in detail in the relevant publications, they will also be implemented in a specific software tool, which will allow the possibility of carrying out specific predictions/simulations at the requested frequencies, or even doing some basic analysis to the data (e.g. aperture photometry, combination of frequencies, etc.). 

  

  

**_Maps and models_ ** . The proposed format follows a **standard HEALPix FITS
binary table** , with:

  * N  side  = 256 

  

  * Maps in RING ordering 

  

  * Maps in Galactic coordinates. 

  

  * Units. In general, they are K_CMB (or MJy/sr for Planck 545 and 857 GHz channels). 

  

  * Map columns called MAP_I, MAP_Q, MAP_U. 

  

  * Noise covariance columns called COV_II, COV_IQ, COV_IU, COV_QQ, COV_QU, COV_UU. 

  

  * Maps without polarisation information contain only MAP_I and COV_II columns. 

  

  * Name of the experiment in the TELESCOP header. 

  

  * Name of the channel in the CHANNEL header. 

  

  * Effective central frequency for that channel in the FREQUENC header. 

  

  * VERSION number of the map. This is relevant for maps that might be updated during the lifetime of the project. 

  

**_Re-using existing data_ **

All the products of ESA’s Planck collaboration are publicly available in the
Planck Legacy Archive (PLA, _http://pla.esac.esa.int/pla/_ ). NASA’s WMAP maps
are also publicly available through the LAMBDA platform (
_http://lambda.gsfc.nasa.gov/_ ). Other ancillary data sets (see table 1.1)
will be obtained from the PLA, the LAMBDA archive, or the CADE database (
_http://cade.irap.omp.eu/_ ). In the case that HEALPIX maps for those
ancillary data are not found in those archives, they will be generated within
the group based on the existing public data. All the maps adapted for this
project will be degraded to a common angular resolution of one degree.

**_Expected size of the RADIOFOREGROUNDS data base_ **

The total size of our database is expected to be approximately 3 GB. In more
detail:  
  
<table>  
<tr>  
<th>

**Data source**

</th>  
<th>

</th>  
<th>

**Description**

</th>  
<th>

**Estimated volume (MB)**

</th> </tr>  
<tr>  
<td>

QUIJOTE

</td>  
<td>

</td>  
<td>

Four Healpix maps containing I,Q,U Stokes parameters, and noise covariances.
Frequencies: 11, 13, 17, 19 GHz.

Angular Resolution: 1 degree.

</td>  
<td>

4 files of 27MB each

</td> </tr>  
<tr>  
<td>

PLANCK

</td>  
<td>

</td>  
<td>

Seven Healpix maps containing I,Q,U Stokes parameters, and noise covariances
(30, 44, 70, 100, 143, 217, 353 GHz), plus two Healpix maps containing only
Stokes I parameter (545 and 857 GHz). Angular resolution: 1 degree.

</td>  
<td>

7 files of 27MB and 2 files of

6MB

</td> </tr>  
<tr>  
<td>

WMAP

</td>  
<td>

</td>  
<td>

Five Healpix maps containing I,Q,U Stokes parameters, and noise covariances.
Frequencies: 23, 33, 41, 61, 94 GHz. Angular Resolution: 1 degree.

</td>  
<td>

5 files of 27MB each

</td> </tr>  
<tr>  
<td>

Ancillary

</td>  
<td>

</td>  
<td>

Healpix maps containing Stoke I parameters, Stokes Q and U (when available),
and noise covariances. Remaining experiments in Table 1.1. Angular Resolution:
1 degree.

</td>  
<td>

Estimated maximum number

of 10 files of 27MB each

</td> </tr>  
<tr>  
<td>

Catalogues

</td>  
<td>

</td>  
<td>

Tables with celestial coordinates (RA-DEC or Galactic), and fluxes at
different frequencies. Detailed format TBC.

</td>  
<td>

200 MB maximum

</td> </tr>  
<tr>  
<td>

Models of foreground emissions

</td>  
<td>

the

</td>  
<td>

Healpix maps containing the coefficients of the parameterised models of the
foreground emission derived in the project.

</td>  
<td>

27MB each map. We expect a maximum number of 100 maps.

</td> </tr> </table>

The data products and software tools to be generated within the
RADIOFOREGROUNDS project will be of enormous importance not only for the
Cosmology community, but also to other communities in Astrophysics. In
particular, this data will be useful to study the gas and dust content of the
Milky Way, the cosmic ray distribution and the Galactic magnetic field
structure (especially at large scales), the physics of molecular clouds, SNRs,
HII regions and other regions with AME, and for the study of evolutionary
models of radio sources both in intensity and polarization. It will also
provide a very valuable resource to estimate the effect of radio foregrounds
on the detection of the CMB B-mode of polarization with future satellite and
sub-orbital experiments, helping to design the configuration of such
experiments in an optimal way. We believe that our proposed software tools
will be very helpful for these communities.

#  2\. FAIR data 2. 1. Making data findable, including provisions for metadata

This section considers the standards that RADIOFOREGROUNDS will use to
represent data generated by the project, and the additional standards around
data security, etc. that might be useful to govern the data used within and
generated by the project. In addition, this section also includes a
consideration and selection of the metadata that will be most effective in
describing the RADIOFOREGROUNDS data set.

This includes a consideration and evaluation of existing standards and our
reasoning for selecting specific standards. At present, the consortium has
agreed to avoid proprietary data formats as far as possible, as these will
make it difficult for both the consortium and any external stakeholders to
utilise the RADIOFOREGROUNDS data after the close of the project. This is for
three specific reasons – first, because proprietary programmes evolve and data
formats may become defunct, thus maintaining proprietary data formats
represents a significant need of investment to keep the data relevant and
accessible. Second, because access to the data would be restricted to those
who have access to the appropriate analysis tools if a proprietary data format
was utilised. Third, because it would be difficult to combine RADIOFOREGROUNDS
data with other data as proprietary data formats often raise interoperability
issues. In addition to these, RADIOFOREGROUNDS will also consider standards
around other issues that could govern the storage and representation of
RADIOFOREGROUNDS data. This includes data security standards such as ISO
27001. As the RADIOFOREGROUNDS data management plan develops alongside the
project, partners will consider each of these relevant standards and make an
informed selection for RADIOFOREGROUNDS toolset and evaluation data. The next
iteration of this document in M18 will provide more information.

Finally, the project will consider and select effective metadata for
describing RADIOFOREGROUNDS toolset and evaluation data. Effective metadata
will assist project partners and potential additional data users by providing
“clear and detailed data descriptions and annotation”, version numbers,
accessibly written, accompanying documentation and any contextual information
that is relevant when the data is re-used (UK Data Service, 2016). This
consideration of meta-data is linked to the next section on data exploitation,
in that the metadata provided should consider the uses to which the data can
be put in order to provide sufficient and relevant information to potential
users.

##  2.2. Making data openly accessible

With respect to the Observatory data provided to RADIOFOREGROUNDS, the
partners have agreed that the project coordinator will manage access to the
data during the duration of the project. Each node has agreed to provide
access to the data required for the project, provided that the data is only
accessed by consortium partners and only for project activities. If needed,
partners will provide the anonymised data directly to the coordinator, who
will store the data in their existing ICT infrastructure. The anonymisation of
the data means that it is acceptable to share it within the consortium,
however partners have agreed not to seek access to raw data. Any partner with
a user password will be able to access the historical data, and the simulated
real-time data to develop or test their algorithm or software. With respect to
the data that will be used for the final testing of the RADIOFOREGROUNDS
solution, this data will be housed within IAC facilities, and partners will be
able to access it during the testing.

The issue of access to RADIOFOREGROUNDS toolset and evaluation data after the
end of the project will be initially considered in the next iteration of this
document in M18, and will be finalised in the last iteration in M36. In
principle, and as described in the original proposal, **the project will make
publicly available all maps, catalogues and models described in Section 1 of
this document** . The consortium will decide if additional data (e.g. maps of
small sky regions) is of interest to the community and can be open access as
well. The data will be made accessible by FITS format, as described in Section
1\. The website will provide a list of public FITS gathered by categories.
Additionally, it will develop a REST API that provides access to the FITS and
other metadata to the community. The users could download the data in the
website or queries can be performed to the API using only your browsers
address bar (only GET methods will work for this approach). For other type of
request (POST, PUT, DELETE) they will need install a RESTful addon or
programmatically (this is the more useful way to consume a REST web services).
API documentation will be provided to end-users. A user manual is created to
describe relevant processes for data access. Basic API Calls are detailed,
separately.

**_Storage and processing._ **

With respect to storage, each of the types of data will be stored and backed-
up slightly differently. IAC will store the provided data in their existing
infrastructure. All of this data will be backed-up in the cluster,
automatically, so that the data can be recovered in the event of an incident.
The security of this data will be maintained via the security policies,
methodologies and mechanisms that Treelogic already has in place for
protecting their sensitive commercial data. These follow existing data and
information security standards.

Information is stored in the cluster using the Hadoop Distributed File Systems
(HDFS). Concretely, Treelogic uses the .20.20x distributions of Hadoop which
focus on security issues by utilizing the following:

  * _Mutual Authentication with Kerberos RPC (SASL/GSSAPI) on RPC connections_ **:** SASL/GSSAPI was used to implement Kerberos and mutually authenticate users, their processes, and Hadoop services on RPC connections. 

  

  * _“Pluggable” Authentication for HTTP Web Consoles:_ meaning that implementers of web applications and web consoles could implement their own authentication mechanism for HTTP connections. This could include (but was not limited to) HTTP SPNEGO authentication. 

  

  * _Enforcement of HDFS file permissions:_ Access control to files in HDFS could be enforced by the NameNode based on file permissions - Access Control Lists (ACLs) of users and groups. 

  

  * _Delegation Tokens for Subsequent Authentication checks:_ **t** hese were used between the various clients and services after their initial authentication in order to reduce the performance overhead and load on the Kerberos KDC after the initial user authentication. Specifically, _delegation tokens_ are used in communication with the NameNode for subsequent authenticated access without using the Kerberos Servers. 

  

  * _Block Access Tokens for Access Control to Data Block_ **:** when access to data blocks were needed, the NameNode would make an access control decision based on HDFS file permissions and would issue _Block access tokens (using HMAC-SHA1)_ that could be sent to the DataNode for block access requests. Because DataNodes have no concept of files or permissions, this was necessary to make the connection between the HDFS permissions and access to the blocks of data. 

  

  * _Job Tokens to Enforce Task Authorization:_ _Job tokens_ are created by the JobTracker and passed onto TaskTrackers, ensuring that Tasks could only do work on the jobs that they are assigned. Tasks could also be configured to run as the user submitting the job, making access control checks simpler. 

  

  * _From “Pluggable Authentication” to HTTP SPNEGO Authentication_ : Although the 2009 security design of Hadoop focused on pluggable authentication, the Hadoop developer community decided that it would be better to use Kerberos consistently, since Kerberos authentication was already being used for RPC connections (users, applications, and Hadoop services). Now, Hadoop web consoles are configured to use HTTP SPNEGO Authentication, an implementation of Kerberos for web consoles. This provided some much-needed consistency. 

  

  * _Network Encryption:_ Connections utilizing SASL can be configured to use a Quality of Protection (QoP) of confidential, enforcing encryption at the network level – this includes connections using Kerberos RPC and subsequent authentication using delegation tokens. Web consoles and MapReduce shuffle operations can be encrypted by configuring them to use SSL. Finally, HDFS File Transfer can also be configured for encryption 

  

Treelogic will use an Apache Kafka end point to provide test data for the
partners. The 0.9.x release used by the RADIOFOREGROUNDS Project includes a
number of features that, whether used separately or together, will increase
security in a Kafka cluster. This includes the following security measures:

  * Authentication of connections to brokers from clients (producers and consumers), other brokers and tools, using either SSL or SASL (Kerberos) 

  

  * Authentication of connections from brokers to ZooKeeper 

  

  * Encryption of data transferred between brokers and clients, between brokers or between brokers and tools using SSL (However, there is a performance degradation when SSL is enabled, and the magnitude of this degradation depends on the CPI type and the JVM implementation utilized. 

  

  * Authorisation of read/write operations by clients 

  

  * Authorisation is pluggable and integration with external authorisation services is supported (Apache Kafka, 2016) 

  

RADIOFOREGROUNDS toolset data and anonymised evaluation data will be stored by
individual partners and in the consortium’s file repository that is managed by
IAC. The sharing of this data within the consortium will create back-ups
should an incident occur, however, like the Observatory data, storing this
data within IAC’s file repository would trigger automated back-ups and enable
recovery. Finally, IAC will store the personal data associated with the
informed consent, if needed, within a separate, but equally secure, storage
space that is not accessible to project partners to protect the personal data
of those participating in the project. IAC’s existing data and information
security protocols and tools will also protect this data.

**_Role-based access control_ **

RBAC is a secure method of restricting account access to authorized users.
This method enables the account owner to add users to the account and assign
each user to specific roles. Each role has specific permissions defined by
Rackspace. RBAC allows users to perform various actions based on the scope of
their assigned role.

##  2.3. Making data interoperable

RADIOFOREGROUNDS data products will follow the **Flexible Image Transport
System** ( **FITS** ) open standard (see _https://fits.gsfc.nasa.gov/_ ). This
is the standard data format widely used by astronomers to transport, analyse,
and archive scientific data files. We will use the rules established by NASA
to create and use the FITS files.

Concerning the information included in the headers of the FITS files and the
formats, we will closely follow the standards adopted by ESA in the Planck
Legacy Archive (PLA, _http://pla.esac.esa.int/pla/_ ), both for maps and
catalogues, which can be found here:
_https://wiki.cosmos.esa.int/planckpla2015/index.php/Main_Page_ .

We note that in the particular case of maps data products, we have adopted the
**H** ierarchical **E** qual **A** rea iso **L** atitude **Pix** elization (
**HEALPIX** ) scheme ( _http://healpix.jpl.nasa.gov/_ ).

A “Hierarchical Progressive Survey”, also called **HiPS** , allows a dedicated
client/browser tool to access and display a survey progressively, based on the
principle that “the more you zoom in on a particular area the more details
show up”. This method is based on HEALPix sky tessellation.

##  2.4. Increase data re-use (through clarifying licences)

We expect that the final data products of RADIOFOREGROUNDS, and in particular,
the frequency maps and catalogues, should be used for decades. This is the
reason why the data products will follow the FITS open standards of the
astrophysical community.

**_Long-term archiving and preservation (including open access)._ **

RADIOFOREGROUNDS partners will use this section of the DMP to outline a
strategy for long-term preservation of the data products beyond the end of the
project. A consideration of these issues needs to take place alongside the
planning of the research process for generating RADIOFOREGROUNDS toolset and
evaluation data, and this section will be updated to reflect these
developments.

In any case, the current baseline is that the IAC node ( _http://www.iac.es_ )
will use its current infrastructure located at the IAC Headquarters (La
Laguna, Tenerife) for archiving and long-term preservation. The applications
will be implemented using virtual servers. In this way, the virtual server can
be allocated with CPU, memory or disk resources as needed. This virtualization
system consists of the VMware hypervisor that is installed on 6 hosts with 120
cores and 750GB of RAM in total. These hosts are connected with fibres and
alternate paths to an EMC VNX 5500 storage enclosure, where the virtual server
disks are located. In addition, hosts are connected with multiple 2GB
aggregates to the corporate network and are protected by a PaloAlto 5050
firewall from external and/or internal attacks.

The specific processes and procedures that will be put into place to guide the
long-term preservation of the data will be included in this section during the
development of the project. This includes a detailed description of how long
the data might be preserved, its exact volume and characteristics as well as
information about how the veracity of the data will be ensured. The project
will evaluate if the current proposed baseline is sufficient, or a larger
system is required. Given the dependency of this evaluation on the larger
development of the research processes and the eventual characteristics of the
final data, this section will be updated in M18 of the project and finalised
in M36.

#  3\. Allocation of resources

As discussed in the last section, the IAC will provide its infrastructure
located at the IAC Headquarters (La Laguna, Tenerife) for archiving and long-
term preservation. This infrastructure is now hosting the project web site (
_http://www.radioforegrounds.eu_ ), and will contain in the future the data
base and associated software tools.

#  4\. Data security

A detailed description on data security has been included in Section 2\.

Concerning the IAC infrastructure for archiving and long-term preservation,
from each of the virtual servers daily backups are made with the Avamar backup
system. These copies are stored in a CPD that is on a different Canary island
(La Palma). The infrastructure is designed in such a way that the service is
not affected by faults of the equipment or during maintenance operations.

#  5\. Ethical aspects

The types of data described above raise specific issues related to
intellectual property, data protection and research ethics that the project
will have to manage appropriately. Where relevant, Spanish law is considered
alongside European law, as Spain is the primary location of the research and
the location in which the data collection and processing activities are taking
place. The following discussion outlines how RADIOFOREGROUNDS will manage each
of the relevant legal requirements, and describes how the agreed data
governance processes around access storage and sharing will also assist in
managing these requirements. Consequently, this section makes consistent
reference to the material to be discussed in Section 1. This section begins by
considering issues related to research ethics through the management of
informed consent.

##  5.1 Informed consent

Addressing issues related to research ethics can largely be addressed though
the management of informed consent when the research is being conducted with
healthy, adult volunteers. However, as the participants are employees of a
partner organisation, there are some risks that they may feel pressured to
participate. This risk will also be managed though the informed consent
process.

**Informed consent** is central to ethical research practice, as adult healthy
volunteers should be empowered to manage their participation and the use of
their information during social science research. Providing transparent and
adequate information to these participants about the purpose of the research,
the data that will be collected, the research funders, the ways in which their
data will be utilised and who will benefit from the research is important to
ensure that participants understand the potential implications of their
participation (See Annex A for a draft of the informed consent form). The
creation of an **information sheet** provides this information in appropriate
detail and in language that is meaningful to the participant (See Annex A for
the draft information sheet). It also sets out information about:

  * how their data will be anonymised or pseudonymised 

  

  * how their data will be stored and shared with other researchers 

  

  * how participants may access the data they provided 

  

  * whether they can make corrections 

  

  * how they can request their data be removed, and 

  

  * where they can go if they have any questions, comments or complaints. 

  

In addition, the information sheet explains any unintended effects that may
result from the research. Combining each of these pieces of information will
enable potential participants to evaluate whether they would like to
participate in the research and whether they might experience any unintended
or adverse effects.

However, given that this research will be carried out with employees of one of
the partner organisations, ensuring voluntary participation will require a few
additional steps. First, following good practice, the information sheet will
advise participants that their participation is purely voluntary. In addition,
partners’ personnel will undertake recruitment and advise participants that
their participation is voluntary. Finally, during the research activity
itself, those conducting the research will invite participants to re-consider
their participation and to excuse themselves from the research activity. Given
that the project does not involve many sensitive topics, this should be
sufficient to ensure voluntary participation. Nevertheless, the project will
remain vigilant about this potential conflict and will carry out a rolling
risk assessment to ensure voluntary participation.

##  5.2 Personal data protection

However, seeking informed consent will raise issues around the protection of
personal data, as personal data, including names and contact information will
be needed to record informed consent. The consortium will manage this using
the following steps:

  1. Participants will immediately be given a participant number linked to their name, and this will replace their name in any stored or shared data. 

  

  2. The link between a participant’s name and number will be stored in a proprietary storage facility by the project coordinator 

  

  3. This information will not be shared with the project partners, and any enquiries about participants’ personal information will be fed through the coordinator 

  

  4. Participants with particular identifying features or experiences may be managed by mixing these with other participants’ characteristics (e.g., switching places of birth) to make each participant less identifiable. Where necessary, some identifying features may be removed from the data if it cannot be anonymised 

  

  5. Participants will be given the right to review their data and make any corrections or erasures should they have any concerns. 

  

  

The project will also avoid the collection of data that is not necessary for
the purposes of the research ( _purpose limitation and data minimisation
principles_ ). Each of these processes will assist in the anonymisation and
pseudonymisation of any personal data, and storing this data with the
coordinator will ensure that participants are adequately protected with
reference to confidentiality. In addition, the information sheet will enable
the project to meet requirements around transparency and provide a mechanism
through which participants can exercise their rights of access, correction and
erasure. The information sheet will also assist the project in meeting
requirements around data retention, as the information sheet sets out how long
the data will be stored and with whom it may be shared.

In addition to these, the project will also meet Spanish data protection
requirements. The project coordinator will register with the AEPD as a
processer of personal data. In addition, should a data breach occur, the
coordinator will inform both the AEPD and research participants about the
breach and provide advice on any consequences.

Thus, the overlapping requirements around ethical research practice and the
protection of personal data can be met simultaneously using both the
information sheet and informed consent forms for the RADIOFOREGROUNDS
research. While the amount of personal data that will be collected by the
project is relatively minimal, the project will use the data protection
principles to guide the collection of all data about human participants,
whether personal or not, to ensure that we meet ethical research requirements.
Attention to both will ensure participants receive the maximum level of
protection and consideration.

##  5.3 Intellectual property rights

As noted above, the Observatory data is subject to intellectual property
protections, and the consortium will take the following specific steps to
address this. First, as noted above, the data will be anonymised so that any
sensitive data about partners’ customers and collaboration is removed. Second,
the data governance procedures around access, storage and sharing discussed in
Chapter 6, below, will ensure that consortium members respect partners’
intellectual property rights. Finally, each of the partners have agreed to
only use the data for the purposes of the RADIOFOREGROUNDS project and the
development and testing of RADIOFOREGROUNDS algorithms and software. This has
been agreed via the RADIOFOREGROUNDS consortium agreement, a legally binding
document that governs the project and partnership arrangements.

The Consortium Agreement and this document will also guide the intellectual
property rights claimed by the consortium with respect to RADIOFOREGROUNDS
toolset data. The consortium will agree a license that adequately describes
how the data will be used and shared within the consortium and at the close of
the project. Underpinning this will be the agreement, contained within the
Consortium Agreement, that each partner owns the intellectual property,
including data, which they create. Nevertheless, the Consortium Agreement also
provides for joint or multiple ownership, and in these cases, relevant
partners will agree on the license to be used. Consideration of these
intellectual property rights will also govern the extent to which
RADIOFOREGROUNDS toolset data can be made openly accessible at the close of
the project. If this option is selected, partners will agree an open license
to manage the use of this data, and will likely select a license such as CC-BY
– a creative commons license that requires users to attribute the data to
those who originally created it. The outcome of these discussions will feed
into the RADIOFOREGROUNDS intellectual property rights and innovation
committee that will undertake the final decision regarding licensing. This
issue will be re-visited in the next iteration of this plan in M18.

#  6\. Other issues

There are no other issues.