Repository: The-Academic-Observatory/observatory-platform (language: python)

Issue:
Telescope template design
@jdddog @rhosking @tuanchien A new issue dedicated to the telescope template design.
Thanks @jdddog for the chat on Friday. Here is a design description/document. I mostly used your draft for the requirements and edited/added some points if that's alright.

There's a link to the UML diagrams here: 
https://lucid.app/lucidchart/invitations/accept/dce00967-c039-461b-b8a3-ec454ef97b01

I tried to stick to the format they describe in the tutorial here:
https://www.youtube.com/watch?v=UI6lqHOVHic&ab_channel=Lucidchart

**Design requirements**: 
*   Abstract interfaces / classes:
    *   Provide a standard set of pure abstract classes that provide interfaces to cover the functionality implemented by the telescopes that we have already implemented. \

        *   These classes will have no implementation, the purpose is to make it clear as to what the interfaces for the telescopes should look like.
    *   ~~Three abstract classes are required, including: a AbstractRelease, AbstractStreamTelescope (processes one release) and AbstractSnapshotTelescope (processes multiple releases).~~
    *   Two abstract classes are required, including an AbstractStreamTelescope and an AbstractSnapshotTelescope
    *   Define abstract methods for common telescope functionality (e.g. download, extract etc) and for building a Release and a DAG.
    *   The definition of the abstract methods should include: inputs, outputs and a textual description of how they are used and what their behaviour is.
*   The implementations of the abstract classes should:
    *   Reduce code duplication.
    *   Ensure that certain attributes are available
    *   Include the telescope type specific methods (such as loading files to BigQuery)
    *   Use the ‘release’ instance as a parameter for the python callable, but inside the task call a function from telescope_utils as much as possible, using the release properties as parameters. This way the more general telescope functions can be used across classes (or other projects/usages). 
    *   Minimise the use of XComs (if possible), because XComs add complexity.
    *   There is only one use case that needs XComs: publishing the date of release(s) that need to be processed.
    *   XComs are currently used in the Scopus and WoS telescopes to pass lists of files around. This can be replaced by having a method in the WosRelease(AbstractRelease) class that lists the files from the file system.
    *   The DAG builder function implementations should automatically build the DAG and setup the task dependencies, based on the parameters that have been set for the instance.
    *   Make it easy to test
*   Other requirements:
    *   A TelescopeRelease class which stores Release properties
    *   Provide an interface for setting the descriptions for DAGs and tasks, which show up in the Airflow UI.
    *   Add the ability to monitor failure, e.g. send a Slack notification when a task fails.

**Design Decisions:**
**Abstract & Implementation class** \
For each of the two types of telescopes there is one abstract class and one class with implementations. The abstract class and implementation class are separated to make it clear for the user which methods need to be implemented.

**Init telescope attributes in Abstract class or Implement class**
The telescope specific attributes, such as dag_id, queue, schedule_interval, dataset_id, etc.. can be defined either in the abstract class (e.g. AbstractStreamTelescope) and then inherited by the implement class (e.g. StreamTelescope), or alternatively they can be defined directly in the implement class. I’m not sure which of the two would be more clear, I’m thinking of defining them in the abstract class, since this is where the user will look to see what they need to write. 

**One TelescopeRelease class**
Instead of having a release class for each telescope type, one class for both is used. Both types use the same attributes and methods (only 2 exist for creating paths/subdir). The only difference is that for the snapshots there might not be a ‘start’ and ‘end’ date, but rather a single release date. We could just set the start date to None in this case and the end date can be used as the release date. \
The class is not abstract as there are no methods that need to be written by the user.

**Download vs Transfer** \
A remaining question is around the transfer vs. download tasks.  \
In a set-up where you would first transfer data from bucket 1 (Other) to bucket 2 (GCP) and then download the data from bucket 2, you would use the ‘transfer’ -> ‘download_transferred’ tasks/methods, both are abstract methods. \
In a set-up where you would first download data from e.g. a url and then upload the data to a GCP bucket, you would use the ‘download’ -> ‘upload_downloaded’ tasks/methods, ‘upload_downloaded’ is already implemented and not an abstract method.

So we have ‘transfer’, ‘download_transferred’ and ‘download’ as abstract methods. The user would always have to write all 3, but depending on which set-up you’re using, one or two of them could suffice with a simple ‘pass’ or ‘return True’ statement.

The tasks are then always executed after each other, ‘download’ -> ‘upload_downloaded’ -> ‘transfer’ -> ‘download_transferred’.

This works, but it would be nicer if the correct methods are defined as abstract for the right set-up, however I don’t see a way of doing this besides creating additional separate classes for transfer / download.

**Direct python callable & make task callable** \
The ‘check_dependencies’ method and method where release info date(s) are obtained are both directly passed on as python callables to the airflow PythonOperators. The other methods are called with the ‘task_callable’ method which makes sure that the release(s) is created and can be passed on as a parameter to the method. Some tasks also require keyword arguments (kwargs), so these need to be passed on to the method as well. This can be done based on the signature of the function, e.g.:  \
```python
        sig = signature(func)
        # Call task function
        if len(sig.parameters) == 2:
            result = func(release, **kwargs)
        else:
            result = func(release)
        return result
```

**Chain sequential tasks**
Instead of looping through a list of tasks to set them upstream/downstream of one another, we could use the ‘chain’ helper function to achieve the same.
```python
            # Link tasks
            for i, current_task in enumerate(tasks):
                # Link each task to the next task
                if i < (len(tasks) - 1):
                    next_task = tasks[i + 1]
                    current_task >> next_task
```
Could be replaced simply by:
```python
chain(*tasks)
```

An example dummy for what the user has to do can be found here:
https://github.com/The-Academic-Observatory/observatory-platform/blob/telescope_template/observatory_platform/templates/updated_example/orcid.py

https://docs.google.com/document/d/1Tpk9Fq-6hrwST9JD7HtHOT2UWhzD9bJfDZp-jD3k7co/edit?usp=sharing


Discussion / hints:

Fix patch:
diff --git a/docs/telescopes/index.rst b/docs/telescopes/index.rst
index 7b273ea6d..b674d5679 100644
--- a/docs/telescopes/index.rst
+++ b/docs/telescopes/index.rst
@@ -8,7 +8,8 @@ Telecopes
     geonames
     grid
     mag
-    unpaywall
+    oapen_metadata
+    scopus
     terraform
+    unpaywall
     wos
-    scopus
diff --git a/docs/telescopes/oapen_metadata.md b/docs/telescopes/oapen_metadata.md
new file mode 100644
index 000000000..53305c546
--- /dev/null
+++ b/docs/telescopes/oapen_metadata.md
@@ -0,0 +1,242 @@
+# OAPEN Metadata
+The OAPEN Metadata telescope collects data from the OAPEN Metadata feeds.
+OAPEN enables libraries and aggregators to use the metadata of all available titles in the OAPEN Library.  
+The metadata is available in different formats and this telescope harvests the data in the CSV format.  
+See the [OAPEN Metadata webpage](https://www.oapen.org/resources/15635975-metadata) for more information. 
+
+| Summary                 |        |
+|-------------------------|--------|
+| Average runtime         |   5min |
+| Average download size   |   50mb |
+| Harvest Type            |   URL  |
+| Harvest Frequency       | Weekly |
+| Runs on remote worker   |  True  |
+| Catchup missed runs     |  False |
+| Table Write Disposition | Append |
+| Update Frequency        |  Daily |
+| Credentials Required    |   No   |
+
+## Schedule
+The CSV file containing metadata is updated daily and this telescope is scheduled to harvest the metadata weekly. 
+
+## Results
+There are two tables containing data related to this telescope:
+  * The main table which contains all the up-to-date metadata. 
+  * A partitioned table, where each partition contains the metadata of one run.
+
+## Tasks
+### download
+This is where the metadata is downloaded. The CSV file containing metadata is downloaded using the CSV url that is 
+available on the OAPEN Metadata webpage mentioned above.
+
+### transform
+The downloaded CSV is transformed into a gzipped jsonl format (this format is accepted by BigQuery). 
+The flattened CSV file only is restructured into nested dictionaries and/or lists where appropriate.  
+An additional field 'classification_code' is added as well. 
+This is based on data in the field 'dc.subject.classification', but reformatted so that the codes are easy to use with other datasets.  
+Below is an example of one row before and after transformation:
+
+Before transformation
+```yaml
+{'id': 'af17b762-8a0c-4719-aa9c-7f5083a0ba52',
+'collection': '20.500.12657/6',
+'BITSTREAM Download URL': 'https://library.oapen.org/bitstream/20.500.12657/29427/1/academia-LLS.2018.10_160x240.pdf',
+'BITSTREAM ISBN': '9789401452113',
+'BITSTREAM License': 'http://creativecommons.org/licenses/by-nd/3.0/',
+'BITSTREAM Webshop URL': '',
+'dc.contributor.advisor': '',
+'dc.contributor.author': 'Missinne  , Lut||Grave , Jaap',
+'dc.contributor.editor': '',
+'dc.contributor.other': '',
+'dc.date.accessioned': '2018-09-12 23:55||2019-04-18 16:22:31||2020-04-01T12:27:51Z',
+'dc.date.available': '2020-04-01T12:27:51Z',
+'dc.date.issued': '2018',
+'dc.description': '',
+'dc.description.abstract': 'The abstract',
+'dc.description.provenance': "Created by dorien on 2018-09-12 14:55:07||Edited by lotte on 2019-04-18 16:22:31||Made available in DSpace on 2020-04-01T12:27:51Z (GMT). No. of bitstreams: 1\nacademia-LLS.2018.10_160x240.pdf: 13825548 bytes,
+checksum: a60085ebcce61de9bc7d22efc2924dce (MD5)\n  Previous issue date: 2018||Item created via OAI harvest from source: https://dataprovider.oapen.org/cgi/oapen on 2020-04-01T12:27:51Z (GMT).  Item's OAI Record identifier: oai:oapen:1000509",
+'dc.description.version': '',
+'dc.identifier': '1000509',
+'dc.identifier.isbn': '',
+'dc.identifier.issn': '',
+'dc.identifier.uri': 'http://library.oapen.org/handle/20.500.12657/29427',
+'dc.identifier.urlwebshop': '',
+'dc.language': 'Dutch; Flemish[dut]||German[ger]',
+'dc.relation.isnodouble': '',
+'dc.relation.ispartofseries': 'Lage Landen Studies',
+'dc.relation.isreplacedbydouble': '',
+'dc.rights.uri': '',
+'dc.source': '',
+'dc.subject.classification': 'bic Book Industry Communication::C Language::CF linguistics::CFP Translation & interpretation||bic Book Industry Communication::D Literature & literary studies||bic Book Industry Communication::D Literature & literary studies::DS Literature: history & criticism::DSK Literary studies: fiction, novelists & prose writers||bic Book Industry Communication::F Fiction & related items',
+'dc.subject.other': 'Dutch fiction||fiction in translation||translation||literary transfer||cultural transfer',
+'dc.title': 'Tussen twee stoelen, tussen twee vuren',
+'dc.title.alternative': 'Nederlandse literatuur op weg naar de buitenlandse lezer',
+'dc.type': 'book',
+'dcterms.abstract': '',
+'eperson.firstname': '',
+'eperson.language': '',
+'eperson.lastname': '',
+'eperson.phone': '',
+'grantor.acronym': '',
+'grantor.doi': '',
+'grantor.jurisdiction': '',
+'grantor.name': '',
+'grantor.number': '',
+'oapen.abstract.otherlanguage': 'Abstract other language',
+'oapen.autodoi': '',
+'oapen.chapternumber': '',
+'oapen.collection': '',
+'oapen.description.otherlanguage': '',
+'oapen.embargo': '',
+'oapen.grant.acronym': '',
+'oapen.grant.number': '',
+'oapen.grant.program': '',
+'oapen.grant.project': '',
+'oapen.identifier': '',
+'oapen.identifier.doi': '',
+'oapen.identifier.isbn': '',
+'oapen.imprint': '',
+'oapen.notes': '',
+'oapen.pages': '222',
+'oapen.place.publication': 'Gent',
+'oapen.redirect': '',
+'oapen.relation.funds': '',
+'oapen.relation.hasChapter': '',
+'oapen.relation.hasChapter_dc.title': '',
+'oapen.relation.isFundedBy': '',
+'oapen.relation.isFundedBy_grantor.name': 'Internationale Vereniging voor Neerlandistiek',
+'oapen.relation.isPartOfBook': '',
+'oapen.relation.isPartOfBook_dc.title': '',
+'oapen.relation.isPublishedBy': '',
+'oapen.relation.isPublishedBy_publisher.name': 'Academia Press',
+'oapen.relation.isPublisherOf': '',
+'oapen.relation.isbn': '',
+'oapen.remark.private': '',
+'oapen.remark.public': '21-7-2020 - No DOI registered in CrossRef for ISBN 9789401452113',
+'oapen.series.number': '10',
+'publisher.country': '',
+'publisher.name': '',
+'publisher.peerreviewpolicy': '',
+'publisher.website': ''}
+```
+
+After transformation
+```yaml
+{
+   "id":"af17b762-8a0c-4719-aa9c-7f5083a0ba52",
+   "collection":[
+      "20.500.12657/6"
+   ],
+   "BITSTREAM_Download_URL":[
+      "https://library.oapen.org/bitstream/20.500.12657/29427/1/academia-LLS.2018.10_160x240.pdf"
+   ],
+   "BITSTREAM_ISBN":[
+      "9789401452113"
+   ],
+   "BITSTREAM_License":[
+      "http://creativecommons.org/licenses/by-nd/3.0/"
+   ],
+   "dc":{
+      "contributor":{
+         "author":[
+            "Missinne  , Lut",
+            "Grave , Jaap"
+         ]
+      },
+      "date":{
+         "accessioned":[
+            "2018-09-12 23:55",
+            "2019-04-18 16:22:31",
+            "2020-04-01T12:27:51Z"
+         ],
+         "available":"2020-04-01T12:27:51Z",
+         "issued":[
+            "2018-01-01"
+         ]
+      },
+      "description":{
+         "abstract":[
+            "The abstract."
+         ],
+         "provenance":[
+            "Created by dorien on 2018-09-12 14:55:07",
+            "Edited by lotte on 2019-04-18 16:22:31",
+            "Made available in DSpace on 2020-04-01T12:27:51Z (GMT). No. of bitstreams: 1\nacademia-LLS.2018.10_160x240.pdf: 13825548 bytes, checksum: a60085ebcce61de9bc7d22efc2924dce (MD5)\n  Previous issue date: 2018",
+            "Item created via OAI harvest from source: https://dataprovider.oapen.org/cgi/oapen on 2020-04-01T12:27:51Z (GMT).  Item's OAI Record identifier: oai:oapen:1000509"
+         ]
+      },
+      "identifier":{
+         "value":[
+            "1000509"
+         ],
+         "uri":"http://library.oapen.org/handle/20.500.12657/29427"
+      },
+      "language":[
+         "Dutch; Flemish[dut]",
+         "German[ger]"
+      ],
+      "relation":{
+         "ispartofseries":[
+            "Lage Landen Studies"
+         ]
+      },
+      "subject":{
+         "classification":[
+            "bic Book Industry Communication::C Language::CF linguistics::CFP Translation & interpretation",
+            "bic Book Industry Communication::D Literature & literary studies",
+            "bic Book Industry Communication::D Literature & literary studies::DS Literature: history & criticism::DSK Literary studies: fiction, novelists & prose writers",
+            "bic Book Industry Communication::F Fiction & related items"
+         ],
+         "classification_code":[
+            "CFP",
+            "D",
+            "DSK",
+            "F"
+         ],
+         "other":[
+            "Dutch fiction",
+            "fiction in translation",
+            "translation",
+            "literary transfer",
+            "cultural transfer"
+         ]
+      },
+      "title":{
+         "value":[
+            "Tussen twee stoelen, tussen twee vuren"
+         ],
+         "alternative":"Nederlandse literatuur op weg naar de buitenlandse lezer"
+      },
+      "type":[
+         "book"
+      ]
+   },
+   "oapen":{
+      "abstract":{
+         "otherlanguage":"Abstract other language."
+      },
+      "pages":"222",
+      "place":{
+         "publication":"Gent"
+      },
+      "relation":{
+         "isFundedBy_grantor":{
+            "name":[
+               "Internationale Vereniging voor Neerlandistiek"
+            ]
+         },
+         "isPublishedBy_publisher":{
+            "name":"Academia Press"
+         }
+      },
+      "remark":{
+         "public":[
+            "21-7-2020 - No DOI registered in CrossRef for ISBN 9789401452113"
+         ]
+      },
+      "series":{
+         "number":"10"
+      }
+   }
+}
+```
\ No newline at end of file
diff --git a/observatory-dags/observatory/dags/dags/grid.py b/observatory-dags/observatory/dags/dags/grid.py
index 9c7c4dc42..acc72f6c2 100644
--- a/observatory-dags/observatory/dags/dags/grid.py
+++ b/observatory-dags/observatory/dags/dags/grid.py
@@ -14,100 +14,10 @@
 
 # Author: James Diprose
 
-"""
-A DAG that harvests the Global Research Identifier Database (GRID): https://grid.ac/
-
-Saved to the BigQuery table: <project_id>.digital_science.gridYYYYMMDD
-"""
-
-from datetime import datetime
-
-from airflow import DAG
-from airflow.operators.python_operator import PythonOperator
-from airflow.operators.python_operator import ShortCircuitOperator
+# The keywords airflow and DAG are required to load the DAGs from this file, see bullet 2 in the Apache Airflow FAQ:
+# https://airflow.apache.org/docs/stable/faq.html
 
 from observatory.dags.telescopes.grid import GridTelescope
 
-default_args = {
-    "owner": "airflow",
-    "start_date": datetime(2015, 9, 1)
-}
-
-with DAG(dag_id=GridTelescope.DAG_ID, schedule_interval="@weekly", default_args=default_args) as dag:
-    # Check that dependencies exist before starting
-    check = PythonOperator(
-        task_id=GridTelescope.TASK_ID_CHECK_DEPENDENCIES,
-        python_callable=GridTelescope.check_dependencies,
-        provide_context=True,
-        queue=GridTelescope.QUEUE
-    )
-
-    # List releases and skip all subsequent tasks if there is no release to process
-    list_releases = ShortCircuitOperator(
-        task_id=GridTelescope.TASK_ID_LIST,
-        python_callable=GridTelescope.list_releases,
-        provide_context=True,
-        queue=GridTelescope.QUEUE
-    )
-
-    # Download the GRID releases for a given interval
-    download = PythonOperator(
-        task_id=GridTelescope.TASK_ID_DOWNLOAD,
-        python_callable=GridTelescope.download,
-        provide_context=True,
-        queue=GridTelescope.QUEUE,
-        retries=GridTelescope.RETRIES
-    )
-
-    # Upload the GRID releases for a given interval
-    upload_downloaded = PythonOperator(
-        task_id=GridTelescope.TASK_ID_UPLOAD_DOWNLOADED,
-        python_callable=GridTelescope.upload_downloaded,
-        provide_context=True,
-        queue=GridTelescope.QUEUE,
-        retries=GridTelescope.RETRIES
-    )
-
-    # Extract the GRID releases for a given interval
-    extract = PythonOperator(
-        task_id=GridTelescope.TASK_ID_EXTRACT,
-        python_callable=GridTelescope.extract,
-        provide_context=True,
-        queue=GridTelescope.QUEUE
-    )
-
-    # Transform the GRID releases for a given interval
-    transform = PythonOperator(
-        task_id=GridTelescope.TASK_ID_TRANSFORM,
-        python_callable=GridTelescope.transform,
-        provide_context=True,
-        queue=GridTelescope.QUEUE
-    )
-
-    # Upload the transformed GRID releases for a given interval to Google Cloud Storage
-    upload_transformed = PythonOperator(
-        task_id=GridTelescope.TASK_ID_UPLOAD_TRANSFORMED,
-        python_callable=GridTelescope.upload_transformed,
-        provide_context=True,
-        queue=GridTelescope.QUEUE,
-        retries=GridTelescope.RETRIES
-    )
-
-    # Load the transformed GRID releases for a given interval to BigQuery
-    # Depends on past so that BigQuery load jobs are not all created at once
-    bq_load = PythonOperator(
-        task_id=GridTelescope.TASK_ID_BQ_LOAD,
-        python_callable=GridTelescope.bq_load,
-        provide_context=True,
-        queue=GridTelescope.QUEUE
-    )
-
-    cleanup = PythonOperator(
-        task_id=GridTelescope.TASK_ID_CLEANUP,
-        python_callable=GridTelescope.cleanup,
-        provide_context=True,
-        queue=GridTelescope.QUEUE
-    )
-
-    # Task dependencies
-    check >> list_releases >> download >> upload_downloaded >> extract >> transform >> upload_transformed >> bq_load >> cleanup
+grid = GridTelescope()
+globals()[grid.dag_id] = grid.make_dag()
diff --git a/observatory-dags/observatory/dags/dags/oapen_metadata.py b/observatory-dags/observatory/dags/dags/oapen_metadata.py
new file mode 100644
index 000000000..87f09e1be
--- /dev/null
+++ b/observatory-dags/observatory/dags/dags/oapen_metadata.py
@@ -0,0 +1,23 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs
+
+# The keywords airflow and DAG are required to load the DAGs from this file, see bullet 2 in the Apache Airflow FAQ:
+# https://airflow.apache.org/docs/stable/faq.html
+
+from observatory.dags.telescopes.oapen_metadata import OapenMetadataTelescope
+
+oapen_metadata = OapenMetadataTelescope()
+globals()[oapen_metadata.dag_id] = oapen_metadata.make_dag()
diff --git a/observatory-dags/observatory/dags/dags/scopus.py b/observatory-dags/observatory/dags/dags/scopus.py
index 857df9f48..c3b448236 100644
--- a/observatory-dags/observatory/dags/dags/scopus.py
+++ b/observatory-dags/observatory/dags/dags/scopus.py
@@ -21,9 +21,8 @@
 from airflow import DAG
 from airflow.operators.python_operator import PythonOperator
 from airflow.operators.subdag_operator import SubDagOperator
-
 from observatory.dags.telescopes.scopus import ScopusTelescope
-from observatory.platform.utils.config_utils import list_connections
+from observatory.platform.utils.airflow_utils import list_connections
 
 default_args = {'owner': 'airflow',
                 'start_date': datetime(2018, 1, 1),
diff --git a/observatory-dags/observatory/dags/dags/wos.py b/observatory-dags/observatory/dags/dags/wos.py
index 6076792e3..d7a39fd3b 100644
--- a/observatory-dags/observatory/dags/dags/wos.py
+++ b/observatory-dags/observatory/dags/dags/wos.py
@@ -21,9 +21,8 @@
 from airflow import DAG
 from airflow.operators.python_operator import PythonOperator
 from airflow.operators.subdag_operator import SubDagOperator
-
 from observatory.dags.telescopes.wos import WosTelescope
-from observatory.platform.utils.config_utils import list_connections
+from observatory.platform.utils.airflow_utils import list_connections
 
 default_args = {'owner': 'airflow',
                 'start_date': datetime(2018, 1, 1),
diff --git a/observatory-dags/observatory/dags/database/schema/oapen_metadata_2020-11-01.json b/observatory-dags/observatory/dags/database/schema/oapen_metadata_2020-11-01.json
new file mode 100644
index 000000000..f5d9fce9e
--- /dev/null
+++ b/observatory-dags/observatory/dags/database/schema/oapen_metadata_2020-11-01.json
@@ -0,0 +1,690 @@
+[
+  {
+    "description": "ID for the book.",
+    "mode": "NULLABLE",
+    "name": "id",
+    "type": "STRING"
+  },
+  {
+    "description": "Collections, see library.oapen.org. Example: https://library.oapen.org/discover?filtertype=collection&filter_relational_operator=equals&filter=Knowledge+Unlatched+%28KU%29",
+    "mode": "REPEATED",
+    "name": "collection",
+    "type": "STRING"
+  },
+  {
+    "description": "URL of the book's PDF file on OAPEN repository.",
+    "mode": "REPEATED",
+    "name": "BITSTREAM_Download_URL",
+    "type": "STRING"
+  },
+  {
+    "description": "ISBN of the book. Can have multiple isbn separated with ||.",
+    "mode": "REPEATED",
+    "name": "BITSTREAM_ISBN",
+    "type": "STRING"
+  },
+  {
+    "description": "Text describing the licence.",
+    "mode": "REPEATED",
+    "name": "BITSTREAM_License",
+    "type": "STRING"
+  },
+  {
+    "description": "URL of the book's web page on publisher site.",
+    "mode": "REPEATED",
+    "name": "BITSTREAM_Webshop_URL",
+    "type": "STRING"
+  },
+  {
+    "description": "dc",
+    "mode": "REQUIRED",
+    "name": "dc",
+    "type": "RECORD",
+    "fields":[
+      {
+        "description": "contributor",
+        "mode": "NULLABLE",
+        "name": "contributor",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Advisor name.",
+            "mode": "NULLABLE",
+            "name": "advisor",
+            "type": "STRING"
+          },
+          {
+            "description": "Author name.",
+            "mode": "REPEATED",
+            "name": "author",
+            "type": "STRING"
+          },
+          {
+            "description": "Editor name.",
+            "mode": "REPEATED",
+            "name": "editor",
+            "type": "STRING"
+          },
+          {
+            "description": "Other contributor(s) name.",
+            "mode": "REPEATED",
+            "name": "other",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "date",
+        "mode": "REQUIRED",
+        "name": "date",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Date made available in OAPEN. Starts from June 2010",
+            "mode": "REPEATED",
+            "name": "accessioned",
+            "type": "STRING"
+          },
+          {
+            "description": "Date of upload to the new DSPACE platform. Starts from April of 2020.",
+            "mode": "REQUIRED",
+            "name": "available",
+            "type": "TIMESTAMP"
+          },
+          {
+            "description": "Year of publication of the book.",
+            "mode": "REPEATED",
+            "name": "issued",
+            "type": "DATE"
+          }
+        ]
+      },
+      {
+        "description": "description",
+        "mode": "NULLABLE",
+        "name": "description",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "description",
+            "mode": "NULLABLE",
+            "name": "value",
+            "type": "STRING"
+          },
+          {
+            "description": "Abstract of the book.",
+            "mode": "REPEATED",
+            "name": "abstract",
+            "type": "STRING"
+          },
+          {
+            "description": "Upload information, on who created the record, when it is uploaded to DSPACE.",
+            "mode": "REPEATED",
+            "name": "provenance",
+            "type": "STRING"
+          },
+          {
+            "description": "?",
+            "mode": "NULLABLE",
+            "name": "version",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "identifier",
+        "mode": "NULLABLE",
+        "name": "identifier",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "'Old' OAPEN ID or ONIX import ID. It can be a number or can be in the form of a string (\"ONIX.20200714.9789811555732.9\")",
+            "mode": "REPEATED",
+            "name": "value",
+            "type": "STRING"
+          },
+          {
+            "description": "Additional ISBN(s) number of the book.",
+            "mode": "REPEATED",
+            "name": "isbn",
+            "type": "STRING"
+          },
+          {
+            "description": "ISSN number.",
+            "mode": "NULLABLE",
+            "name": "issn",
+            "type": "STRING"
+          },
+          {
+            "description": "Web page of the book on OAPEN (HANDLE = unique ID).",
+            "mode": "NULLABLE",
+            "name": "uri",
+            "type": "STRING"
+          },
+          {
+            "description": "[Blank]",
+            "mode": "NULLABLE",
+            "name": "urlwebshop",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "Language of the book.",
+        "mode": "REPEATED",
+        "name": "language",
+        "type": "STRING"
+      },
+      {
+        "description": "relation",
+        "mode": "NULLABLE",
+        "name": "relation",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "?",
+            "mode": "REPEATED",
+            "name": "isnodouble",
+            "type": "STRING"
+          },
+          {
+            "description": "Name of the book series.",
+            "mode": "REPEATED",
+            "name": "ispartofseries",
+            "type": "STRING"
+          },
+          {
+            "description": "?",
+            "mode": "REPEATED",
+            "name": "isreplacedbydouble",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "rights",
+        "mode": "NULLABLE",
+        "name": "rights",
+        "type": "RECORD",
+        "fields": [
+            {
+              "description": "[Blank]",
+              "mode": "NULLABLE",
+              "name": "uri",
+              "type": "STRING"
+            }
+        ]
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "source",
+        "type": "STRING"
+      },
+      {
+        "description": "subject",
+        "mode": "NULLABLE",
+        "name": "subject",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "BIC subject category.",
+            "mode": "REPEATED",
+            "name": "classification",
+            "type": "STRING"
+          },
+          {
+            "description": "BIC subject category code.",
+            "mode": "REPEATED",
+            "name": "classification_code",
+            "type": "STRING"
+          },
+          {
+            "description": "Keywords.",
+            "mode": "REPEATED",
+            "name": "other",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "title",
+        "mode": "NULLABLE",
+        "name": "title",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Title of the book.",
+            "mode": "REPEATED",
+            "name": "value",
+            "type": "STRING"
+          },
+          {
+            "description": "Alternative.",
+            "mode": "NULLABLE",
+            "name": "alternative",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "Can be 'book',  'chapter', 'book||book'.",
+        "mode": "REPEATED",
+        "name": "type",
+        "type": "STRING"
+      }
+    ]
+  },
+  {
+    "description": "[Blank]",
+    "mode": "NULLABLE",
+    "name": "dcterms",
+    "type": "RECORD",
+    "fields": [
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "abstract",
+        "type": "STRING"
+      }
+    ]
+  },
+  {
+    "description": "[Blank]",
+    "mode": "NULLABLE",
+    "name": "eperson",
+    "type": "RECORD",
+    "fields": [
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "firstname",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "language",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "lastname",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "phone",
+        "type": "STRING"
+      }
+    ]
+  },
+  {
+    "description": "[Blank]",
+    "mode": "NULLABLE",
+    "name": "grantor",
+    "type": "RECORD",
+    "fields": [
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "acronym",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "doi",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "jurisdiction",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "name",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "REPEATED",
+        "name": "number",
+        "type": "STRING"
+      }
+    ]
+  },
+  {
+    "description": "oapen",
+    "mode": "NULLABLE",
+    "name": "oapen",
+    "type": "RECORD",
+    "fields": [
+      {
+        "description": "abstract",
+        "mode": "NULLABLE",
+        "name": "abstract",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Abstract in other language.",
+            "mode": "NULLABLE",
+            "name": "otherlanguage",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "",
+        "mode": "NULLABLE",
+        "name": "autodoi",
+        "type": "STRING"
+      },
+      {
+        "description": "Number of chapter.",
+        "mode": "NULLABLE",
+        "name": "chapternumber",
+        "type": "INTEGER"
+      },
+      {
+        "description": "OAPEN Collection.",
+        "mode": "REPEATED",
+        "name": "collection",
+        "type": "STRING"
+      },
+      {
+        "description": "Description in another language.",
+        "mode": "NULLABLE",
+        "name": "description",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Description in another language.",
+            "mode": "NULLABLE",
+            "name": "otherlanguage",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "embargo",
+        "type": "STRING"
+      },
+      {
+        "description": "Grant.",
+        "mode": "NULLABLE",
+        "name": "grant",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Acronym of the grant.",
+            "mode": "REPEATED",
+            "name": "acronym",
+            "type": "STRING"
+          },
+          {
+            "description": "Grant number, can include strings.",
+            "mode": "REPEATED",
+            "name": "number",
+            "type": "STRING"
+          },
+          {
+            "description": "Grant program name.",
+            "mode": "REPEATED",
+            "name": "program",
+            "type": "STRING"
+          },
+          {
+            "description": "Grant project name.",
+            "mode": "REPEATED",
+            "name": "project",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "Identifier",
+        "mode": "NULLABLE",
+        "name": "identifier",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "Identifier value.",
+            "mode": "NULLABLE",
+            "name": "value",
+            "type": "STRING"
+          },
+          {
+            "description": "DOI of the book. Can also include full URL.",
+            "mode": "NULLABLE",
+            "name": "doi",
+            "type": "STRING"
+          },
+          {
+            "description": "ISBN of the book.",
+            "mode": "NULLABLE",
+            "name": "isbn",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "Imprint of the book.",
+        "mode": "REPEATED",
+        "name": "imprint",
+        "type": "STRING"
+      },
+      {
+        "description": "",
+        "mode": "REPEATED",
+        "name": "notes",
+        "type": "STRING"
+      },
+      {
+        "description": "Number of pages. Can include arabic page notation, as well as strings.",
+        "mode": "NULLABLE",
+        "name": "pages",
+        "type": "STRING"
+      },
+      {
+        "description": "Place",
+        "mode": "NULLABLE",
+        "name": "place",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "City of publisher.",
+            "mode": "NULLABLE",
+            "name": "publication",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "",
+        "mode": "REPEATED",
+        "name": "redirect",
+        "type": "STRING"
+      },
+      {
+        "description": "Relation",
+        "mode": "NULLABLE",
+        "name": "relation",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "ID of book - part of funder record",
+            "mode": "NULLABLE",
+            "name": "funds",
+            "type": "STRING"
+          },
+          {
+            "description": "[ID of chapter - part of book record",
+            "mode": "NULLABLE",
+            "name": "hasChapter",
+            "type": "STRING"
+          },
+          {
+            "description": "",
+            "mode": "NULLABLE",
+            "name": "hasChapter_dc",
+            "type": "RECORD",
+            "fields": [
+              {
+                "description": "Chapter name of chapter records.",
+                "mode": "REPEATED",
+                "name": "title",
+                "type": "STRING"
+              }
+            ]
+          },
+          {
+            "description": "ID of funder - part of book or chapter record",
+            "mode": "NULLABLE",
+            "name": "isFundedBy",
+            "type": "STRING"
+          },
+          {
+            "description": "",
+            "mode": "NULLABLE",
+            "name": "isFundedBy_grantor",
+            "type": "RECORD",
+            "fields": [
+              {
+                "description": "Funder name.",
+                "mode": "REPEATED",
+                "name": "name",
+                "type": "STRING"
+              }
+            ]
+          },
+          {
+            "description": "ID of book - part of chapter record",
+            "mode": "NULLABLE",
+            "name": "isPartOfBook",
+            "type": "STRING"
+          },
+          {
+            "description": "",
+            "mode": "NULLABLE",
+            "name": "isPartOfBook_dc",
+            "type": "RECORD",
+            "fields": [
+              {
+                "description": "Chapter's book title.",
+                "mode": "NULLABLE",
+                "name": "title",
+                "type": "STRING"
+              }
+            ]
+          },
+          {
+            "description": "Publisher name",
+            "mode": "NULLABLE",
+            "name": "isPublishedBy",
+            "type": "STRING"
+          },
+          {
+            "description": "",
+            "mode": "NULLABLE",
+            "name": "isPublishedBy_publisher",
+            "type": "RECORD",
+            "fields": [
+              {
+                "description": "",
+                "mode": "NULLABLE",
+                "name": "name",
+                "type": "STRING"
+              }
+            ]
+          },
+          {
+            "description": "ID of book - part of publisher record",
+            "mode": "REPEATED",
+            "name": "isPublisherOf",
+            "type": "STRING"
+          },
+          {
+            "description": "",
+            "mode": "REPEATED",
+            "name": "isbn",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "Remark",
+        "mode": "NULLABLE",
+        "name": "remark",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "",
+            "mode": "NULLABLE",
+            "name": "private",
+            "type": "STRING"
+          },
+          {
+            "description": "",
+            "mode": "REPEATED",
+            "name": "public",
+            "type": "STRING"
+          }
+        ]
+      },
+      {
+        "description": "",
+        "mode": "NULLABLE",
+        "name": "series",
+        "type": "RECORD",
+        "fields": [
+          {
+            "description": "",
+            "mode": "NULLABLE",
+            "name": "number",
+            "type": "STRING"
+          }
+        ]
+      }
+    ]
+  },
+  {
+    "description": "Publisher",
+    "mode": "NULLABLE",
+    "name": "publisher",
+    "type": "RECORD",
+    "fields": [
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "country",
+        "type": "STRING"
+      },
+      {
+        "description": "[Blank]",
+        "mode": "NULLABLE",
+        "name": "name",
+        "type": "STRING"
+      },
+      {
+        "description": "Text describing policy",
+        "mode": "NULLABLE",
+        "name": "peerreviewpolicy",
+        "type": "STRING"
+      },
+      {
+        "description": "",
+        "mode": "NULLABLE",
+        "name": "website",
+        "type": "STRING"
+      }
+    ]
+  }
+]
diff --git a/observatory-dags/observatory/dags/database/sql/merge_delete_matched.sql.jinja2 b/observatory-dags/observatory/dags/database/sql/merge_delete_matched.sql.jinja2
new file mode 100644
index 000000000..b1ea3fe1d
--- /dev/null
+++ b/observatory-dags/observatory/dags/database/sql/merge_delete_matched.sql.jinja2
@@ -0,0 +1,24 @@
+
+{# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Author: Aniek Roelofs#}
+
+MERGE
+  {{ dataset }}.{{ main_table }} M
+USING
+  (SELECT {{ merge_condition_field }} AS id, {{ updated_date_field }} AS date FROM {{ dataset }}.{{ partitioned_table }} WHERE _PARTITIONDATE >= '{{ start_date }}' AND _PARTITIONDATE < '{{ end_date }}') P
+ON
+  M.{{ merge_condition_field }} = P.id
+WHEN MATCHED AND M.{{ updated_date_field }} <= P.date OR M.{{ updated_date_field }} is null THEN
+  DELETE
\ No newline at end of file
diff --git a/observatory-dags/observatory/dags/telescopes/crossref_metadata.py b/observatory-dags/observatory/dags/telescopes/crossref_metadata.py
index 0e13a5d26..40d5dd95f 100644
--- a/observatory-dags/observatory/dags/telescopes/crossref_metadata.py
+++ b/observatory-dags/observatory/dags/telescopes/crossref_metadata.py
@@ -37,9 +37,13 @@
 from natsort import natsorted
 
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import (AirflowVars, AirflowConns, SubFolder, find_schema, telescope_path,
-                                                     check_variables, check_connections, test_data_path)
+from observatory.platform.utils.airflow_utils import AirflowConns, \
+    AirflowVariable as Variable, \
+    AirflowVars, \
+    check_connections, \
+    check_variables
+from observatory.platform.utils.config_utils import (find_schema)
+from observatory.platform.utils.template_utils import SubFolder, telescope_path, test_data_path
 from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
                                                  bigquery_table_exists,
                                                  create_bigquery_dataset,
diff --git a/observatory-dags/observatory/dags/telescopes/fundref.py b/observatory-dags/observatory/dags/telescopes/fundref.py
index 3f66b06d0..7df927c49 100644
--- a/observatory-dags/observatory/dags/telescopes/fundref.py
+++ b/observatory-dags/observatory/dags/telescopes/fundref.py
@@ -34,19 +34,18 @@
 from airflow.exceptions import AirflowException
 from airflow.models.taskinstance import TaskInstance
 from google.cloud.bigquery import SourceFormat
-from pendulum import Pendulum
-
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import (AirflowVars, SubFolder, find_schema, telescope_path,
-                                                     check_variables, test_data_path)
+from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, AirflowVars, check_variables
+from observatory.platform.utils.config_utils import (find_schema)
 from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
                                                  bigquery_table_exists,
                                                  create_bigquery_dataset,
                                                  load_bigquery_table,
                                                  upload_file_to_cloud_storage)
 from observatory.platform.utils.proc_utils import wait_for_process
+from observatory.platform.utils.template_utils import SubFolder, blob_name, telescope_path, test_data_path
 from observatory.platform.utils.url_utils import retry_session
+from pendulum import Pendulum
 
 
 def list_releases(start_date: Pendulum, end_date: Pendulum) -> List[FundrefRelease]:
@@ -536,9 +535,8 @@ def check_dependencies(**kwargs):
         :return: None.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         if not vars_valid:
             raise AirflowException('Required variables are missing')
 
@@ -588,7 +586,8 @@ def list_releases(**kwargs):
 
     @staticmethod
     def download(**kwargs):
-        """ Download release to file. If develop environment, copy debug file from this repository to the right location.
+        """ Download release to file. If develop environment, copy debug file from this repository to the right
+        location.
         Else download from url.
 
         :param kwargs: the context passed from the PythonOperator. See
@@ -630,7 +629,7 @@ def upload_downloaded(**kwargs):
 
         # Upload each release
         for release in releases_list:
-            upload_file_to_cloud_storage(bucket_name, release.get_blob_name(SubFolder.downloaded),
+            upload_file_to_cloud_storage(bucket_name, blob_name(release.filepath_download),
                                          file_path=release.filepath_download)
 
     @staticmethod
@@ -690,7 +689,7 @@ def upload_transformed(**kwargs):
 
         # Upload each release
         for release in releases_list:
-            upload_file_to_cloud_storage(bucket_name, release.get_blob_name(SubFolder.transformed),
+            upload_file_to_cloud_storage(bucket_name, blob_name(release.filepath_transform),
                                          file_path=release.filepath_transform)
 
     @staticmethod
@@ -729,7 +728,7 @@ def bq_load(**kwargs):
                 exit(os.EX_CONFIG)
 
             # Load BigQuery table
-            uri = f"gs://{bucket_name}/{release.get_blob_name(SubFolder.transformed)}"
+            uri = f"gs://{bucket_name}/{blob_name(release.filepath_transform)}"
             logging.info(f"URI: {uri}")
             load_bigquery_table(uri, dataset_id, data_location, table_id, schema_file_path,
                                 SourceFormat.NEWLINE_DELIMITED_JSON)
diff --git a/observatory-dags/observatory/dags/telescopes/geonames.py b/observatory-dags/observatory/dags/telescopes/geonames.py
index 4b8ba2c1b..c6dc2c3f6 100644
--- a/observatory-dags/observatory/dags/telescopes/geonames.py
+++ b/observatory-dags/observatory/dags/telescopes/geonames.py
@@ -28,17 +28,16 @@
 from airflow.exceptions import AirflowException
 from airflow.models.taskinstance import TaskInstance
 from google.cloud.bigquery import SourceFormat
-from pendulum import Pendulum
-
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import (AirflowVars, SubFolder, find_schema, telescope_path,
-                                                     check_variables, test_data_path)
+from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, AirflowVars, check_variables
+from observatory.platform.utils.config_utils import (find_schema)
 from observatory.platform.utils.data_utils import get_file
 from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
                                                  create_bigquery_dataset,
                                                  load_bigquery_table,
                                                  upload_file_to_cloud_storage)
+from observatory.platform.utils.template_utils import SubFolder, telescope_path, test_data_path
+from pendulum import Pendulum
 
 
 def pull_release(ti: TaskInstance):
@@ -188,9 +187,8 @@ def check_dependencies(**kwargs):
         for a list of the keyword arguments that are passed to this argument.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         if not vars_valid:
             raise AirflowException('Required variables are missing')
 
diff --git a/observatory-dags/observatory/dags/telescopes/grid.py b/observatory-dags/observatory/dags/telescopes/grid.py
index a24bb02b2..5e49acf1b 100644
--- a/observatory-dags/observatory/dags/telescopes/grid.py
+++ b/observatory-dags/observatory/dags/telescopes/grid.py
@@ -12,205 +12,233 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Author: James Diprose
+# Author: James Diprose, Aniek Roelofs 
 
+from __future__ import annotations
 
-import glob
+import datetime
 import gzip
 import io
 import json
 import logging
 import os
-import pathlib
-import shutil
+import re
+from datetime import datetime
 from shutil import copyfile
-from typing import Dict, List, Tuple
+from typing import List
 from zipfile import BadZipFile, ZipFile
 
-import jsonlines
 import pendulum
 from airflow.exceptions import AirflowException
 from airflow.models.taskinstance import TaskInstance
-from google.cloud.bigquery import SourceFormat
-from pendulum import Pendulum
-
-from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import AirflowVars, SubFolder, find_schema, telescope_path, check_variables
+from observatory.platform.telescopes.snapshot_telescope import SnapshotRelease, SnapshotTelescope
+from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, AirflowVars
 from observatory.platform.utils.data_utils import get_file
-from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
-                                                 create_bigquery_dataset,
-                                                 load_bigquery_table,
-                                                 upload_file_to_cloud_storage)
+from observatory.platform.utils.template_utils import upload_files_from_list
+from observatory.platform.utils.telescope_utils import list_to_jsonl_gz
 from observatory.platform.utils.url_utils import retry_session
+from pendulum import Pendulum
 
-GRID_DATASET_URL = "https://api.figshare.com/v2/collections/3812929/articles?page_size=1000"
-GRID_FILE_URL = "https://api.figshare.com/v2/articles/{article_id}/files"
 
+class GridRelease(SnapshotRelease):
+    def __init__(self, dag_id: str, article_ids: List[str], release_date: Pendulum):
+        """ Construct a GridRelease.
+        :param article_ids: the titles of the Figshare articles.
+        :param release_date: the release date.
+        """
+        self.dag_id = dag_id
+        self.article_ids = article_ids
+        self.release_date = release_date
+        self.project_id = Variable.get(AirflowVars.PROJECT_ID)
+        self.data_location = Variable.get(AirflowVars.DATA_LOCATION)
+
+        download_files_regex = self.dag_id + "\.[a-zA-Z]+"
+        extract_files_regex = "grid.json"
+        transform_files_regex = f"{self.dag_id}.jsonl.gz"
+
+        super().__init__(self.dag_id, release_date, download_files_regex, extract_files_regex, transform_files_regex)
+
+    @property
+    def transform_path(self) -> str:
+        return os.path.join(self.transform_folder, f'{self.dag_id}.jsonl.gz')
+
+    def download(self, timeout: float = 30.) -> List[str]:
+        """ Downloads an individual GRID release from Figshare.
+        :param timeout: the timeout in seconds when calling the Figshare API.
+        :return: the paths on the system of the downloaded files.
+        """
 
-def list_grid_releases(timeout: float = 30.) -> List[Dict]:
-    """ List all GRID releases available on Figshare.
+        downloads = []
+        for article_id in self.article_ids:
+            response = retry_session().get(GridTelescope.GRID_FILE_URL.format(article_id=article_id), timeout=timeout,
+                                           headers={
+                                               'Accept-encoding': 'gzip'
+                                           })
+            article_files = json.loads(response.text)
+
+            for i, article_file in enumerate(article_files):
+                real_file_name = article_file['name']
+                supplied_md5 = article_file['supplied_md5']
+                download_url = article_file['download_url']
+                file_type = os.path.splitext(real_file_name)[1]
+
+                if file_type == '.csv':
+                    continue
+
+                # Download
+                logging.info(f"Downloading file: {real_file_name}, md5: {supplied_md5}, url: {download_url}")
+                file_path, updated = get_file(f'{self.dag_id}{file_type}', download_url, md5_hash=supplied_md5,
+                                              cache_subdir='', cache_dir=self.download_folder)
+                downloads.append(file_path)
+
+        return downloads
+
+    def extract(self) -> None:
+        """ Extract a single GRID release to a given extraction path. The release will be extracted into the following
+        directory structure: extraction_path/file_name (without extension).
+        If the release is a .zip file, it will be extracted, otherwise it will be copied to a directory within the
+        extraction path.
+        :return: None.
+        """
 
-    :param timeout: the number of seconds to wait until timing out.
-    :return: the list of GRID releases.
-    """
+        logging.info(f'Download files {self.download_files}')
+        # Extract files
+        for file_path in self.download_files:
+            # Extract zip files
+            if file_path.endswith(".zip"):
+                unzip_folder_path = self.extract_folder
+                logging.info(f"Extracting file: {file_path}")
+                try:
+                    with ZipFile(file_path) as zip_file:
+                        zip_file.extractall(unzip_folder_path)
+                except BadZipFile:
+                    logging.error("Not a zip file")
+                logging.info(f"File extracted to: {unzip_folder_path}")
+            else:
+                # File is already uncompressed (.json or .csv), so make a directory and copy it into it
+                output_file_path = os.path.join(self.extract_folder, os.path.basename(file_path))
+                copyfile(file_path, output_file_path)
+                logging.info(f"File saved to: {output_file_path}")
+
+    def transform(self) -> str:
+        """ Transform an extracted GRID release .json file into json lines format and gzip the result.
+        :return: the GRID version, the file name and the file path.
+        """
 
-    response = retry_session().get(GRID_DATASET_URL, timeout=timeout, headers={
-        'Accept-encoding': 'gzip'
-    })
-    return json.loads(response.text)
+        extract_files = self.extract_files
+
+        # Only process one JSON file
+        if len(extract_files) == 1:
+            release_json_file = extract_files[0]
+            logging.info(f'Transforming file: {release_json_file}')
+
+        else:
+            raise AirflowException(f"{len(extract_files)} extracted grid.json file found: {extract_files}")
+
+        with open(release_json_file) as json_file:
+            # Load GRID release JSON file
+            data = json.load(json_file)
+            version = data['version']
+            institutes = data['institutes']
 
+            # Transform GRID release into JSON Lines format saving in memory buffer
+            # Save in memory buffer to gzipped file
+            list_to_jsonl_gz(self.transform_path, institutes)
 
-def download_grid_release(download_path: str, article_id: int, title: str, timeout: float = 30.) -> List[str]:
-    """ Downloads an individual GRID release from Figshare.
+        return version
 
-    :param download_path: the directory where the downloaded GRID release should be saved.
-    :param article_id: the Figshare article id of the GRID release. Called 'id' in the returned dictionary.
-    :param title: the title of the Figshare article.
-    :param timeout: the timeout in seconds when calling the Figshare API.
-    :return: the paths on the system of the downloaded files.
+
+def list_grid_records(start_date: Pendulum, end_date: Pendulum, grid_dataset_url: str, timeout: float = 30.) -> List[
+    dict]:
+    """ List all GRID records available on Figshare between two dates.
+    :param timeout: the number of seconds to wait until timing out.
+    :return: the list of GRID releases with required variables stored as a dictionary.
     """
 
-    response = retry_session().get(GRID_FILE_URL.format(article_id=article_id), timeout=timeout, headers={
+    response = retry_session().get(grid_dataset_url, timeout=timeout, headers={
         'Accept-encoding': 'gzip'
     })
-    article_files = json.loads(response.text)
-
-    downloads = []
-    for i, article_file in enumerate(article_files):
-        real_file_name = article_file['name']
-        supplied_md5 = article_file['supplied_md5']
-        download_url = article_file['download_url']
-        file_type = os.path.splitext(real_file_name)[1]
-
-        # Download
-        logging.info(f"Downloading file: {real_file_name}, md5: {supplied_md5}, url: {download_url}")
-        # The title is used for the filename because they seem to be labelled more reliably than the files.
-        # Replace spaces with underscores and hyphens with underscores for consistency.
-        # The index is added onto the end in case there are multiple files in one article
-        dir_name = f'{title.replace(" ", "_").replace("-", "_")}-{i}'
-        file_name = f"{dir_name}{file_type}"
-        file_path, updated = get_file(file_name, download_url, md5_hash=supplied_md5, cache_subdir='',
-                                      cache_dir=download_path)
-        downloads.append(file_path)
-
-    return downloads
-
-
-def extract_grid_release(file_path: str, extraction_path: str) -> str:
-    """ Extract a single GRID release to a given extraction path. The release will be extracted into the following
-    directory structure: extraction_path/file_name (without extension).
-
-    If the release is a .zip file, it will be extracted, otherwise it will be copied to a directory within the
-    extraction path.
-
-    :param file_path: the path to the downloaded GRID release.
-    :param extraction_path: the folder to extract the files into.
-    :return: the path to folder that the release was extracted or copied into.
-    """
+    response_json = json.loads(response.text)
 
-    # Make file and directory names
-    file_name = os.path.basename(file_path)
-    dir_name = os.path.basename(file_path).split(".")[0]
-
-    # Extract zip files, leave other files such as .json and .csv
-    unzip_folder_path = os.path.join(extraction_path, dir_name)
-    if file_path.endswith(".zip"):
-        logging.info(f"Extracting file: {file_path}")
-        try:
-            with ZipFile(file_path) as zip_file:
-                zip_file.extractall(unzip_folder_path)
-        except BadZipFile:
-            logging.error("Not a zip file")
-        extracted_folder_path = unzip_folder_path
-        logging.info(f"File extracted to: {extracted_folder_path}")
-    else:
-        # File is already uncompressed, so make a directory and copy it into it
-        extracted_folder_path = os.path.join(extraction_path, dir_name)
-        extracted_file_path = os.path.join(extracted_folder_path, file_name)
-        if not os.path.exists(extracted_folder_path):
-            os.makedirs(extracted_folder_path, exist_ok=True)
-        copyfile(file_path, extracted_file_path)
-        logging.info(f"File saved to: {extracted_file_path}")
-
-    return extracted_folder_path
-
-
-def transform_grid_release(release_json_path: str, transformed_path: str) -> Tuple[str, str, str]:
-    """ Transform an extracted GRID release .json file into json lines format and gzip the result.
-
-    :param release_json_path: the path to GRID release .json file.
-    :param transformed_path: the path to save the results.
-    :return: the GRID version, the file name and the file path.
-    """
+    records: List[dict] = []
+    release_articles = {}
+    for item in response_json:
+        published_date: Pendulum = pendulum.parse(item['published_date'])
 
-    with open(release_json_path) as json_file:
-        # Load GRID release JSON file
-        data = json.load(json_file)
-        version = data['version']
-        institutes = data['institutes']
-
-        # Transform GRID release into JSON Lines format saving in memory buffer
-        # Save in memory buffer to gzipped file
-        with io.BytesIO() as bytes_io:
-            with gzip.GzipFile(fileobj=bytes_io, mode='w') as gzip_file:
-                with jsonlines.Writer(gzip_file) as writer:
-                    writer.write_all(institutes)
-
-            file_name = f"grid_{version.replace('release_', '')}.jsonl.gz"
-            file_path = os.path.join(transformed_path, file_name)
-            with open(file_path, 'wb') as jsonl_gzip_file:
-                jsonl_gzip_file.write(bytes_io.getvalue())
-
-    return version, file_name, file_path
-
-
-class GridTelescope:
-    """ A container for holding the constants and static functions for the GRID telescope. """
-
-    DAG_ID = 'grid'
-    DATASET_ID = 'digital_science'
-    DESCRIPTION = 'The Global Research Identifier Database (GRID): https://grid.ac/'
-    RELEASES_TOPIC_NAME = 'releases'
-    QUEUE = 'default'
-    RETRIES = 3
-
-    TASK_ID_CHECK_DEPENDENCIES = 'check_dependencies'
-    TASK_ID_LIST = 'list_releases'
-    TASK_ID_DOWNLOAD = 'download'
-    TASK_ID_UPLOAD_DOWNLOADED = 'upload_downloaded'
-    TASK_ID_EXTRACT = 'extract'
-    TASK_ID_TRANSFORM = 'transform'
-    TASK_ID_UPLOAD_TRANSFORMED = 'upload_transformed'
-    TASK_ID_BQ_LOAD = 'bq_load'
-    TASK_ID_CLEANUP = 'cleanup'
-    TASK_ID_STOP = 'stop_dag'
-
-    @staticmethod
-    def check_dependencies(**kwargs):
-        """ Check that all variables exist that are required to run the DAG.
+        if start_date <= published_date < end_date:
+            article_id = item['id']
+            title = item['title']
 
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
-        :return: None.
+            # Parse date:
+            # The publish date is not used as the release date because the dataset is often
+            # published after the release date
+            date_matches = re.search("([0-9]{4}\-[0-9]{2}\-[0-9]{2})", title)
+            if date_matches is None:
+                raise ValueError(f'No release date found in GRID title: {title}')
+            release_date = pendulum.parse(date_matches[0])
+
+            try:
+                release_articles[release_date].append(article_id)
+            except KeyError:
+                release_articles[release_date] = [article_id]
+
+    for release_date in release_articles:
+        article_ids = release_articles[release_date]
+        records.append({
+            'article_ids': article_ids,
+            'release_date': release_date
+        })
+    return records
+
+
+class GridTelescope(SnapshotTelescope):
+    """ The Global Research Identifier Database (GRID): https://grid.ac/ """
+    GRID_FILE_URL = "https://api.figshare.com/v2/articles/{article_id}/files"
+    GRID_DATASET_URL = "https://api.figshare.com/v2/collections/3812929/articles?page_size=1000"
+
+    def __init__(self, dag_id: str = 'grid', start_date: datetime = datetime(2015, 9, 1),
+                 schedule_interval: str = '@weekly', dataset_id: str = 'digital_science', catchup: bool = True,
+                 airflow_vars=None):
+        """ Construct a GridTelescope instance.
+        :param dag_id: the id of the DAG.
+        :param start_date: the start date of the DAG.
+        :param schedule_interval: the schedule interval of the DAG.
+        :param catchup: whether to catchup the DAG or not.
+        :param airflow_vars: list of airflow variable keys, for each variable it is checked if it exists in airflow
         """
+        if airflow_vars is None:
+            airflow_vars = [AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                            AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET]
+        super().__init__(dag_id, start_date, schedule_interval, dataset_id, catchup=catchup, airflow_vars=airflow_vars)
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
-        if not vars_valid:
-            raise AirflowException('Required variables are missing')
+        self.add_setup_task_chain([self.check_dependencies, self.list_releases])
+        self.add_task_chain(
+            [self.download, self.upload_downloaded, self.extract, self.transform, self.upload_transformed, self.bq_load,
+             self.cleanup])
 
-    @staticmethod
-    def list_releases(**kwargs):
-        """ Task to lists all GRID releases for a given month.
+    def make_release(self, **kwargs) -> List[GridRelease]:
+        """ Make release instances. The release is passed as an argument to the function (TelescopeFunction) that is
+        called in 'task_callable'.
+
+        :param kwargs: the context passed from the PythonOperator. See
+        https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are
+        passed to this argument.
+        :return: A list of grid release instances
+        """
+        ti: TaskInstance = kwargs['ti']
+        records = ti.xcom_pull(key=GridTelescope.RELEASE_INFO, task_ids=self.list_releases.__name__,
+                               include_prior_dates=False)
+        releases = []
+        for record in records:
+            article_ids = record['article_ids']
+            release_date = record['release_date']
 
-        Pushes the following xcom:
-            article_id (str): the Figshare article id.
-            title (str): the Figshare article title.
-            published_date (Pendulum): the published date of the Figshare article.
+            releases.append(GridRelease(self.dag_id, article_ids, release_date))
+        return releases
 
+    def list_releases(self, **kwargs):
+        """ Lists all GRID releases for a given month and publishes their article_id's and
+        release_date's as an XCom.
         :param kwargs: the context passed from the BranchPythonOperator. See
         https://airflow.apache.org/docs/stable/macros-ref.html
         for a list of the keyword arguments that are passed to this argument.
@@ -219,301 +247,48 @@ def list_releases(**kwargs):
 
         execution_date = kwargs['execution_date']
         next_execution_date = kwargs['next_execution_date']
-        grid_releases = list_grid_releases()
-
-        # Select the GRID releases that were published on or after the execution_date and before the next_execution_date
-        msgs_out = []
-        for release in grid_releases:
-            published_date: Pendulum = pendulum.parse(release['published_date'])
+        records = list_grid_records(execution_date, next_execution_date, GridTelescope.GRID_DATASET_URL)
 
-            if execution_date <= published_date < next_execution_date:
-                msg = dict()
-                msg['article_id'] = release['id']
-                msg['title'] = release['title']
-                msg['published_date'] = published_date
-                msgs_out.append(msg)
-
-        continue_dag = len(msgs_out)
+        continue_dag = len(records)
         if continue_dag:
             # Push messages
             ti: TaskInstance = kwargs['ti']
-            ti.xcom_push(GridTelescope.RELEASES_TOPIC_NAME, msgs_out, execution_date)
+            ti.xcom_push(GridTelescope.RELEASE_INFO, records, execution_date)
+
         return continue_dag
 
-    @staticmethod
-    def download(**kwargs):
+    def download(self, releases: List[GridRelease], **kwargs):
         """ Task to download the GRID releases for a given month.
-
-        Pushes the following xcom:
-            download_path (str): the path to the downloaded GRID release.
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
+        :param releases: a list of GRID releases.
         :return: None.
         """
+        # Download each release
+        for release in releases:
+            release.download()
 
-        # Pull messages
-        ti: TaskInstance = kwargs['ti']
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_LIST,
-                               include_prior_dates=False)
-
-        # Prepare paths
-        grid_download_path = telescope_path(SubFolder.downloaded, GridTelescope.DAG_ID)
-
-        # Download and extract each release posted this month
-        msgs_out = []
-        for msg_in in msgs_in:
-            # Download release
-            article_id = msg_in['article_id']
-            title = msg_in['title']
-            files = download_grid_release(grid_download_path, article_id, title)
-
-            # Prepare metadata
-            for file_path in files:
-                msg_out = dict()
-                msg_out['download_path'] = file_path
-                msgs_out.append(msg_out)
-
-        # Push the selected GRID releases to an XCOM so that the next task knows which releases to download
-        ti.xcom_push(GridTelescope.RELEASES_TOPIC_NAME, msgs_out, kwargs['execution_date'])
-
-    @staticmethod
-    def upload_downloaded(**kwargs):
+    def upload_downloaded(self, releases: List[GridRelease], **kwargs):
         """ Task to upload the downloaded GRID releases for a given month.
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
+        :param releases: a list of GRID releases.
         :return: None.
         """
+        # Upload each downloaded release
+        for release in releases:
+            upload_files_from_list(release.download_files, release.download_bucket)
 
-        # Get bucket name
-        bucket_name = Variable.get(AirflowVars.DOWNLOAD_BUCKET)
-
-        # Pull messages
-        ti: TaskInstance = kwargs['ti']
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_DOWNLOAD,
-                               include_prior_dates=False)
-
-        # Upload each release
-        for msg_in in msgs_in:
-            download_path = msg_in['download_path']
-            file_name = os.path.basename(download_path)
-            blob_name = f'telescopes/grid/{file_name}'
-            upload_file_to_cloud_storage(bucket_name, blob_name, file_path=download_path)
-
-    @staticmethod
-    def extract(**kwargs):
+    def extract(self, releases: List[GridRelease], **kwargs):
         """ Task to extract the GRID releases for a given month.
-
-        Pushes the following xcom:
-            extracted_path (str): the path to the extracted GRID release.
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
+        :param releases: a list of GRID releases.
         :return: None.
         """
+        # Extract each release
+        for release in releases:
+            release.extract()
 
-        # Pull messages
-        ti: TaskInstance = kwargs['ti']
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_DOWNLOAD,
-                               include_prior_dates=False)
-
-        # Prepare paths
-        grid_extraction_path = telescope_path(SubFolder.extracted, GridTelescope.DAG_ID)
-
-        # Download and extract each release posted this month
-        msgs_out = []
-        for msg_in in msgs_in:
-            # Extract release
-            download_path = msg_in['download_path']
-            release_extracted_path = extract_grid_release(download_path, grid_extraction_path)
-
-            msg_out = dict()
-            msg_out['extracted_path'] = release_extracted_path
-            msgs_out.append(msg_out)
-
-        # Push the selected GRID releases to an XCOM so that the next task knows which releases to download
-        ti.xcom_push(GridTelescope.RELEASES_TOPIC_NAME, msgs_out, kwargs['execution_date'])
-
-    @staticmethod
-    def transform(**kwargs):
+    def transform(self, releases: List[GridRelease], **kwargs):
         """ Task to transform the GRID releases for a given month.
-
-        Pushes the following xcom:
-            version (str): the version of the GRID release.
-            json_gz_file_name (str): the file name for the transformed GRID release.
-            json_gz_file_path (str): the path to the transformed GRID release (including file name).
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
+        :param releases: a list of GRID releases.
         :return: None.
         """
-
-        # Pull GRID releases to transform
-        ti: TaskInstance = kwargs['ti']
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_EXTRACT,
-                               include_prior_dates=False)
-
-        # Prepare paths
-        grid_transformed_path = telescope_path(SubFolder.transformed, GridTelescope.DAG_ID)
-
         # Transform each release
-        msgs_out = []
-        for msg_in in msgs_in:
-            extracted_path = msg_in['extracted_path']
-            json_files = glob.glob(f"{extracted_path}/*.json")
-
-            # Only process JSON files, skip the rest
-            if len(json_files) > 0:
-                release_json_file = json_files[0]
-                logging.info(f'Transforming file: {release_json_file}')
-
-                version, file_name, file_path = transform_grid_release(release_json_file, grid_transformed_path)
-
-                # Prepare messages
-                msg_out = dict()
-                msg_out['version'] = version
-                msg_out['json_gz_file_name'] = file_name
-                msg_out['json_gz_file_path'] = file_path
-                msgs_out.append(msg_out)
-            else:
-                logging.info(f"Skipping transforming as no JSON files in extracted path: {extracted_path}")
-
-        # Push messages for next task to consume
-        ti: TaskInstance = kwargs['ti']
-        ti.xcom_push(GridTelescope.RELEASES_TOPIC_NAME, msgs_out, kwargs['execution_date'])
-
-    @staticmethod
-    def upload_transformed(**kwargs):
-        """ Task to upload the transformed GRID releases for a given month to Google Cloud Storage .
-
-        Pushes the following xcom:
-            release_date (str): the release date of the GRID release.
-            blob_name (str): the name of the blob on the Google Cloud storage bucket.
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
-        :return: None.
-        """
-
-        # Pull messages
-        ti: TaskInstance = kwargs['ti']
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_TRANSFORM,
-                               include_prior_dates=False)
-
-        # Upload each release
-        bucket_name = Variable.get(AirflowVars.TRANSFORM_BUCKET)
-        msgs_out = []
-        for msg_in in msgs_in:
-            file_name = msg_in['json_gz_file_name']
-            file_path = msg_in['json_gz_file_path']
-
-            # Upload to cloud storage
-            release_date = file_name.replace("grid_", "").replace(".jsonl.gz", "")
-            blob_name = f'telescopes/grid/{file_name}'
-            upload_file_to_cloud_storage(bucket_name, blob_name, file_path=file_path)
-
-            # Prepare metadata
-            msg_out = dict()
-            msg_out['release_date'] = release_date
-            msg_out['blob_name'] = blob_name
-            msgs_out.append(msg_out)
-
-        # Push messages for next task to consume
-        ti: TaskInstance = kwargs['ti']
-        ti.xcom_push(GridTelescope.RELEASES_TOPIC_NAME, msgs_out, kwargs['execution_date'])
-
-    @staticmethod
-    def bq_load(**kwargs):
-        """ Task to load the transformed GRID releases for a given month to BigQuery.
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
-        :return: None.
-        """
-
-        ti: TaskInstance = kwargs['ti']
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_UPLOAD_TRANSFORMED,
-                               include_prior_dates=False)
-
-        # Upload each release
-        project_id = Variable.get(AirflowVars.PROJECT_ID)
-        data_location = Variable.get(AirflowVars.DATA_LOCATION)
-        bucket_name = Variable.get(AirflowVars.TRANSFORM_BUCKET)
-
-        # Create dataset
-        create_bigquery_dataset(project_id, GridTelescope.DATASET_ID, data_location, GridTelescope.DESCRIPTION)
-
-        analysis_schema_path = schema_path()
-        table_name = 'grid'
-
-        # Load into BigQuery table
-        for msg_in in msgs_in:
-            release_date_str = msg_in['release_date']
-            release_date = pendulum.parse(release_date_str.replace("_", "-"))
-            blob_name = msg_in['blob_name']
-
-            # Select schema file based on release date
-            schema_file_path = find_schema(analysis_schema_path, table_name, release_date)
-            if schema_file_path is None:
-                logging.error(f'No schema found with search parameters: analysis_schema_path={analysis_schema_path}, '
-                              f'table_name={table_name}, release_date={release_date}')
-                exit(os.EX_CONFIG)
-
-            # Create table id
-            table_id = bigquery_partitioned_table_id(GridTelescope.DAG_ID, release_date)
-
-            # Load BigQuery table
-            uri = f"gs://{bucket_name}/{blob_name}"
-            logging.info(f"URI: {uri}")
-            load_bigquery_table(uri, GridTelescope.DATASET_ID, data_location, table_id, schema_file_path,
-                                SourceFormat.NEWLINE_DELIMITED_JSON)
-
-    @staticmethod
-    def cleanup(**kwargs):
-        """ Delete files of downloaded, extracted and transformed releases.
-
-        :param kwargs: the context passed from the PythonOperator. See
-        https://airflow.apache.org/docs/stable/macros-ref.html
-        for a list of the keyword arguments that are passed to this argument.
-        :return: None.
-        """
-
-        # Pull releases
-        ti: TaskInstance = kwargs['ti']
-
-        # Remove downloaded files
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_DOWNLOAD,
-                               include_prior_dates=False)
-        for msg_in in msgs_in:
-            file_path = msg_in['download_path']
-            try:
-                pathlib.Path(file_path).unlink()
-            except FileNotFoundError as e:
-                logging.warning(f"No such file or directory {file_path}: {e}")
-
-        # Remove extracted files
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_EXTRACT,
-                               include_prior_dates=False)
-        for msg_in in msgs_in:
-            file_path = msg_in['extracted_path']
-            try:
-                shutil.rmtree(file_path)
-            except FileNotFoundError as e:
-                logging.warning(f"No such file or directory {file_path}: {e}")
-
-        # Remove transformed files
-        msgs_in = ti.xcom_pull(key=GridTelescope.RELEASES_TOPIC_NAME, task_ids=GridTelescope.TASK_ID_TRANSFORM,
-                               include_prior_dates=False)
-        for msg_in in msgs_in:
-            file_path = msg_in['json_gz_file_path']
-            try:
-                pathlib.Path(file_path).unlink()
-            except FileNotFoundError as e:
-                logging.warning(f"No such file or directory {file_path}: {e}")
+        for release in releases:
+            release.transform()
diff --git a/observatory-dags/observatory/dags/telescopes/mag.py b/observatory-dags/observatory/dags/telescopes/mag.py
index ff39adfbb..4605397f7 100644
--- a/observatory-dags/observatory/dags/telescopes/mag.py
+++ b/observatory-dags/observatory/dags/telescopes/mag.py
@@ -35,27 +35,24 @@
 from google.cloud.storage import Blob
 from mag_archiver.mag import MagArchiverClient, MagDateType, MagRelease, MagState
 from natsort import natsorted
-from pendulum import Pendulum
-
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import (AirflowConns,
-                                                     AirflowVars,
-                                                     SubFolder,
-                                                     check_connections,
-                                                     check_variables,
-                                                     telescope_path,
-                                                     test_data_path)
+from observatory.platform.utils.airflow_utils import AirflowConns, \
+    AirflowVariable as Variable, \
+    AirflowVars, \
+    check_connections, \
+    check_variables
 from observatory.platform.utils.config_utils import find_schema
 from observatory.platform.utils.gc_utils import (azure_to_google_cloud_storage_transfer,
                                                  bigquery_partitioned_table_id,
+                                                 bigquery_table_exists,
                                                  create_bigquery_dataset,
                                                  download_blobs_from_cloud_storage,
                                                  load_bigquery_table,
                                                  table_name_from_blob,
-                                                 upload_files_to_cloud_storage,
-                                                 bigquery_table_exists)
+                                                 upload_files_to_cloud_storage)
 from observatory.platform.utils.proc_utils import wait_for_process
+from observatory.platform.utils.template_utils import SubFolder, telescope_path, test_data_path
+from pendulum import Pendulum
 
 
 def pull_releases(ti: TaskInstance) -> List[MagRelease]:
@@ -195,9 +192,8 @@ def check_dependencies(**kwargs):
         :return: None.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         conns_valid = check_connections(AirflowConns.MAG_RELEASES_TABLE, AirflowConns.MAG_SNAPSHOTS_CONTAINER)
 
         if not vars_valid or not conns_valid:
diff --git a/observatory-dags/observatory/dags/telescopes/oapen_metadata.py b/observatory-dags/observatory/dags/telescopes/oapen_metadata.py
new file mode 100644
index 000000000..61579ebb6
--- /dev/null
+++ b/observatory-dags/observatory/dags/telescopes/oapen_metadata.py
@@ -0,0 +1,287 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs
+
+import csv
+import logging
+import os
+from datetime import datetime
+from typing import Tuple
+
+import pendulum
+from airflow.exceptions import AirflowException
+from airflow.models.taskinstance import TaskInstance
+from observatory.platform.telescopes.stream_telescope import (StreamRelease, StreamTelescope)
+from observatory.platform.utils.airflow_utils import AirflowVars
+from observatory.platform.utils.telescope_utils import convert, list_to_jsonl_gz
+from observatory.platform.utils.template_utils import upload_files_from_list
+from observatory.platform.utils.url_utils import get_ao_user_agent, retry_session
+
+
+class OapenMetadataRelease(StreamRelease):
+    def __init__(self, dag_id: str, start_date: pendulum.Pendulum, end_date: pendulum.Pendulum, first_release: bool):
+        """ Construct a OapenMetadataRelease instance
+        :param dag_id: the id of the DAG.
+        :param start_date: the start_date of the release.
+        :param end_date: the end_date of the release.
+        :param first_release: whether this is the first release that is processed for this DAG
+        """
+        super().__init__(dag_id, start_date, end_date, first_release)
+
+    @property
+    def csv_path(self) -> str:
+        """ Path to store the original oapen metadata csv file"""
+        return os.path.join(self.download_folder, 'oapen_metadata.csv')
+
+    @property
+    def transform_path(self) -> str:
+        """ Path to store the transformed oapen metadata file"""
+        return os.path.join(self.transform_folder, 'metadata.jsonl.gz')
+
+    def download(self) -> bool:
+        """ Download Oapen metadata CSV.
+        :return: True if download is successful
+        """
+        logging.info(f'Downloading csv from url: {OapenMetadataTelescope.CSV_URL}')
+        headers = {
+            'User-Agent': f'{get_ao_user_agent()}'
+        }
+        response = retry_session().get(OapenMetadataTelescope.CSV_URL, headers=headers)
+        if response.status_code == 200:
+            with open(self.csv_path, 'w') as f:
+                f.write(response.content.decode('utf-8'))
+            logging.info(f'Downloaded csv successful to {self.csv_path}')
+            return True
+        else:
+            raise AirflowException(f'Download csv unsuccessful, {response.text}')
+
+    def transform(self):
+        """ Transform the oapen metadata csv file by storing in a jsonl format and restructuring lists/dicts.
+        Values of field names with '.' are formatted into nested dictionaries.
+        Values of field names in list_fields are split on - , ; and ||.
+        Values of the field 'dc.subject.classification' are parsed to create a custom field 'classification_code'.
+        See our readthedocs for an example row before and after transformation
+        :return: None
+        """
+        with open(self.csv_path, 'r') as f:
+            csv_entries = [{k: v for k, v in row.items()} for row in csv.DictReader(f, skipinitialspace=True)]
+
+        nested_fields = get_nested_fieldnames(csv_entries)
+        # values of these fields should be transformed to a list
+        list_fields = {'oapen.grant.number', 'oapen.grant.acronym', 'oapen.relation.hasChapter_dc.title',
+                       'dc.subject.classification', 'oapen.grant.program', 'oapen.redirect', 'oapen.imprint',
+                       'dc.subject.other', 'oapen.notes', 'dc.title', 'collection', 'dc.relation.ispartofseries',
+                       'BITSTREAM Download URL', 'BITSTREAM ISBN', 'oapen.collection', 'dc.contributor.author',
+                       'oapen.remark.public', 'oapen.relation.isPublisherOf', 'dc.contributor.other',
+                       'dc.contributor.editor', 'dc.relation.isreplacedbydouble', 'dc.date.issued',
+                       'dc.description.abstract', 'BITSTREAM Webshop URL', 'dc.type', 'dc.identifier.isbn',
+                       'dc.description.provenance', 'oapen.grant.project', 'oapen.relation.isbn', 'dc.identifier',
+                       'dc.date.accessioned', 'BITSTREAM License', 'oapen.relation.isFundedBy_grantor.name',
+                       'dc.relation.isnodouble', 'dc.language', 'grantor.number'}
+        entries = transform_dict(csv_entries, convert, nested_fields, list_fields)
+
+        # Transform release into JSON Lines format saving in memory buffer
+        # Save in memory buffer to gzipped file
+        list_to_jsonl_gz(self.transform_path, entries)
+
+
+class OapenMetadataTelescope(StreamTelescope):
+    """ Oapen Metadata telescope """
+    CSV_URL = 'https://library.oapen.org/download-export?format=csv'
+
+    def __init__(self, dag_id: str = 'oapen_metadata', start_date: datetime = datetime(2018, 5, 14),
+                 schedule_interval: str = '@weekly', dataset_id: str = 'oapen', merge_partition_field: str = 'id',
+                 updated_date_field: str = 'dc.date.available', bq_merge_days: int = 7, schema_prefix: str =
+                 'oapen_', airflow_vars=None):
+        """ Construct a OapenMetadataTelescope instance.
+        :param dag_id: the id of the DAG.
+        :param start_date: the start date of the DAG.
+        :param schedule_interval: the schedule interval of the DAG.
+        :param dataset_id: the dataset id.
+        :param merge_partition_field: the BigQuery field used to match partitions for a merge
+        :param updated_date_field: the BigQuery field used to determine newest entry for a merge
+        :param bq_merge_days: how often partitions should be merged (every x days)
+        :param schema_prefix: the prefix used to find the schema path
+        :param airflow_vars: list of airflow variable keys, for each variable it is checked if it exists in airflow
+        """
+        if airflow_vars is None:
+            airflow_vars = [AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                            AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET]
+        super().__init__(dag_id, start_date, schedule_interval, dataset_id, merge_partition_field,
+                         updated_date_field, bq_merge_days, schema_prefix=schema_prefix, airflow_vars=airflow_vars)
+
+        self.add_setup_task_chain([self.check_dependencies,
+                                   self.get_release_info])
+        self.add_task_chain([self.download,
+                             self.upload_downloaded,
+                             self.transform,
+                             self.upload_transformed,
+                             self.bq_load_partition])
+        self.add_task_chain(self.make_operators([self.bq_delete_old,
+                                                 self.bq_append_new,
+                                                 self.cleanup]))
+
+    def make_release(self, **kwargs) -> 'OapenMetadataRelease':
+        # Make Release instance
+        ti: TaskInstance = kwargs['ti']
+        start_date, end_date, first_release = ti.xcom_pull(key=OapenMetadataTelescope.RELEASE_INFO,
+                                                           include_prior_dates=True)
+
+        release = OapenMetadataRelease(self.dag_id, start_date, end_date, first_release)
+        return release
+
+    def download(self, release: OapenMetadataRelease, **kwargs):
+        """ Task to download the OapenMetadataRelease release.
+        :param release: an OapenMetadataRelease instance.
+        :return: None.
+        """
+        # Download release
+        release.download()
+
+    def upload_downloaded(self, release: OapenMetadataRelease, **kwargs):
+        """ Task to upload the downloadeded OapenMetadataRelease release.
+        :param release: an OapenMetadataRelease instance.
+        :return: None.
+        """
+        # Upload each downloaded release
+        upload_files_from_list(release.download_files, release.download_bucket)
+
+    def transform(self, release: OapenMetadataRelease, **kwargs):
+        """ Task to transform the OapenMetadataRelease release.
+        :param release: an OapenMetadataRelease instance.
+        :return: None.
+        """
+        # Transform each release
+        release.transform()
+
+
+def get_nested_fieldnames(csv_entries: dict) -> set:
+    """ Fieldnames with '.' should be converted to nested dictionaries. This function will return a set of
+    fieldnames for nested dictionaries from the highest to second lowest levels.
+    E.g these fieldnames: dc.date.available, dc.date.issued, dc.description
+    Will give this set: dc, dc.date, dc.description
+
+    :param csv_entries: Dictionary with csv entries
+    :return: Set of field names which should be converted to nested fields
+    """
+    keys = csv_entries[0].keys()
+    nested_fields = set()
+    for key in keys:
+        # split string in two, starting from the right
+        split_key = key.rsplit('.', 1)
+        # add key to set if there is at least one '.' in key name
+        if len(split_key) > 1:
+            nested_fields.add(split_key[0])
+    return nested_fields
+
+
+def transform_value_to_list(k: str, v: str) -> Tuple[list, list]:
+    """ Takes a key and value from the dictionary of csv entries. The value is always in a string format. Based on the
+    key name (k) the delimiter in the the value (v) is replaced with '||'. All values are then split on '||' so they
+    are transformed into a list. The values of 'dc.subject.classification' are parsed and stored in a variable,
+    so they can later be added to a custom key of the csv entries dictionary.
+
+    :param k: Dictionary key
+    :param v: Dictionary value (string)
+    :return: Dictionary value (list) and classification code info.
+    """
+    # Get classification code for custom added column
+    classification_code = []
+    if k == 'BITSTREAM ISBN':
+        v = v.replace('-', '')
+    if k == 'dc.subject.other':
+        v = v.replace(';', '||')
+        v = v.replace(',', '||')
+    v = list(dict.fromkeys([x.strip() for x in v.split('||')]))
+    if k == 'dc.date.issued':
+        v = [pendulum.parse(date).to_date_string() for date in v]
+    if k == 'dc.subject.classification':
+        for c in v:
+            if c.startswith('bic Book Industry Communication::'):
+                code = c.split('::')[-1].split(' ')[0]
+                classification_code.append(code)
+            else:
+                classification_code.append(c)
+        classification_code = list(dict.fromkeys(classification_code))
+    return v, classification_code
+
+
+def transform_key_to_nested_dict(k: str, v, nested_fields: set, list_fields: set, classification_code: list, new: dict):
+    """ Takes a dictionary key and value. The key is split on '.', a nested dictionary is created and the value will be
+    added to the most nested level. The dictionary is updated in place so it is not returned.
+    For example first k = 'dc.date.issued', the dictionary = {'dc': {'date': {'issued': v1}}
+    Second k = 'dc.date.accessed', the dictionary = {'dc': {'date': {'issued': v1, 'accessed': v2}}
+
+    :param k: Dictionary key
+    :param v: Dictionary value
+    :param nested_fields: Set of field names for which the values should be a nested dictionary
+    :param list_fields: Set of field names for which the values should be a list
+    :param classification_code: List of classification code abbreviations
+    :param new: New, updated dictionary
+    :return: None.
+    """
+    # Get all (nested) fields of a specific key
+    fields = k.split('.')
+    tmp = new
+    # Create one dictionary for each field
+    for key in fields:
+        key = convert(key)
+        try:
+            tmp[key]
+        except KeyError:
+            # Add the value to the most nested level
+            if key == fields[-1]:
+                tmp[key] = transform_dict(v, convert, nested_fields, list_fields)
+                if key == 'classification':
+                    # Add classification code column
+                    tmp['classification_code'] = transform_dict(classification_code, convert, nested_fields,
+                                                                list_fields)
+            # Create empty dictionary for key
+            else:
+                tmp[key] = {}
+        tmp = tmp[key]
+
+
+def transform_dict(obj, convert, nested_fields, list_fields):
+    """ Recursively goes through the dictionary obj, replaces keys with the convert function.
+    :param obj: object, can be of any type
+    :param convert: convert function
+    :param nested_fields: fields
+    :param list_fields:
+    :return: Updated dictionary
+    """
+    if isinstance(obj, (str, int, float)):
+        return obj
+    if isinstance(obj, dict):
+        new = obj.__class__()
+        for k, v in list(obj.items()):
+            classification_code = []
+            if v == '':
+                continue
+            if k in list_fields:
+                v, classification_code = transform_value_to_list(k, v)
+            # change key name for top level of nested dictionary
+            if k in nested_fields:
+                k = k + '.value'
+            # parse key/value of lower levels of nested dictionary
+            if k.rsplit('.', 1)[0] in nested_fields:
+                transform_key_to_nested_dict(k, v, nested_fields, list_fields, classification_code, new)
+            else:
+                new[convert(k)] = transform_dict(v, convert, nested_fields, list_fields)
+    elif isinstance(obj, (list, set, tuple)):
+        new = obj.__class__(transform_dict(v, convert, nested_fields, list_fields) for v in obj)
+    else:
+        return obj
+    return new
diff --git a/observatory-dags/observatory/dags/telescopes/open_citations.py b/observatory-dags/observatory/dags/telescopes/open_citations.py
index 2b92c45cb..c56c0ec48 100644
--- a/observatory-dags/observatory/dags/telescopes/open_citations.py
+++ b/observatory-dags/observatory/dags/telescopes/open_citations.py
@@ -27,20 +27,19 @@
 from airflow.exceptions import AirflowException
 from airflow.models.taskinstance import TaskInstance
 from google.cloud.bigquery import SourceFormat
-from pendulum import Pendulum
-
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import AirflowVars, SubFolder, telescope_path, check_variables
+from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, AirflowVars, check_variables
 from observatory.platform.utils.config_utils import find_schema
 from observatory.platform.utils.data_utils import get_file
 from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
+                                                 bigquery_table_exists,
                                                  create_bigquery_dataset,
                                                  load_bigquery_table,
-                                                 bigquery_table_exists,
                                                  upload_files_to_cloud_storage)
 from observatory.platform.utils.proc_utils import wait_for_process
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
 from observatory.platform.utils.url_utils import retry_session
+from pendulum import Pendulum
 
 OPEN_CITATIONS_ARTICLE_ID = 6741422
 OPEN_CITATIONS_VERSION_URL = "https://api.figshare.com/v2/articles/{article_id}/versions"
@@ -76,22 +75,19 @@ def release_name(self) -> str:
 
     @property
     def download_path(self) -> str:
-        return os.path.join(telescope_path(SubFolder.downloaded, OpenCitationsTelescope.DAG_ID),
-                            self.release_name)
+        return os.path.join(telescope_path(SubFolder.downloaded, OpenCitationsTelescope.DAG_ID), self.release_name)
 
     @property
     def extract_path(self) -> str:
-        return os.path.join(telescope_path(SubFolder.extracted, OpenCitationsTelescope.DAG_ID),
-                            self.release_name)
+        return os.path.join(telescope_path(SubFolder.extracted, OpenCitationsTelescope.DAG_ID), self.release_name)
 
     @property
     def transformed_blob_path(self) -> str:
         return f'telescopes/{OpenCitationsTelescope.DAG_ID}/{self.release_name}/*.csv'
 
 
-def list_open_citations_releases(start_date: Pendulum = None,
-                                 end_date: Pendulum = None,
-                                 timeout: float = 30.) -> List[OpenCitationsRelease]:
+def list_open_citations_releases(start_date: Pendulum = None, end_date: Pendulum = None, timeout: float = 30.) -> List[
+    OpenCitationsRelease]:
     """ List the Open Citations releases for a given time period.
 
     :param start_date: the start date.
@@ -136,8 +132,7 @@ def fetch_open_citations_versions():
 
 def pull_releases(ti: TaskInstance) -> List[OpenCitationsRelease]:
     return ti.xcom_pull(key=OpenCitationsTelescope.RELEASES_TOPIC_NAME,
-                        task_ids=OpenCitationsTelescope.TASK_ID_LIST_RELEASES,
-                        include_prior_dates=False)
+                        task_ids=OpenCitationsTelescope.TASK_ID_LIST_RELEASES, include_prior_dates=False)
 
 
 class OpenCitationsTelescope:
@@ -168,9 +163,8 @@ def check_dependencies(**kwargs):
         for a list of the keyword arguments that are passed to this argument.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         if not vars_valid:
             raise AirflowException('Required variables are missing')
 
@@ -227,11 +221,8 @@ def download(**kwargs):
         for release in releases:
             os.makedirs(release.download_path, exist_ok=True)
             for file in release.files:
-                file_path, updated = get_file(file.name,
-                                              file.download_url,
-                                              cache_subdir='',
-                                              cache_dir=release.download_path,
-                                              md5_hash=file.md5hash)
+                file_path, updated = get_file(file.name, file.download_url, cache_subdir='',
+                                              cache_dir=release.download_path, md5_hash=file.md5hash)
 
     @staticmethod
     def upload_downloaded(**kwargs):
@@ -293,8 +284,7 @@ def extract(**kwargs):
                     logging.info(stdout)
 
                 if stderr:
-                    raise AirflowException(
-                        f"bash command failed for {file_path}, {release.release_date}: {stderr}")
+                    raise AirflowException(f"bash command failed for {file_path}, {release.release_date}: {stderr}")
 
                 logging.info(f"File extracted to: {release.extract_path}")
 
diff --git a/observatory-dags/observatory/dags/telescopes/scopus.py b/observatory-dags/observatory/dags/telescopes/scopus.py
index 748c21be5..03f340aec 100644
--- a/observatory-dags/observatory/dags/telescopes/scopus.py
+++ b/observatory-dags/observatory/dags/telescopes/scopus.py
@@ -21,9 +21,10 @@
 import os
 import urllib.request
 from concurrent.futures import ThreadPoolExecutor, as_completed
-from queue import Queue, Empty
+from queue import Empty, Queue
 from threading import Event
-from typing import List, Tuple, Type, Union, Dict, Any
+from time import sleep
+from typing import Any, Dict, List, Tuple, Type, Union
 from urllib.error import HTTPError
 from urllib.parse import quote_plus
 
@@ -32,38 +33,32 @@
 from airflow.models import Variable
 from airflow.models.taskinstance import TaskInstance
 from google.cloud.bigquery import SourceFormat, WriteDisposition
-from ratelimit import limits, sleep_and_retry
-from time import sleep
-
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import (AirflowVars, SubFolder, find_schema, telescope_path,
-                                                     check_variables)
-from observatory.platform.utils.gc_utils import (
-    bigquery_partitioned_table_id,
-    create_bigquery_dataset,
-    load_bigquery_table,
-    upload_telescope_file_list,
-)
-from observatory.platform.utils.telescope_utils import (
-    build_schedule,
-    delete_msg_files,
-    get_entry_or_none,
-    get_as_list,
-    json_to_db,
-    validate_date,
-    write_to_file,
-    zip_files,
-)
+from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, AirflowVars, check_variables
+from observatory.platform.utils.config_utils import (find_schema)
+from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
+                                                 create_bigquery_dataset,
+                                                 load_bigquery_table, )
+from observatory.platform.utils.telescope_utils import (build_schedule,
+                                                        delete_msg_files,
+                                                        get_as_list,
+                                                        get_entry_or_none,
+                                                        json_to_db,
+                                                        upload_telescope_file_list,
+                                                        validate_date,
+                                                        write_to_file,
+                                                        zip_files)
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
 from observatory.platform.utils.url_utils import get_ao_user_agent
+from ratelimit import limits, sleep_and_retry
 
 
 class ScopusRelease:
     """ Used to store info on a given SCOPUS release. """
 
-    def __init__(self, inst_id: str, scopus_inst_id: List[str], release_date: pendulum.date,
-                 start_date: pendulum.date, end_date: pendulum.date, project_id: str,
-                 download_bucket_name: str, transform_bucket_name: str, data_location: str, schema_ver: str):
+    def __init__(self, inst_id: str, scopus_inst_id: List[str], release_date: pendulum.date, start_date: pendulum.date,
+                 end_date: pendulum.date, project_id: str, download_bucket_name: str, transform_bucket_name: str,
+                 data_location: str, schema_ver: str):
         """ Constructor.
 
         :param inst_id: institution id from the airflow connection (minus the scopus_)
@@ -141,9 +136,8 @@ def check_dependencies(**kwargs):
         :return: None.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         if not vars_valid:
             raise AirflowException('Required variables are missing')
 
@@ -207,8 +201,8 @@ def check_dependencies(**kwargs):
                                 data_location=data_location, schema_ver=ScopusTelescope.SCHEMA_VER)
 
         logging.info(
-            f'ScopusRelease contains:\ndownload_bucket_name: {release.download_bucket_name}, transform_bucket_name: ' +
-            f'{release.transform_bucket_name}, data_location: {release.data_location}')
+            f'ScopusRelease contains:\ndownload_bucket_name: {release.download_bucket_name}, transform_bucket_name: '
+            + f'{release.transform_bucket_name}, data_location: {release.data_location}')
 
         ti: TaskInstance = kwargs['ti']
         ti.xcom_push(ScopusTelescope.XCOM_RELEASES, release)
@@ -444,7 +438,7 @@ class ScopusClient:
     MAX_RESULTS = 5000  # Upper limit on number of results returned
     QUOTA_EXCEED_ERROR_PREFIX = 'QuotaExceeded. Resets at: '
 
-    def __init__(self, api_key: str, view: str='standard'):
+    def __init__(self, api_key: str, view: str = 'standard'):
         """ Constructor.
 
         :param api_key: API key.
@@ -656,8 +650,7 @@ def update_reset_date(conn: str, error_msg: str, worker: ScopusUtilWorker,
         renews_ts = ScopusClient.get_reset_date_from_error(error_msg)
         worker.quota_reset_date = pendulum.from_timestamp(renews_ts)
 
-        logging.warning(
-            f'{conn} worker {worker.client_id}: quoted exceeded. New reset date: {worker.quota_reset_date}')
+        logging.warning(f'{conn} worker {worker.client_id}: quoted exceeded. New reset date: {worker.quota_reset_date}')
 
         if now > curr_reset_date:
             return worker.quota_reset_date
@@ -790,9 +783,8 @@ def download_parallel(workers: List[ScopusUtilWorker], taskq: Queue, conn: str,
             thread_exit = Event()
 
             for worker in workers:
-                futures.append(
-                    executor.submit(ScopusUtility.download_worker, worker, thread_exit, taskq, conn, inst_id,
-                                    download_path))
+                futures.append(executor.submit(ScopusUtility.download_worker, worker, thread_exit, taskq, conn, inst_id,
+                                               download_path))
 
             taskq.join()  # Wait until all tasks done
             logging.info(f'{conn}: all tasks fetched. Signalling threads to exit.')
@@ -863,7 +855,7 @@ class ScopusJsonParser:
     """ Helper methods to process the json from SCOPUS into desired structure. """
 
     @staticmethod
-    def get_affiliations(data: Dict[str, Any]) -> Union[None,List[Dict[str, Any]]]:
+    def get_affiliations(data: Dict[str, Any]) -> Union[None, List[Dict[str, Any]]]:
         """ Get the affiliation field.
 
         :param data: json response from SCOPUS.
diff --git a/observatory-dags/observatory/dags/telescopes/unpaywall.py b/observatory-dags/observatory/dags/telescopes/unpaywall.py
index 63defbdb4..a7fe32551 100644
--- a/observatory-dags/observatory/dags/telescopes/unpaywall.py
+++ b/observatory-dags/observatory/dags/telescopes/unpaywall.py
@@ -28,13 +28,9 @@
 from airflow.models import Variable
 from airflow.models.taskinstance import TaskInstance
 from google.cloud.bigquery import SourceFormat
-from pendulum import Pendulum
-
 from observatory.dags.config import schema_path
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable
-from observatory.platform.utils.config_utils import AirflowVars, SubFolder, find_schema, telescope_path
-from observatory.platform.utils.config_utils import check_variables
-from observatory.platform.utils.config_utils import test_data_path
+from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, AirflowVars, check_variables
+from observatory.platform.utils.config_utils import find_schema
 from observatory.platform.utils.data_utils import get_file
 from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
                                                  bigquery_table_exists,
@@ -42,7 +38,9 @@
                                                  load_bigquery_table,
                                                  upload_file_to_cloud_storage)
 from observatory.platform.utils.proc_utils import wait_for_process
+from observatory.platform.utils.template_utils import SubFolder, telescope_path, test_data_path
 from observatory.platform.utils.url_utils import retry_session
+from pendulum import Pendulum
 
 
 def pull_releases(ti: TaskInstance) -> List:
@@ -252,9 +250,8 @@ def check_dependencies(**kwargs):
         :return: None.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         if not vars_valid:
             raise AirflowException('Required variables are missing')
 
diff --git a/observatory-dags/observatory/dags/telescopes/wos.py b/observatory-dags/observatory/dags/telescopes/wos.py
index ff332e73a..597fc37f7 100644
--- a/observatory-dags/observatory/dags/telescopes/wos.py
+++ b/observatory-dags/observatory/dags/telescopes/wos.py
@@ -21,7 +21,8 @@
 import urllib.request
 from collections import OrderedDict
 from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import List, Type, Union, Any, Dict
+from math import ceil
+from typing import Any, Dict, List, Type, Union
 from urllib.error import URLError
 
 import backoff
@@ -31,38 +32,28 @@
 from airflow.models import Variable
 from airflow.models.taskinstance import TaskInstance
 from google.cloud.bigquery import SourceFormat, WriteDisposition
-from math import ceil
+from observatory.dags.config import schema_path
+from observatory.platform.utils.airflow_utils import AirflowVars, check_variables
+from observatory.platform.utils.config_utils import (find_schema, )
+from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
+                                                 create_bigquery_dataset,
+                                                 load_bigquery_table, )
+from observatory.platform.utils.telescope_utils import (build_schedule,
+                                                        delete_msg_files,
+                                                        get_as_list,
+                                                        get_as_list_or_none,
+                                                        get_entry_or_none,
+                                                        json_to_db,
+                                                        upload_telescope_file_list,
+                                                        validate_date,
+                                                        write_to_file,
+                                                        write_xml_to_json,
+                                                        zip_files)
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
 from ratelimit import limits, sleep_and_retry
 from suds import WebFault
 from wos import WosClient
 
-from observatory.dags.config import schema_path
-from observatory.platform.utils.config_utils import (
-    AirflowVars,
-    SubFolder,
-    check_variables,
-    find_schema,
-    telescope_path,
-)
-from observatory.platform.utils.gc_utils import (
-    bigquery_partitioned_table_id,
-    create_bigquery_dataset,
-    load_bigquery_table,
-    upload_telescope_file_list,
-)
-from observatory.platform.utils.telescope_utils import (
-    build_schedule,
-    delete_msg_files,
-    get_as_list,
-    get_as_list_or_none,
-    get_entry_or_none,
-    json_to_db,
-    validate_date,
-    write_xml_to_json,
-    write_to_file,
-    zip_files,
-)
-
 
 class WosUtilConst:
     """ Class containing some WosUtility constants. Makes these values accessible by decorators. """
@@ -127,9 +118,8 @@ def parse_query(records: Any) -> (dict, str):
         prefix_len = len(WosTelescope.SCHEMA_ID_PREFIX)
         prefix_loc = schema_string.find(WosTelescope.SCHEMA_ID_PREFIX)
         if prefix_loc == -1:
-            logging.warning(
-                f'WOS schema has changed.\nExpecting prefix: {WosTelescope.SCHEMA_ID_PREFIX}\n'
-                f'Received: {schema_string}')
+            logging.warning(f'WOS schema has changed.\nExpecting prefix: {WosTelescope.SCHEMA_ID_PREFIX}\n'
+                            f'Received: {schema_string}')
             return None
 
         ver_start = prefix_len
@@ -326,8 +316,8 @@ class WosRelease:
     """
 
     def __init__(self, inst_id: str, wos_inst_id: List[str], release_date: pendulum.Pendulum,
-                 dag_start: pendulum.Pendulum, project_id: str,
-                 download_bucket_name: str, transform_bucket_name: str, data_location: str, schema_ver: str):
+                 dag_start: pendulum.Pendulum, project_id: str, download_bucket_name: str, transform_bucket_name: str,
+                 data_location: str, schema_ver: str):
         self.inst_id = inst_id
         self.wos_inst_id = wos_inst_id
         self.release_date = release_date
@@ -391,9 +381,8 @@ def check_dependencies(**kwargs):
         :return: None.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
         if not vars_valid:
             raise AirflowException('Required variables are missing')
 
@@ -572,7 +561,7 @@ def transform_xml(**kwargs):
         json_harvest = list()
         for file in json_file_list:
             file_name = os.path.basename(file)
-            start = file_name.find('_')+1
+            start = file_name.find('_') + 1
             end = file_name.rfind('-')
             harvest_datetime = file_name[start:end]
             json_path.append(file)
@@ -599,12 +588,10 @@ def transform_db_format(**kwargs):
         release: WosRelease = WosTelescope.pull_release(ti)
 
         # Pull messages
-        json_files = ti.xcom_pull(key=WosTelescope.XCOM_JSON_PATH,
-                                  task_ids=WosTelescope.TASK_ID_TRANSFORM_XML,
+        json_files = ti.xcom_pull(key=WosTelescope.XCOM_JSON_PATH, task_ids=WosTelescope.TASK_ID_TRANSFORM_XML,
                                   include_prior_dates=False)
 
-        harvest_times = ti.xcom_pull(key=WosTelescope.XCOM_JSON_HARVEST,
-                                     task_ids=WosTelescope.TASK_ID_TRANSFORM_XML,
+        harvest_times = ti.xcom_pull(key=WosTelescope.XCOM_JSON_HARVEST, task_ids=WosTelescope.TASK_ID_TRANSFORM_XML,
                                      include_prior_dates=False)
 
         json_harvest_pair = list(zip(json_files, harvest_times))
@@ -637,8 +624,7 @@ def upload_transformed(**kwargs):
         release: WosRelease = WosTelescope.pull_release(ti)
 
         # Pull messages
-        jsonl_paths = ti.xcom_pull(key=WosTelescope.XCOM_JSONL_PATH,
-                                   task_ids=WosTelescope.TASK_ID_TRANSFORM_DB_FORMAT,
+        jsonl_paths = ti.xcom_pull(key=WosTelescope.XCOM_JSONL_PATH, task_ids=WosTelescope.TASK_ID_TRANSFORM_DB_FORMAT,
                                    include_prior_dates=False)
 
         # Upload each snapshot
@@ -665,8 +651,7 @@ def bq_load(**kwargs):
         release: WosRelease = WosTelescope.pull_release(ti)
 
         jsonl_zip_blobs = ti.xcom_pull(key=WosTelescope.XCOM_JSONL_BLOB_PATH,
-                                       task_ids=WosTelescope.TASK_ID_UPLOAD_TRANSFORMED,
-                                       include_prior_dates=False)
+                                       task_ids=WosTelescope.TASK_ID_UPLOAD_TRANSFORMED, include_prior_dates=False)
 
         # Create dataset
         create_bigquery_dataset(release.project_id, WosTelescope.DATASET_ID, release.data_location,
@@ -679,8 +664,8 @@ def bq_load(**kwargs):
         analysis_schema_path = schema_path()
 
         for file in jsonl_zip_blobs:
-            schema_file_path = find_schema(analysis_schema_path, WosTelescope.TABLE_NAME,
-                                           release.release_date, '', release.schema_ver)
+            schema_file_path = find_schema(analysis_schema_path, WosTelescope.TABLE_NAME, release.release_date, '',
+                                           release.schema_ver)
             if schema_file_path is None:
                 logging.error(
                     f'No schema found with search parameters: analysis_schema_path={WosTelescope.SCHEMA_PATH}, '
@@ -693,8 +678,7 @@ def bq_load(**kwargs):
             logging.info(f"URI: {uri}")
 
             load_bigquery_table(uri, WosTelescope.DATASET_ID, release.data_location, table_id, schema_file_path,
-                                SourceFormat.NEWLINE_DELIMITED_JSON,
-                                write_disposition=WriteDisposition.WRITE_APPEND)
+                                SourceFormat.NEWLINE_DELIMITED_JSON, write_disposition=WriteDisposition.WRITE_APPEND)
 
     @staticmethod
     def cleanup(**kwargs):
@@ -938,7 +922,10 @@ def get_languages(data: dict) -> List[Dict[str, str]]:
 
         languages = get_as_list(data['static_data']['fullrecord_metadata']['languages'], 'language')
         for entry in languages:
-            lang_list.append({"type": entry['@type'], "name": entry['#text']})
+            lang_list.append({
+                                 "type": entry['@type'],
+                                 "name": entry['#text']
+                             })
         return lang_list
 
     @staticmethod
diff --git a/observatory-dags/observatory/dags/workflows/doi.py b/observatory-dags/observatory/dags/workflows/doi.py
index 36c9796cb..eecdc9195 100644
--- a/observatory-dags/observatory/dags/workflows/doi.py
+++ b/observatory-dags/observatory/dags/workflows/doi.py
@@ -21,19 +21,21 @@
 import pendulum
 from airflow.exceptions import AirflowException
 from airflow.models import Variable
-from pendulum import Pendulum
-
 from observatory.dags.config import workflow_sql_templates_path
 from observatory.dags.telescopes.crossref_metadata import CrossrefMetadataTelescope
 from observatory.dags.telescopes.fundref import FundrefTelescope
 from observatory.dags.telescopes.grid import GridTelescope
 from observatory.dags.telescopes.mag import MagTelescope
 from observatory.dags.telescopes.unpaywall import UnpaywallTelescope
-from observatory.platform.utils.config_utils import AirflowVars, check_variables
-from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id, create_bigquery_table_from_query,
-                                                 run_bigquery_query, create_bigquery_dataset, copy_bigquery_table,
-                                                 create_bigquery_view)
-from observatory.platform.utils.jinja2_utils import render_template, make_sql_jinja2_filename
+from observatory.platform.utils.airflow_utils import AirflowVars, check_variables
+from observatory.platform.utils.gc_utils import (bigquery_partitioned_table_id,
+                                                 copy_bigquery_table,
+                                                 create_bigquery_dataset,
+                                                 create_bigquery_table_from_query,
+                                                 create_bigquery_view,
+                                                 run_bigquery_query)
+from observatory.platform.utils.jinja2_utils import make_sql_jinja2_filename, render_template
+from pendulum import Pendulum
 
 
 def set_task_state(success: bool, task_id: str):
@@ -59,25 +61,22 @@ def select_table_suffixes(project_id: str, dataset_id: str, table_id: str, end_d
     """
 
     template_path = os.path.join(workflow_sql_templates_path(), make_sql_jinja2_filename('select_table_suffixes'))
-    query = render_template(template_path,
-                            project_id=project_id,
-                            dataset_id=dataset_id,
-                            table_id=table_id,
-                            end_date=end_date.strftime('%Y-%m-%d'),
-                            limit=limit)
+    query = render_template(template_path, project_id=project_id, dataset_id=dataset_id, table_id=table_id,
+                            end_date=end_date.strftime('%Y-%m-%d'), limit=limit)
     rows = run_bigquery_query(query)
     suffixes = [row['suffix'] for row in rows]
     return suffixes
 
 
-def create_aggregate_table(project_id: str, release_date: Pendulum, aggregation_field: str, group_by_time_field: str, table_id: str,
-                           data_location: str, task_id: str):
+def create_aggregate_table(project_id: str, release_date: Pendulum, aggregation_field: str, group_by_time_field: str,
+                           table_id: str, data_location: str, task_id: str):
     """ Runs the aggregate table query.
 
     :param project_id: the Google Cloud project id.
     :param release_date: the release date of the release.
     :param aggregation_field: the field to aggregate on, e.g. institution, publisher etc.
-    :group_by_time_field: either published_year or published_year_month depending on the granularity required for the time dimension
+    :group_by_time_field: either published_year or published_year_month depending on the granularity required for the
+    time dimension
     :param table_id: the table id.
     :param data_location: the location for the table.
     :param task_id: the Airflow task id (for printing messages).
@@ -86,19 +85,13 @@ def create_aggregate_table(project_id: str, release_date: Pendulum, aggregation_
 
     # Create processed dataset
     template_path = os.path.join(workflow_sql_templates_path(), DoiWorkflow.AGGREGATE_DOI_FILENAME)
-    sql = render_template(template_path,
-                          project_id=project_id,
-                          release_date=release_date,
-                          aggregation_field=aggregation_field,
-                          group_by_time_field=group_by_time_field)
+    sql = render_template(template_path, project_id=project_id, release_date=release_date,
+                          aggregation_field=aggregation_field, group_by_time_field=group_by_time_field)
 
     processed_table_id = bigquery_partitioned_table_id(table_id, release_date)
-    success = create_bigquery_table_from_query(sql=sql,
-                                               project_id=project_id,
+    success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                dataset_id=DoiWorkflow.OBSERVATORY_DATASET_ID,
-                                               table_id=processed_table_id,
-                                               location=data_location,
-                                               cluster=True,
+                                               table_id=processed_table_id, location=data_location, cluster=True,
                                                clustering_fields=['id'])
 
     set_task_state(success, task_id)
@@ -149,9 +142,8 @@ def check_dependencies(**kwargs):
         :return: None.
         """
 
-        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID,
-                                     AirflowVars.DATA_LOCATION, AirflowVars.DOWNLOAD_BUCKET,
-                                     AirflowVars.TRANSFORM_BUCKET)
+        vars_valid = check_variables(AirflowVars.DATA_PATH, AirflowVars.PROJECT_ID, AirflowVars.DATA_LOCATION,
+                                     AirflowVars.DOWNLOAD_BUCKET, AirflowVars.TRANSFORM_BUCKET)
 
         if not vars_valid:
             raise AirflowException('Required variables are missing')
@@ -199,16 +191,12 @@ def extend_grid(**kwargs):
         # Create processed table
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_EXTEND_GRID))
-        sql = render_template(template_path,
-                              project_id=project_id,
-                              grid_release_date=grid_release_date)
+        sql = render_template(template_path, project_id=project_id, grid_release_date=grid_release_date)
 
         processed_table_id = bigquery_partitioned_table_id('grid_extended', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_EXTEND_GRID)
 
@@ -234,11 +222,9 @@ def aggregate_crossref_events(**kwargs):
         # TODO: perhaps only include records up until the end date of this query?
 
         processed_table_id = bigquery_partitioned_table_id('crossref_events', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_CROSSREF_EVENTS)
 
@@ -263,11 +249,9 @@ def aggregate_orcid(**kwargs):
         sql = render_template(template_path, project_id=project_id)
 
         processed_table_id = bigquery_partitioned_table_id('orcid', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_ORCID)
 
@@ -297,16 +281,12 @@ def aggregate_mag(**kwargs):
         # Create processed table
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_AGGREGATE_MAG))
-        sql = render_template(template_path,
-                              project_id=project_id,
-                              release_date=mag_release_date)
+        sql = render_template(template_path, project_id=project_id, release_date=mag_release_date)
 
         processed_table_id = bigquery_partitioned_table_id(MagTelescope.DAG_ID, release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_MAG)
 
@@ -336,16 +316,12 @@ def aggregate_unpaywall(**kwargs):
         # Create processed table
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_AGGREGATE_UNPAYWALL))
-        sql = render_template(template_path,
-                              project_id=project_id,
-                              release_date=unpaywall_release_date)
+        sql = render_template(template_path, project_id=project_id, release_date=unpaywall_release_date)
 
         processed_table_id = bigquery_partitioned_table_id(UnpaywallTelescope.DAG_ID, release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_UNPAYWALL)
 
@@ -365,8 +341,8 @@ def extend_crossref_funders(**kwargs):
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
 
         # Get last Funref and Crossref Metadata release dates before current end date
-        fundref_release_date = select_table_suffixes(project_id, FundrefTelescope.DATASET_ID,
-                                                     FundrefTelescope.DAG_ID, release_date)
+        fundref_release_date = select_table_suffixes(project_id, FundrefTelescope.DATASET_ID, FundrefTelescope.DAG_ID,
+                                                     release_date)
         crossref_metadata_release_date = select_table_suffixes(project_id, CrossrefMetadataTelescope.DATASET_ID,
                                                                CrossrefMetadataTelescope.DAG_ID, release_date)
         if len(fundref_release_date) and len(crossref_metadata_release_date):
@@ -379,17 +355,14 @@ def extend_crossref_funders(**kwargs):
         # Create processed table
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_EXTEND_CROSSREF_FUNDERS))
-        sql = render_template(template_path,
-                              project_id=project_id,
+        sql = render_template(template_path, project_id=project_id,
                               crossref_metadata_release_date=crossref_metadata_release_date,
                               fundref_release_date=fundref_release_date)
 
         processed_table_id = bigquery_partitioned_table_id('crossref_funders_extended', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_EXTEND_CROSSREF_FUNDERS)
 
@@ -409,8 +382,8 @@ def aggregate_open_citations(**kwargs):
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
 
         # Get last Open Citations release date before current end date
-        open_citations_release_date = select_table_suffixes(project_id, 'open_citations',
-                                                            'open_citations', release_date)
+        open_citations_release_date = select_table_suffixes(project_id, 'open_citations', 'open_citations',
+                                                            release_date)
         if len(open_citations_release_date):
             open_citations_release_date = open_citations_release_date[0]
         else:
@@ -419,16 +392,12 @@ def aggregate_open_citations(**kwargs):
         # Create processed dataset
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_AGGREGATE_OPEN_CITATIONS))
-        sql = render_template(template_path,
-                              project_id=project_id,
-                              release_date=open_citations_release_date)
+        sql = render_template(template_path, project_id=project_id, release_date=open_citations_release_date)
 
         processed_table_id = bigquery_partitioned_table_id('open_citations', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_OPEN_CITATIONS)
 
@@ -450,16 +419,13 @@ def aggregate_wos(**kwargs):
         # Create
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_AGGREGATE_WOS))
-        sql = render_template(template_path,
-                              project_id=project_id)
+        sql = render_template(template_path, project_id=project_id)
         # TODO: only include records up until the end date
 
         processed_table_id = bigquery_partitioned_table_id('wos', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_WOS)
 
@@ -481,15 +447,12 @@ def aggregate_scopus(**kwargs):
         # Create processed dataset
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_AGGREGATE_SCOPUS))
-        sql = render_template(template_path,
-                              project_id=project_id)
+        sql = render_template(template_path, project_id=project_id)
 
         processed_table_id = bigquery_partitioned_table_id('scopus', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_AGGREGATE_SCOPUS)
 
@@ -519,18 +482,13 @@ def create_doi(**kwargs):
         # Create processed dataset
         template_path = os.path.join(workflow_sql_templates_path(),
                                      make_sql_jinja2_filename(DoiWorkflow.TASK_ID_CREATE_DOI))
-        sql = render_template(template_path,
-                              project_id=project_id,
-                              dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
-                              release_date=release_date,
-                              crossref_metadata_release_date=crossref_metadata_release_date)
+        sql = render_template(template_path, project_id=project_id, dataset_id=DoiWorkflow.PROCESSED_DATASET_ID,
+                              release_date=release_date, crossref_metadata_release_date=crossref_metadata_release_date)
 
         processed_table_id = bigquery_partitioned_table_id('doi', release_date)
-        success = create_bigquery_table_from_query(sql=sql,
-                                                   project_id=project_id,
+        success = create_bigquery_table_from_query(sql=sql, project_id=project_id,
                                                    dataset_id=DoiWorkflow.OBSERVATORY_DATASET_ID,
-                                                   table_id=processed_table_id,
-                                                   location=data_location)
+                                                   table_id=processed_table_id, location=data_location)
 
         set_task_state(success, DoiWorkflow.TASK_ID_CREATE_DOI)
 
@@ -549,12 +507,13 @@ def create_country(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'countries'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'country'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_COUNTRY)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_COUNTRY)
 
     @staticmethod
     def create_funder(**kwargs):
@@ -571,12 +530,13 @@ def create_funder(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'funders'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'funder'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_FUNDER)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_FUNDER)
 
     @staticmethod
     def create_group(**kwargs):
@@ -593,12 +553,13 @@ def create_group(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'groupings'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'group'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_GROUP)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_GROUP)
 
     @staticmethod
     def create_institution(**kwargs):
@@ -615,13 +576,14 @@ def create_institution(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'institutions'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'institution'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_INSTITUTION)
-    
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_INSTITUTION)
+
     @staticmethod
     def create_author(**kwargs):
         """ Create author snapshot.
@@ -637,12 +599,13 @@ def create_author(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'authors'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'author'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_AUTHOR)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_AUTHOR)
 
     @staticmethod
     def create_journal(**kwargs):
@@ -659,12 +622,13 @@ def create_journal(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'journals'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'journal'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_JOURNAL)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_JOURNAL)
 
     @staticmethod
     def create_publisher(**kwargs):
@@ -681,12 +645,13 @@ def create_publisher(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'publishers'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'publisher'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_PUBLISHER)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_PUBLISHER)
 
     @staticmethod
     def create_region(**kwargs):
@@ -703,12 +668,13 @@ def create_region(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'regions'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'region'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_REGION)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_REGION)
 
     @staticmethod
     def create_subregion(**kwargs):
@@ -725,12 +691,13 @@ def create_subregion(**kwargs):
         data_location = Variable.get(AirflowVars.DATA_LOCATION)
         release_date = kwargs['next_execution_date'].subtract(microseconds=1).date()
         aggregation_field = 'subregions'
-        group_by_time_field="published_year"
+        group_by_time_field = "published_year"
         table_id = 'subregion'
 
         # Aggregate
-        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field, group_by_time_field=group_by_time_field, 
-                                table_id=table_id, data_location=data_location, task_id=DoiWorkflow.TASK_ID_CREATE_SUBREGION)
+        create_aggregate_table(project_id=project_id, release_date=release_date, aggregation_field=aggregation_field,
+                               group_by_time_field=group_by_time_field, table_id=table_id, data_location=data_location,
+                               task_id=DoiWorkflow.TASK_ID_CREATE_SUBREGION)
 
     @staticmethod
     def copy_tables(**kwargs):
@@ -768,8 +735,5 @@ def create_views(**kwargs):
         # Create views
         for table_name in table_names:
             view_name = f'{table_name}_comparison'
-            query = render_template(template_path,
-                                    project_id=project_id,
-                                    dataset_id=dataset_id,
-                                    table_id=table_name)
+            query = render_template(template_path, project_id=project_id, dataset_id=dataset_id, table_id=table_name)
             create_bigquery_view(project_id, dataset_id, view_name, query)
diff --git a/observatory-platform/observatory/platform/dags/load_dags.py b/observatory-platform/observatory/platform/dags/load_dags.py
index bd601eead..e0f5811fb 100644
--- a/observatory-platform/observatory/platform/dags/load_dags.py
+++ b/observatory-platform/observatory/platform/dags/load_dags.py
@@ -18,8 +18,8 @@
 import logging
 
 from airflow.models import DagBag, Variable
-
-from observatory.platform.utils.config_utils import module_file_path, AirflowVars
+from observatory.platform.utils.airflow_utils import AirflowVars
+from observatory.platform.utils.config_utils import module_file_path
 
 
 def load_dag_bag(path: str) -> None:
diff --git a/observatory-platform/observatory/platform/observatory_config.py b/observatory-platform/observatory/platform/observatory_config.py
index 262659488..cdfc83df6 100644
--- a/observatory-platform/observatory/platform/observatory_config.py
+++ b/observatory-platform/observatory/platform/observatory_config.py
@@ -22,16 +22,15 @@
 import os
 from dataclasses import dataclass, field
 from enum import Enum
-from typing import Dict, Tuple, List, ClassVar, Any, Union
+from typing import Any, ClassVar, Dict, List, Tuple, Union
 
 import cerberus.validator
 import yaml
 from airflow.configuration import generate_fernet_key
 from cerberus import Validator
-
 from observatory.platform.terraform_api import TerraformVariable
-from observatory.platform.utils.airflow_utils import AirflowVariable
-from observatory.platform.utils.config_utils import AirflowVars, module_file_path
+from observatory.platform.utils.airflow_utils import AirflowVariable, AirflowVars
+from observatory.platform.utils.config_utils import module_file_path
 from observatory.platform.utils.jinja2_utils import render_template
 
 
@@ -153,9 +152,7 @@ def from_dict(dict_: Dict) -> Airflow:
         secret_key = dict_.get('secret_key')
         ui_user_password = dict_.get('ui_user_password')
         ui_user_email = dict_.get('ui_user_email')
-        return Airflow(fernet_key, secret_key,
-                       ui_user_password=ui_user_password,
-                       ui_user_email=ui_user_email)
+        return Airflow(fernet_key, secret_key, ui_user_password=ui_user_password, ui_user_email=ui_user_email)
 
 
 @dataclass
@@ -225,13 +222,8 @@ def from_dict(dict_: Dict) -> GoogleCloud:
         data_location = dict_.get('data_location')
         buckets = CloudStorageBucket.parse_buckets(dict_.get('buckets', dict()))
 
-        return GoogleCloud(
-            project_id=project_id,
-            credentials=credentials,
-            region=region,
-            zone=zone,
-            data_location=data_location,
-            buckets=buckets)
+        return GoogleCloud(project_id=project_id, credentials=credentials, region=region, zone=zone,
+            data_location=data_location, buckets=buckets)
 
 
 def parse_dict_to_list(dict_: Dict, cls: ClassVar) -> List[Any]:
@@ -513,14 +505,9 @@ class ValidationError:
 
 class ObservatoryConfig:
 
-    def __init__(self,
-                 backend: Backend = None,
-                 airflow: Airflow = None,
-                 google_cloud: GoogleCloud = None,
-                 terraform: Terraform = None,
-                 airflow_variables: List[AirflowVariable] = None,
-                 airflow_connections: List[AirflowConnection] = None,
-                 dags_projects: List[DagsProject] = None,
+    def __init__(self, backend: Backend = None, airflow: Airflow = None, google_cloud: GoogleCloud = None,
+                 terraform: Terraform = None, airflow_variables: List[AirflowVariable] = None,
+                 airflow_connections: List[AirflowConnection] = None, dags_projects: List[DagsProject] = None,
                  validator: ObservatoryConfigValidator = None):
         """ Create an ObservatoryConfig instance.
 
@@ -587,9 +574,7 @@ def make_airflow_variables(self) -> List[AirflowVariable]:
         """
 
         # Create airflow variables from fixed config file values
-        variables = [
-            AirflowVariable(AirflowVars.ENVIRONMENT, self.backend.environment.value)
-        ]
+        variables = [AirflowVariable(AirflowVars.ENVIRONMENT, self.backend.environment.value)]
 
         if self.google_cloud.project_id is not None:
             variables.append(AirflowVariable(AirflowVars.PROJECT_ID, self.google_cloud.project_id))
@@ -610,9 +595,8 @@ def make_airflow_variables(self) -> List[AirflowVariable]:
         return variables
 
     @staticmethod
-    def _parse_fields(dict_: Dict) -> Tuple[Backend, Airflow, GoogleCloud, Terraform,
-                                            List[AirflowVariable], List[AirflowConnection],
-                                            List[DagsProject]]:
+    def _parse_fields(dict_: Dict) -> Tuple[
+        Backend, Airflow, GoogleCloud, Terraform, List[AirflowVariable], List[AirflowConnection], List[DagsProject]]:
         backend = Backend.from_dict(dict_.get('backend', dict()))
         airflow = Airflow.from_dict(dict_.get('airflow', dict()))
         google_cloud = GoogleCloud.from_dict(dict_.get('google_cloud', dict()))
@@ -652,9 +636,7 @@ def _save_default(config_path: str, template_file_name: str):
         except ModuleNotFoundError:
             observatory_dags_path = '/path/to/observatory-platform/observatory-dags'
 
-        render = render_template(template_path,
-                                 fernet_key=fernet_key,
-                                 secret_key=secret_key,
+        render = render_template(template_path, fernet_key=fernet_key, secret_key=secret_key,
                                  observatory_dags_path=observatory_dags_path)
 
         # Save file
@@ -677,17 +659,12 @@ def from_dict(cls, dict_: Dict) -> ObservatoryConfig:
         is_valid = validator.validate(dict_, schema)
 
         if is_valid:
-            (backend, airflow, google_cloud, terraform,
-             airflow_variables, airflow_connections, dags_projects) = ObservatoryConfig._parse_fields(dict_)
-
-            return ObservatoryConfig(backend,
-                                     airflow,
-                                     google_cloud=google_cloud,
-                                     terraform=terraform,
-                                     airflow_variables=airflow_variables,
-                                     airflow_connections=airflow_connections,
-                                     dags_projects=dags_projects,
-                                     validator=validator)
+            (backend, airflow, google_cloud, terraform, airflow_variables, airflow_connections,
+             dags_projects) = ObservatoryConfig._parse_fields(dict_)
+
+            return ObservatoryConfig(backend, airflow, google_cloud=google_cloud, terraform=terraform,
+                                     airflow_variables=airflow_variables, airflow_connections=airflow_connections,
+                                     dags_projects=dags_projects, validator=validator)
         else:
             return ObservatoryConfig(validator=validator)
 
@@ -716,18 +693,11 @@ def load(cls, path: str):
 class TerraformConfig(ObservatoryConfig):
     WORKSPACE_PREFIX = "observatory-"
 
-    def __init__(self,
-                 backend: Backend = None,
-                 airflow: Airflow = None,
-                 google_cloud: GoogleCloud = None,
-                 terraform: Terraform = None,
-                 airflow_variables: List[AirflowVariable] = None,
-                 airflow_connections: List[AirflowConnection] = None,
-                 dags_projects: List[DagsProject] = None,
-                 cloud_sql_database: CloudSqlDatabase = None,
-                 airflow_main_vm: VirtualMachine = None,
-                 airflow_worker_vm: VirtualMachine = None,
-                 validator: ObservatoryConfigValidator = None):
+    def __init__(self, backend: Backend = None, airflow: Airflow = None, google_cloud: GoogleCloud = None,
+                 terraform: Terraform = None, airflow_variables: List[AirflowVariable] = None,
+                 airflow_connections: List[AirflowConnection] = None, dags_projects: List[DagsProject] = None,
+                 cloud_sql_database: CloudSqlDatabase = None, airflow_main_vm: VirtualMachine = None,
+                 airflow_worker_vm: VirtualMachine = None, validator: ObservatoryConfigValidator = None):
         """ Create a TerraformConfig instance.
 
         :param backend: the backend config.
@@ -743,14 +713,9 @@ def __init__(self,
         :param validator: an ObservatoryConfigValidator instance.
         """
 
-        super().__init__(backend=backend,
-                         airflow=airflow,
-                         google_cloud=google_cloud,
-                         terraform=terraform,
-                         airflow_variables=airflow_variables,
-                         airflow_connections=airflow_connections,
-                         dags_projects=dags_projects,
-                         validator=validator)
+        super().__init__(backend=backend, airflow=airflow, google_cloud=google_cloud, terraform=terraform,
+                         airflow_variables=airflow_variables, airflow_connections=airflow_connections,
+                         dags_projects=dags_projects, validator=validator)
         self.cloud_sql_database = cloud_sql_database
         self.airflow_main_vm = airflow_main_vm
         self.airflow_worker_vm = airflow_worker_vm
@@ -771,9 +736,7 @@ def make_airflow_variables(self) -> List[AirflowVariable]:
         """
 
         # Create airflow variables from fixed config file values
-        variables = [
-            AirflowVariable(AirflowVars.ENVIRONMENT, self.backend.environment.value)
-        ]
+        variables = [AirflowVariable(AirflowVars.ENVIRONMENT, self.backend.environment.value)]
 
         if self.google_cloud.project_id is not None:
             variables.append(AirflowVariable(AirflowVars.PROJECT_ID, self.google_cloud.project_id))
@@ -803,8 +766,7 @@ def terraform_variables(self) -> List[TerraformVariable]:
                                   hcl=True),
                 TerraformVariable('airflow_main_vm', self.airflow_main_vm.to_hcl(), hcl=True),
                 TerraformVariable('airflow_worker_vm', self.airflow_worker_vm.to_hcl(), hcl=True),
-                TerraformVariable('airflow_variables', list_to_hcl(self.airflow_variables), hcl=True,
-                                  sensitive=False),
+                TerraformVariable('airflow_variables', list_to_hcl(self.airflow_variables), hcl=True, sensitive=False),
                 TerraformVariable('airflow_connections', list_to_hcl(self.airflow_connections), hcl=True,
                                   sensitive=sensitive)]
 
@@ -824,23 +786,17 @@ def from_dict(cls, dict_: Dict) -> TerraformConfig:
         is_valid = validator.validate(dict_, schema)
 
         if is_valid:
-            (backend, airflow, google_cloud, terraform,
-             airflow_variables, airflow_connections, dags_projects) = ObservatoryConfig._parse_fields(dict_)
+            (backend, airflow, google_cloud, terraform, airflow_variables, airflow_connections,
+             dags_projects) = ObservatoryConfig._parse_fields(dict_)
 
             cloud_sql_database = CloudSqlDatabase.from_dict(dict_.get('cloud_sql_database', dict()))
             airflow_main_vm = VirtualMachine.from_dict(dict_.get('airflow_main_vm', dict()))
             airflow_worker_vm = VirtualMachine.from_dict(dict_.get('airflow_worker_vm', dict()))
 
-            return TerraformConfig(backend,
-                                   airflow,
-                                   google_cloud=google_cloud,
-                                   terraform=terraform,
-                                   airflow_variables=airflow_variables,
-                                   airflow_connections=airflow_connections,
-                                   dags_projects=dags_projects,
-                                   cloud_sql_database=cloud_sql_database,
-                                   airflow_main_vm=airflow_main_vm,
-                                   airflow_worker_vm=airflow_worker_vm,
+            return TerraformConfig(backend, airflow, google_cloud=google_cloud, terraform=terraform,
+                                   airflow_variables=airflow_variables, airflow_connections=airflow_connections,
+                                   dags_projects=dags_projects, cloud_sql_database=cloud_sql_database,
+                                   airflow_main_vm=airflow_main_vm, airflow_worker_vm=airflow_worker_vm,
                                    validator=validator)
         else:
             return TerraformConfig(validator=validator)
diff --git a/observatory-platform/observatory/platform/tasks/terraform.py b/observatory-platform/observatory/platform/tasks/terraform.py
index 06a02b732..bf9c9465b 100644
--- a/observatory-platform/observatory/platform/tasks/terraform.py
+++ b/observatory-platform/observatory/platform/tasks/terraform.py
@@ -18,7 +18,6 @@
 from datetime import datetime
 from typing import Union
 
-from airflow.contrib.hooks.slack_webhook_hook import SlackWebhookHook
 from airflow.exceptions import AirflowException
 from airflow.hooks.base_hook import BaseHook
 from airflow.models.dag import DAG
@@ -26,12 +25,16 @@
 from airflow.models.dagrun import DagRun
 from airflow.models.taskinstance import TaskInstance
 from croniter import croniter
-
 from observatory.platform.observatory_config import TerraformConfig
 from observatory.platform.observatory_config import VirtualMachine
 from observatory.platform.terraform_api import TerraformApi
-from observatory.platform.utils.airflow_utils import AirflowVariable as Variable, change_task_log_level
-from observatory.platform.utils.config_utils import (check_variables, check_connections, AirflowVars, AirflowConns)
+from observatory.platform.utils.airflow_utils import AirflowConns, \
+    AirflowVariable, \
+    AirflowVars, \
+    change_task_log_level, \
+    check_connections, \
+    check_variables, \
+    create_slack_webhook
 
 
 def get_workspace_id() -> str:
@@ -43,10 +46,10 @@ def get_workspace_id() -> str:
     terraform_api = TerraformApi(token)
 
     # Get organization
-    organization = Variable.get(AirflowVars.TERRAFORM_ORGANIZATION)
+    organization = AirflowVariable.get(AirflowVars.TERRAFORM_ORGANIZATION)
 
     # Get workspace
-    environment = Variable.get(AirflowVars.ENVIRONMENT)
+    environment = AirflowVariable.get(AirflowVars.ENVIRONMENT)
     workspace = TerraformConfig.WORKSPACE_PREFIX + environment
 
     # Get workspace ID
@@ -55,36 +58,6 @@ def get_workspace_id() -> str:
     return workspace_id
 
 
-def create_slack_webhook(comments: str = "", **kwargs) -> SlackWebhookHook:
-    """ Creates a slack webhook using the token in the slack airflow connection.
-
-    :param comments: Additional comments in slack message
-    :param kwargs: the context passed from the PythonOperator. See
-    https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed to
-    this  argument.
-    :return: slack webhook
-    """
-
-    ti: TaskInstance = kwargs['ti']
-
-    # Get project id
-    project_id = Variable.get(AirflowVars.PROJECT_ID)
-
-    message = """
-    :red_circle: Task Alert.
-    *Task*: {task}
-    *Dag*: {dag}
-    *Execution Time*: {exec_date}
-    *Log Url*: {log_url}
-    *Project id*: {project_id}
-    *Comments*: {comments}
-    """.format(task=ti.task_id, dag=ti.dag_id, ti=ti, exec_date=kwargs['execution_date'], log_url=ti.log_url,
-               comments=comments, project_id=project_id)
-    slack_conn = BaseHook.get_connection(AirflowConns.SLACK)
-    slack_hook = SlackWebhookHook(http_conn_id=slack_conn.conn_id, webhook_token=slack_conn.password, message=message)
-    return slack_hook
-
-
 def get_last_execution_prev(dag: DAG, dag_id: str, prev_start_time_vm: Union[datetime, None]) -> Union[datetime, None]:
     """ Find the execution date of the last DAG run before the previous time the VM was turned on.
     If there aren't any DAG runs before this time or the time is None (first/second time turning off VM) the
@@ -198,7 +171,8 @@ def check_dependencies():
 
     @staticmethod
     def get_variable_create(**kwargs) -> bool:
-        """ Retrieves the current value of the terraform variable from the cloud workspace, if this already has the value
+        """ Retrieves the current value of the terraform variable from the cloud workspace, if this already has the
+        value
         that the DAG is planning to set it to, it will return False and remaining tasks are skipped.
 
         :param kwargs: the context passed from the PythonOperator. See
@@ -225,8 +199,8 @@ def get_variable_create(**kwargs) -> bool:
         print(f"DAG RUN: {kwargs['dag_run'].dag_id}")
         print(f"VM state: {vm_is_on}")
 
-        if (kwargs["dag_run"].dag_id == TerraformTasks.DAG_ID_CREATE_VM and vm_is_on) or \
-                (kwargs["dag_run"].dag_id == TerraformTasks.DAG_ID_DESTROY_VM and not vm_is_on):
+        if (kwargs["dag_run"].dag_id == TerraformTasks.DAG_ID_CREATE_VM and vm_is_on) or (
+                kwargs["dag_run"].dag_id == TerraformTasks.DAG_ID_DESTROY_VM and not vm_is_on):
             logging.info(f'VM is already in this state: {"on" if vm_is_on else "off"}')
             return False
 
@@ -319,6 +293,7 @@ def check_terraform_run_status(**kwargs):
 
         ti: TaskInstance = kwargs['ti']
         run_id = ti.xcom_pull(key=TerraformTasks.XCOM_TERRAFORM_RUN_ID, task_ids=TerraformTasks.TASK_ID_RUN)
+        project_id = AirflowVariable.get(AirflowVars.PROJECT_ID)
 
         token = BaseHook.get_connection(AirflowConns.TERRAFORM).password
         terraform_api = TerraformApi(token)
@@ -332,7 +307,7 @@ def check_terraform_run_status(**kwargs):
         logging.info(f'Run status: {run_status}')
         comments = f'Terraform run status: {run_status}'
         logging.info(f'Sending slack notification: "{comments}"')
-        slack_hook = create_slack_webhook(comments, **kwargs)
+        slack_hook = create_slack_webhook(comments, project_id, **kwargs)
         slack_hook.execute()
 
     @staticmethod
@@ -350,13 +325,11 @@ def check_success_dags(**kwargs) -> str:
 
         ti: TaskInstance = kwargs['ti']
         prev_start_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_PREV_START_TIME_VM,
-                                          task_ids=TerraformTasks.TASK_ID_RUN,
-                                          dag_id=TerraformTasks.DAG_ID_CREATE_VM, include_prior_dates=True)
-        start_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_START_TIME_VM,
-                                     task_ids=TerraformTasks.TASK_ID_RUN,
+                                          task_ids=TerraformTasks.TASK_ID_RUN, dag_id=TerraformTasks.DAG_ID_CREATE_VM,
+                                          include_prior_dates=True)
+        start_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_START_TIME_VM, task_ids=TerraformTasks.TASK_ID_RUN,
                                      dag_id=TerraformTasks.DAG_ID_CREATE_VM, include_prior_dates=True)
-        destroy_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_DESTROY_TIME_VM,
-                                       task_ids=TerraformTasks.TASK_ID_RUN,
+        destroy_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_DESTROY_TIME_VM, task_ids=TerraformTasks.TASK_ID_RUN,
                                        dag_id=TerraformTasks.DAG_ID_DESTROY_VM, include_prior_dates=True)
         logging.info(f'prev_start_time_vm: {prev_start_time_vm}, start_time_vm: {start_time_vm}, '
                      f'destroy_time_vm: {destroy_time_vm}\n')
@@ -428,8 +401,7 @@ def check_runtime_vm(**kwargs):
         last_warning_time = ti.xcom_pull(key=TerraformTasks.XCOM_WARNING_TIME,
                                          task_ids=TerraformTasks.TASK_ID_VM_RUNTIME,
                                          dag_id=TerraformTasks.DAG_ID_DESTROY_VM, include_prior_dates=True)
-        start_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_START_TIME_VM,
-                                     task_ids=TerraformTasks.TASK_ID_RUN,
+        start_time_vm = ti.xcom_pull(key=TerraformTasks.XCOM_START_TIME_VM, task_ids=TerraformTasks.TASK_ID_RUN,
                                      dag_id=TerraformTasks.DAG_ID_CREATE_VM, include_prior_dates=True)
 
         if start_time_vm:
@@ -448,9 +420,11 @@ def check_runtime_vm(**kwargs):
             if hours_on > TerraformTasks.VM_RUNTIME_H_WARNING:
                 #  check if no warning was sent before or last time was longer ago than warning frequency
                 if not hours_since_warning or hours_since_warning > TerraformTasks.WARNING_FREQUENCY_H:
-                    comments = f'Worker VM has been on since {start_time_vm}. No. hours passed since then: {hours_on}.' \
+                    comments = f'Worker VM has been on since {start_time_vm}. No. hours passed since then: ' \
+                               f'{hours_on}.' \
                                f' Warning limit: {TerraformTasks.VM_RUNTIME_H_WARNING}H'
-                    slack_hook = create_slack_webhook(comments, **kwargs)
+                    project_id = AirflowVariable.get(AirflowVars.PROJECT_ID)
+                    slack_hook = create_slack_webhook(comments, project_id, **kwargs)
 
                     # http_hook outputs the secret token, suppressing logging 'info' by setting level to 'warning'
                     old_levels = change_task_log_level(logging.WARNING)
diff --git a/observatory-platform/observatory/platform/telescopes/__init__.py b/observatory-platform/observatory/platform/telescopes/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/observatory-platform/observatory/platform/telescopes/snapshot_telescope.py b/observatory-platform/observatory/platform/telescopes/snapshot_telescope.py
new file mode 100644
index 000000000..9cc912ebe
--- /dev/null
+++ b/observatory-platform/observatory/platform/telescopes/snapshot_telescope.py
@@ -0,0 +1,101 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs, James Diprose, Tuan Chien
+
+import datetime
+from typing import List
+
+import pendulum
+from observatory.platform.telescopes.telescope import Release, Telescope
+from observatory.platform.utils.airflow_utils import AirflowVars
+from observatory.platform.utils.template_utils import blob_name, \
+    bq_load_shard, \
+    table_ids_from_path, \
+    upload_files_from_list
+
+
+class SnapshotRelease(Release):
+    def __init__(self, dag_id: str, release_date: pendulum.Pendulum, download_files_regex: str = None,
+                 extract_files_regex: str = None, transform_files_regex: str = None):
+        """ Construct a SnapshotRelease instance
+        :param dag_id: the id of the DAG.
+        :param release_date: the release date (used to construct release_id).
+        :param download_files_regex: regex pattern that is used to find files in download folder
+        :param extract_files_regex: regex pattern that is used to find files in extract folder
+        :param transform_files_regex: regex pattern that is used to find files in transform folder
+        """
+        self.release_date = release_date
+        release_id = f'{self.dag_id}_{self.release_date.strftime("%Y_%m_%d")}'
+        super().__init__(dag_id, release_id, download_files_regex, extract_files_regex, transform_files_regex)
+
+
+class SnapshotTelescope(Telescope):
+    def __init__(self, dag_id: str, start_date: datetime, schedule_interval: str, dataset_id: str, catchup: bool = True,
+                 queue: str = 'default', max_retries: int = 3, max_active_runs: int = 1, schema_prefix: str = '',
+                 schema_version: str = None, airflow_vars: list = None, airflow_conns: list = None):
+        """ Construct a SnapshotTelescope instance.
+        :param dag_id: the id of the DAG.
+        :param start_date: the start date of the DAG.
+        :param schedule_interval: the schedule interval of the DAG.
+        :param dataset_id: the dataset id.
+        :param catchup: whether to catchup the DAG or not.
+        :param queue: the Airflow queue name.
+        :param max_retries: the number of times to retry each task.
+        :param max_active_runs: the maximum number of DAG runs that can be run at once.
+        :param schema_prefix: the prefix used to find the schema path
+        :param schema_version: the version used to find the schema path
+        :param airflow_vars: list of airflow variable keys, for each variable it is checked if it exists in airflow
+        :param airflow_conns: list of airflow connection keys, for each connection it is checked if it exists in airflow
+        """
+        super().__init__(dag_id, start_date, schedule_interval, catchup, queue, max_retries, max_active_runs,
+                         schema_prefix, schema_version, airflow_vars, airflow_conns)
+        self.dataset_id = dataset_id
+        # Set transform_bucket_name as required airflow variable
+        if not airflow_vars:
+            airflow_vars = []
+        self.airflow_vars = list(set([AirflowVars.TRANSFORM_BUCKET] + airflow_vars))
+        self.airflow_conns = airflow_conns
+
+    def upload_transformed(self, releases: List[SnapshotRelease], **kwargs):
+        """ Task to upload each transformed release to a google cloud bucket
+        :param releases:
+        :param kwargs:
+        :return:
+        """
+        for release in releases:
+            upload_files_from_list(release.transform_files, release.transform_bucket)
+
+    def bq_load(self, releases: List[SnapshotRelease], **kwargs):
+        """ Task to load each transformed release to BigQuery.
+        :param releases: a list of GRID releases.
+        :return: None.
+        """
+
+        # Load each transformed release
+        for release in releases:
+            for transform_path in release.transform_files:
+                transform_blob = blob_name(transform_path)
+                table_id, _ = table_ids_from_path(transform_path)
+                bq_load_shard(release.release_date, transform_blob, self.dataset_id, table_id, self.schema_prefix,
+                              self.schema_version, self.description)
+
+    def cleanup(self, releases: List[SnapshotRelease], **kwargs):
+        """ Delete files of downloaded, extracted and transformed release.
+        :param releases: the oapen metadata release
+        :return: None.
+        """
+        # Delete files from each release
+        for release in releases:
+            release.cleanup()
diff --git a/observatory-platform/observatory/platform/telescopes/stream_telescope.py b/observatory-platform/observatory/platform/telescopes/stream_telescope.py
new file mode 100644
index 000000000..b260276a3
--- /dev/null
+++ b/observatory-platform/observatory/platform/telescopes/stream_telescope.py
@@ -0,0 +1,224 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs, James Diprose, Tuan Chien
+
+import datetime
+import logging
+from datetime import timedelta
+from functools import partial
+from typing import Callable, List
+
+import pendulum
+from airflow.exceptions import AirflowSkipException
+from airflow.models.taskinstance import TaskInstance
+from airflow.operators.python_operator import PythonOperator
+from observatory.platform.telescopes.telescope import Release, Telescope
+from observatory.platform.utils.airflow_utils import AirflowVars
+from observatory.platform.utils.template_utils import blob_name, \
+    bq_append_from_file, \
+    bq_append_from_partition, \
+    bq_delete_old, \
+    bq_load_partition, \
+    table_ids_from_path, \
+    upload_files_from_list
+
+
+class StreamRelease(Release):
+    def __init__(self, dag_id: str, start_date: pendulum.Pendulum, end_date: pendulum.Pendulum,
+                 first_release: bool, download_files_regex: str = None, extract_files_regex: str = None,
+                 transform_files_regex: str = None):
+        """ Construct a StreamRelease instance
+        :param dag_id: the id of the DAG.
+        :param start_date: the start_date of the release.
+        :param end_date: the end_date of the release.
+        :param first_release: whether this is the first release that is processed for this DAG
+        :param download_files_regex: regex pattern that is used to find files in download folder
+        :param extract_files_regex: regex pattern that is used to find files in extract folder
+        :param transform_files_regex: regex pattern that is used to find files in transform folder
+        """
+        self.start_date = start_date
+        self.end_date = end_date
+        self.first_release = first_release
+        release_id = self.start_date.strftime("%Y_%m_%d") + "-" + self.end_date.strftime("%Y_%m_%d")
+        super().__init__(dag_id, release_id, download_files_regex, extract_files_regex, transform_files_regex)
+
+
+class StreamTelescope(Telescope):
+    def __init__(self, dag_id: str, start_date: datetime, schedule_interval: str, dataset_id: str,
+                 merge_partition_field: str, updated_date_field: str, bq_merge_days: int, catchup: bool = False,
+                 queue: str = 'default', max_retries: int = 3, max_active_runs: int = 1, schema_prefix: str = '',
+                 schema_version: str = None, airflow_vars: list = None, airflow_conns: list = None):
+        """ Construct a StreamTelescope instance.
+        :param dag_id: the id of the DAG.
+        :param start_date: the start date of the DAG.
+        :param schedule_interval: the schedule interval of the DAG.
+        :param dataset_id: the dataset id.
+        :param merge_partition_field: the BigQuery field used to match partitions for a merge
+        :param updated_date_field: the BigQuery field used to determine newest entry for a merge
+        :param bq_merge_days: how often partitions should be merged (every x days)
+        :param catchup: whether to catchup the DAG or not.
+        :param queue: the Airflow queue name.
+        :param max_retries: the number of times to retry each task.
+        :param max_active_runs: the maximum number of DAG runs that can be run at once.
+        :param schema_prefix: the prefix used to find the schema path
+        :param schema_version: the version used to find the schema path
+        :param airflow_vars: list of airflow variable keys, for each variable it is checked if it exists in airflow
+        :param airflow_conns: list of airflow connection keys, for each connection it is checked if it exists in airflow
+        """
+        super().__init__(dag_id, start_date, schedule_interval, catchup, queue, max_retries, max_active_runs,
+                         schema_prefix, schema_version, airflow_vars, airflow_conns)
+
+        self.dataset_id = dataset_id
+        # Set transform_bucket_name as required airflow variable
+        if not airflow_vars:
+            airflow_vars = []
+        self.airflow_vars = list(set([AirflowVars.TRANSFORM_BUCKET] + airflow_vars))
+
+        self.merge_partition_field = merge_partition_field
+        self.updated_date_field = updated_date_field
+        self.bq_merge_days = bq_merge_days
+
+    def make_operator(self, func: Callable) -> Callable:
+        """ Make a partial PythonOperator that can be attached to an airflow DAG. This PythonOperator differs from
+        the one used in the Telescope class in it's 'trigger_rule'.
+        :param func: A callable function
+        :return: The partial PythonOperator
+        """
+        operator = partial(PythonOperator, task_id=func.__name__, python_callable=partial(self.task_callable, func),
+                           queue=self.queue, trigger_rule='none_failed', default_args=self.default_args,
+                           provide_context=True)
+
+        return operator
+
+    def make_operators(self, funcs: List[Callable]) -> List[Callable]:
+        """ Make a partial PythonOperator for each function in the funcs list.
+        :param funcs: A list of callable functions
+        :return: A list of partial PythonOperators
+        """
+        operators = []
+        for func in funcs:
+            operator = self.make_operator(func)
+            operators.append(operator)
+        return operators
+
+    def get_release_info(self, **kwargs) -> bool:
+        """ Create a release instance and update the xcom value with the last start date.
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        ti: TaskInstance = kwargs['ti']
+
+        first_release = False
+        release_info = ti.xcom_pull(key=self.RELEASE_INFO, include_prior_dates=True)
+        if not release_info:
+            first_release = True
+            # set start date to the start of the DAG
+            start_date = pendulum.instance(kwargs['dag'].default_args['start_date']).start_of('day')
+        else:
+            # set start date to end date of previous DAG run
+            start_date = release_info[1]
+        end_date = pendulum.now('UTC') - timedelta(days=1)
+        logging.info(f'Start date: {start_date}, end date: {end_date}, first release: {first_release}')
+
+        ti.xcom_push(self.RELEASE_INFO, (start_date, end_date, first_release))
+        return True
+
+    def upload_transformed(self, release: StreamRelease, **kwargs):
+        """ Upload the transformed files for the given release.
+        :param release: a StreamRelease instance
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        upload_files_from_list(release.transform_files, release.transform_bucket)
+
+    def bq_load_partition(self, release: StreamRelease, **kwargs):
+        """ For each file listed in transform_files, create a time based partition in BigQuery
+        :param release: a StreamRelease instance
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        if release.first_release:
+            # The main table is the same as the partition, so no need to upload the partition as well. Especially
+            # because a first release can be relatively big in size.
+            raise AirflowSkipException('Skipped, because first release')
+
+        for transform_path in release.transform_files:
+            transform_blob = blob_name(transform_path)
+            main_table_id, partition_table_id = table_ids_from_path(transform_path)
+            date_partition_table_id = partition_table_id + f'${pendulum.today().strftime("%Y%m%d")}'
+            bq_load_partition(release.end_date, transform_blob, self.dataset_id, main_table_id, date_partition_table_id,
+                              self.schema_prefix, self.schema_version, self.description)
+
+    def bq_delete_old(self, release: StreamRelease, **kwargs):
+        """ Delete old rows from the 'main' table, based on rows that are in a partition of the 'partitions' table.
+        :param release: a StreamRelease instance
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        ti: TaskInstance = kwargs['ti']
+        if release.first_release:
+            # don't use AirflowSkipException, to ensure that task is in 'success' state, this is used below in
+            # ti.previous_start_date_success
+            logging.info('Skipped, because first release')
+            return
+
+        start_date = pendulum.instance(ti.previous_start_date_success)
+        end_date = pendulum.instance(ti.start_date)
+        if (end_date - start_date).days >= self.bq_merge_days:
+            logging.info(f'Deleting old data from main table from partitions between dates: {start_date}, {end_date}')
+            for transform_path in release.transform_files:
+                main_table_id, partition_table_id = table_ids_from_path(transform_path)
+                bq_delete_old(start_date, end_date, self.dataset_id, main_table_id, partition_table_id,
+                              self.merge_partition_field, self.updated_date_field)
+        else:
+            raise AirflowSkipException(f'Skipped, only delete old records every {self.bq_merge_days} days. '
+                                       f'Last delete was {(end_date - start_date).days} days ago')
+
+    def bq_append_new(self, release: StreamRelease, **kwargs):
+        """ Append rows to the 'main' table, based on rows that are in a partition of the 'partitions' table.
+        :param release: a StreamRelease instance
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        ti: TaskInstance = kwargs['ti']
+
+        if release.first_release:
+            for transform_path in release.transform_files:
+                transform_blob = blob_name(transform_path)
+                main_table_id, partition_table_id = table_ids_from_path(transform_path)
+                bq_append_from_file(release.end_date, transform_blob, self.dataset_id, main_table_id,
+                                    self.schema_prefix, self.schema_version, self.description)
+            return
+
+        start_date = pendulum.instance(ti.previous_start_date_success)
+        end_date = pendulum.instance(ti.start_date)
+        if (end_date - start_date).days >= self.bq_merge_days:
+            logging.info(f'Appending data to main table from partitions between dates: {start_date}, {end_date}')
+            for transform_path in release.transform_files:
+                main_table_id, partition_table_id = table_ids_from_path(transform_path)
+                bq_append_from_partition(start_date, end_date, self.dataset_id, main_table_id, partition_table_id,
+                                         self.schema_prefix, self.schema_version)
+        else:
+            raise AirflowSkipException(f'Skipped, not first release and only append new records every '
+                                       f'{self.bq_merge_days} days. Last append was'
+                                       f' {(end_date - start_date).days} days ago')
+
+    def cleanup(self, release: StreamRelease, **kwargs):
+        """ Delete downloaded, extracted and transformed files of the release.
+        :param release: a StreamRelease instance
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        release.cleanup()
diff --git a/observatory-platform/observatory/platform/telescopes/telescope.py b/observatory-platform/observatory/platform/telescopes/telescope.py
new file mode 100644
index 000000000..33693b521
--- /dev/null
+++ b/observatory-platform/observatory/platform/telescopes/telescope.py
@@ -0,0 +1,452 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs, James Diprose, Tuan Chien
+
+import datetime
+import functools
+import logging
+import shutil
+from abc import ABC, abstractmethod
+from functools import partial
+from typing import Any, Callable, List
+from typing import Union
+
+from airflow import DAG
+from airflow.exceptions import AirflowException
+from airflow.models.variable import Variable
+from airflow.operators.python_operator import PythonOperator, ShortCircuitOperator
+from airflow.utils.helpers import chain
+from observatory.platform.utils.airflow_utils import AirflowVars, check_connections, check_variables
+from observatory.platform.utils.file_utils import list_files
+from observatory.platform.utils.template_utils import SubFolder, on_failure_callback, telescope_path
+from typing_extensions import Protocol
+
+
+class ReleaseFunction(Protocol):
+    def __call__(self, release: 'AbstractRelease', **kwargs: Any) -> Any: ...
+    """ 
+    :param release: A single instance of an AbstractRelease
+    :param kwargs: the context passed from the PythonOperator. See 
+    https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed to 
+    this argument.
+    :return: Any.
+    """
+
+
+class ListReleaseFunction(Protocol):
+    def __call__(self, releases: List['AbstractRelease'], **kwargs: Any) -> Any: ...
+    """ 
+    :param releases: A list of AbstractRelease instances
+    :param kwargs: the context passed from the PythonOperator. See 
+    https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed to 
+    this argument.
+    :return: Any.
+    """
+
+
+TelescopeFunction = Union[ReleaseFunction, ListReleaseFunction]
+
+
+class AbstractTelescope(ABC):
+    @abstractmethod
+    def add_setup_task(self, func: Callable):
+        """ Add a setup task, which is used to run tasks before 'Release' objects are created, e.g. checking
+        dependencies, fetching available releases etc.
+
+        A setup task has the following properties:
+        - Has the signature 'def func(self, **kwargs) -> bool', where
+        kwargs is the context passed from the PythonOperator. See https://airflow.apache.org/docs/stable/macros-ref.html
+        for a list of the keyword arguments that are passed to this argument.
+        - Run by a ShortCircuitOperator, meaning that a setup task can stop a DAG prematurely, e.g. if there is
+        nothing to process.
+        - func Needs to return a boolean
+        :param func: the function that will be called by the ShortCircuitOperator task.
+        :return: None.
+        """
+        pass
+
+    @abstractmethod
+    def add_setup_task_chain(self, funcs: List[Callable]):
+        """ Add a list of setup tasks, which are used to run tasks before 'Release' objects are created, e.g. checking
+        dependencies, fetching available releases etc. (See add_setup_task for more info.)
+
+        :param funcs: The list of functions that will be called by the ShortCircuitOperator task.
+        :return: None.
+        """
+        pass
+
+    @abstractmethod
+    def add_task(self, func: Callable):
+        """ Add a task, which is used to process releases. A task has the following properties:
+
+        - Has one of the following signatures 'def func(self, release: Release, **kwargs)' or 'def func(self,
+        releases: List[Release], **kwargs)'
+        - kwargs is the context passed from the PythonOperator. See https://airflow.apache.org/docs/stable/macros-ref.html
+        for a list of the keyword arguments that are passed to this argument.
+        - Run by a PythonOperator.
+        :param func: the function that will be called by the PythonOperator task.
+        :return: None.
+        """
+        pass
+
+    @abstractmethod
+    def add_task_chain(self, funcs: List[Callable]):
+        """ Add a list of tasks, which are used to process releases. (See add_task for more info.)
+
+        :param funcs: The list of functions that will be called by the PythonOperator task.
+        :return: None.
+        """
+        pass
+
+    @abstractmethod
+    def task_callable(self, func: TelescopeFunction, **kwargs) -> Any:
+        """ Invoke a task callable. Creates a Release instance or Release instances and calls the given task method.
+
+        :param func: the task method.
+        :param kwargs: the context passed from the PythonOperator. See
+        https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed
+        to this argument.
+        :return: Any.
+        """
+        pass
+
+    @abstractmethod
+    def make_release(self, **kwargs) -> Union['AbstractRelease', List['AbstractRelease']]:
+        """ Make a release instance. The release is passed as an argument to the function (TelescopeFunction) that is
+        called in 'task_callable'.
+
+        :param kwargs: the context passed from the PythonOperator. See
+        https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed
+        to this argument.
+        :return: A release instance or list of release instances
+        """
+        pass
+
+    @abstractmethod
+    def make_dag(self) -> DAG:
+        """ Make an Airflow DAG for a telescope.
+
+        :return: the DAG object.
+        """
+        pass
+
+
+class Telescope(AbstractTelescope):
+    RELEASE_INFO = 'releases'
+
+    def __init__(self, dag_id: str, start_date: datetime, schedule_interval: str, catchup: bool = False,
+                 queue: str = 'default', max_retries: int = 3, max_active_runs: int = 1, schema_prefix: str = '',
+                 schema_version: str = None, airflow_vars: list = None, airflow_conns: list = None):
+        """ Construct a Telescope instance.
+        :param dag_id: the id of the DAG.
+        :param start_date: the start date of the DAG.
+        :param schedule_interval: the schedule interval of the DAG.
+        :param catchup: whether to catchup the DAG or not.
+        :param queue: the Airflow queue name.
+        :param max_retries: the number of times to retry each task.
+        :param max_active_runs: the maximum number of DAG runs that can be run at once.
+        :param schema_prefix: the prefix used to find the schema path
+        :param schema_version: the version used to find the schema path
+        :param airflow_vars: list of airflow variable keys, for each variable it is checked if it exists in airflow
+        :param airflow_conns: list of airflow connection keys, for each connection it is checked if it exists in airflow
+        """
+        self.dag_id = dag_id
+        self.start_date = start_date
+        self.schedule_interval = schedule_interval
+        self.catchup = catchup
+        self.queue = queue
+        self.max_retries = max_retries
+        self.max_active_runs = max_active_runs
+        self.schema_prefix = schema_prefix
+        self.schema_version = schema_version
+        self.airflow_vars = airflow_vars
+        self.airflow_conns = airflow_conns
+
+        self.setup_task_funcs = []
+        self.task_funcs = []
+        self.default_args = {
+            "owner": "airflow",
+            "start_date": self.start_date,
+            'on_failure_callback': on_failure_callback
+        }
+        self.description = self.__doc__
+        self.dag = DAG(dag_id=self.dag_id, schedule_interval=self.schedule_interval, default_args=self.default_args,
+                       catchup=self.catchup, max_active_runs=1, doc_md=self.__doc__)
+
+    def add_setup_task(self, func: Callable):
+        """ Add a setup task, which is used to run tasks before 'Release' objects are created, e.g. checking
+        dependencies, fetching available releases etc.
+
+        A setup task has the following properties:
+        - Has the signature 'def func(self, **kwargs) -> bool', where
+        kwargs is the context passed from the PythonOperator. See https://airflow.apache.org/docs/stable/macros-ref.html
+        for a list of the keyword arguments that are passed to this argument.
+        - Run by a ShortCircuitOperator, meaning that a setup task can stop a DAG prematurely, e.g. if there is
+        nothing to process.
+        - func Needs to return a boolean
+        :param func: the function that will be called by the ShortCircuitOperator task.
+        :return: None.
+        """
+        self.setup_task_funcs.append(func)
+
+    def add_setup_task_chain(self, funcs: List[Callable]):
+        """ Add a list of setup tasks, which are used to run tasks before 'Release' objects are created, e.g. checking
+        dependencies, fetching available releases etc. (See add_setup_task for more info.)
+
+        :param funcs: The list of functions that will be called by the ShortCircuitOperator task.
+        :return: None.
+        """
+        self.setup_task_funcs += funcs
+
+    def add_task(self, func: Callable):
+        """ Add a task, which is used to process releases. A task has the following properties:
+
+        - Has one of the following signatures 'def func(self, release: Release, **kwargs)' or 'def func(self,
+        releases: List[Release], **kwargs)'
+        - kwargs is the context passed from the PythonOperator. See https://airflow.apache.org/docs/stable/macros-ref.html
+        for a list of the keyword arguments that are passed to this argument.
+        - Run by a PythonOperator.
+        :param func: the function that will be called by the PythonOperator task.
+        :return: None.
+        """
+        self.task_funcs.append(func)
+
+    def add_task_chain(self, funcs: List[Callable]):
+        """ Add a list of tasks, which are used to process releases. (See add_task for more info.)
+
+        :param funcs: The list of functions that will be called by the PythonOperator task.
+        :return: None.
+        """
+        self.task_funcs += funcs
+
+    def task_callable(self, func: TelescopeFunction, **kwargs) -> Any:
+        """ Invoke a task callable. Creates a Release instance and calls the given task method. The result can be
+        pulled as an xcom in Airflow.
+        :param func: the task method.
+        :param kwargs: the context passed from the PythonOperator.
+        See https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed
+        to this argument.
+        :return: Any.
+        """
+
+        release = self.make_release(**kwargs)
+        result = func(release, **kwargs)
+        return result
+
+    def make_dag(self) -> DAG:
+        """ Make an Airflow DAG for a telescope.
+        :return: the DAG object.
+        """
+        tasks = []
+        with self.dag:
+            # Process setup tasks first, which are always ShortCircuitOperators
+            for func in self.setup_task_funcs:
+                if isinstance(func, functools.partial):
+                    task = func()
+                else:
+                    task = ShortCircuitOperator(task_id=func.__name__, python_callable=func, queue=self.queue,
+                                                default_args=self.default_args, provide_context=True)
+                tasks.append(task)
+
+            # Process all other tasks next, which are always PythonOperators
+            for func in self.task_funcs:
+                if isinstance(func, functools.partial):
+                    task = func()
+                else:
+                    task = PythonOperator(task_id=func.__name__, python_callable=partial(self.task_callable, func),
+                                          queue=self.queue, default_args=self.default_args, provide_context=True)
+                tasks.append(task)
+            chain(*tasks)
+
+        return self.dag
+
+    def check_dependencies(self, **kwargs) -> bool:
+        """ Checks the 'telescope' attributes, airflow variables & connections and possibly additional custom checks.
+        :param kwargs: The context passed from the PythonOperator.
+        :return: None.
+        """
+        # check that vars and connections are available
+        vars_valid = True
+        conns_valid = True
+        if self.airflow_vars:
+            vars_valid = check_variables(*self.airflow_vars)
+        if self.airflow_conns:
+            conns_valid = check_connections(*self.airflow_conns)
+
+        if not vars_valid or not conns_valid:
+            raise AirflowException('Required variables or connections are missing')
+
+        return True
+
+
+class AbstractRelease(ABC):
+    """ The abstract release interface """
+
+    @property
+    @abstractmethod
+    def download_bucket(self):
+        """ The download bucket name.
+        :return: the download bucket name.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def transform_bucket(self):
+        """ The transform bucket name.
+        :return: the transform bucket name.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def download_folder(self) -> str:
+        """ The download folder path for the release, e.g. /path/to/telescopes/download/{dag_id}/{release_id}/
+        :return: the download folder path.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def extract_folder(self) -> str:
+        """ The extract folder path for the release, e.g. /path/to/telescopes/extract/{dag_id}/{release_id}/
+        :return: the extract folder path.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def transform_folder(self) -> str:
+        """ The transform folder path for the release, e.g. /path/to/telescopes/transform/{dag_id}/{release_id}/
+        :return: the transform folder path.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def download_files(self) -> List[str]:
+        """ List all files downloaded as a part of this release.
+        :return: list of downloaded files.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def extract_files(self) -> List[str]:
+        """ List all files extracted as a part of this release.
+        :return: list of extracted files.
+        """
+        pass
+
+    @property
+    @abstractmethod
+    def transform_files(self) -> List[str]:
+        """ List all files transformed as a part of this release.
+        :return: list of transformed files.
+        """
+        pass
+
+    @abstractmethod
+    def cleanup(self) -> None:
+        """ Delete all files and folders associated with this release.
+        :return: None.
+        """
+
+        pass
+
+
+class Release(AbstractRelease):
+    """ Used to store info on a given release"""
+
+    def __init__(self, dag_id: str, release_id: str, download_files_regex: str = None, extract_files_regex: str = None,
+                 transform_files_regex: str = None):
+        """ Construct a Release instance
+        :param dag_id: the id of the DAG.
+        :param release_id: the id of the release.
+        :param download_files_regex: regex pattern that is used to find files in download folder
+        :param extract_files_regex: regex pattern that is used to find files in extract folder
+        :param transform_files_regex: regex pattern that is used to find files in transform folder
+        """
+        self.dag_id = dag_id
+        self.release_id = release_id
+        self.download_files_regex = download_files_regex
+        self.extract_files_regex = extract_files_regex
+        self.transform_files_regex = transform_files_regex
+
+    @property
+    def download_folder(self) -> str:
+        """ The download folder path for the release, e.g. /path/to/telescopes/download/{dag_id}/{release_id}/
+        :return: the download folder path.
+        """
+        return telescope_path(SubFolder.downloaded.value, self.dag_id, self.release_id)
+
+    @property
+    def extract_folder(self) -> str:
+        """ The extract folder path for the release, e.g. /path/to/telescopes/extract/{dag_id}/{release_id}/
+        :return: the extract folder path.
+        """
+        return telescope_path(SubFolder.extracted.value, self.dag_id, self.release_id)
+
+    @property
+    def transform_folder(self) -> str:
+        """ The transform folder path for the release, e.g. /path/to/telescopes/transform/{dag_id}/{release_id}/
+        :return: the transform folder path.
+        """
+        return telescope_path(SubFolder.transformed.value, self.dag_id, self.release_id)
+
+    @property
+    def download_files(self) -> List[str]:
+        """ List all files downloaded as a part of this release.
+        :return: list of downloaded files.
+        """
+        return list_files(self.download_folder, self.download_files_regex)
+
+    @property
+    def extract_files(self) -> List[str]:
+        """ List all files extracted as a part of this release.
+        :return: list of extracted files.
+        """
+        return list_files(self.extract_folder, self.extract_files_regex)
+
+    @property
+    def transform_files(self) -> List[str]:
+        """ List all files transformed as a part of this release.
+        :return: list of transformed files.
+        """
+        return list_files(self.transform_folder, self.transform_files_regex)
+
+    @property
+    def download_bucket(self):
+        """ The download bucket name.
+        :return: the download bucket name.
+        """
+        return Variable.get(AirflowVars.DOWNLOAD_BUCKET)
+
+    @property
+    def transform_bucket(self):
+        """ The transform bucket name.
+        :return: the transform bucket name.
+        """
+        return Variable.get(AirflowVars.TRANSFORM_BUCKET)
+
+    def cleanup(self) -> None:
+        """ Delete all files and folders associated with this release.
+        :return: None.
+        """
+        for path in [self.download_folder, self.extract_folder, self.transform_folder]:
+            try:
+                shutil.rmtree(path)
+            except FileNotFoundError as e:
+                logging.warning(f"No such file or directory {path}: {e}")
diff --git a/observatory-platform/observatory/platform/utils/airflow_utils.py b/observatory-platform/observatory/platform/utils/airflow_utils.py
index 30e9d9725..c05db7855 100644
--- a/observatory-platform/observatory/platform/utils/airflow_utils.py
+++ b/observatory-platform/observatory/platform/utils/airflow_utils.py
@@ -12,17 +12,51 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Author: Author: Aniek Roelofs
+# Author: Author: Aniek Roelofs, Tuan Chien
+
+"""
+Airflow utility functions (independent of telescope or google cloud usage)
+"""
 
 import json
 import logging
 from typing import Any, List, Optional, Union
 
 import airflow.secrets
-from airflow.models import Variable
+from airflow.contrib.hooks.slack_webhook_hook import SlackWebhookHook
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import Connection, TaskInstance, Variable
+from airflow.utils.db import create_session
 from google.api_core.exceptions import PermissionDenied
 
 
+class AirflowVars:
+    """ Common Airflow Variable names used with the Observatory Platform """
+
+    TEST_DATA_PATH = "test_data_path"
+    DATA_PATH = "data_path"
+    ENVIRONMENT = "environment"
+    PROJECT_ID = "project_id"
+    DATA_LOCATION = "data_location"
+    DOWNLOAD_BUCKET = "download_bucket"
+    TRANSFORM_BUCKET = "transform_bucket"
+    TERRAFORM_ORGANIZATION = "terraform_organization"
+    DAGS_MODULE_NAMES = "dags_module_names"
+    KIBANA_SPACES = "kibana_spaces"
+
+
+class AirflowConns:
+    """ Common Airflow Connection names used with the Observatory Platform """
+
+    CROSSREF = "crossref"
+    MAG_RELEASES_TABLE = "mag_releases_table"
+    MAG_SNAPSHOTS_CONTAINER = "mag_snapshots_container"
+    TERRAFORM = "terraform"
+    SLACK = "slack"
+    ELASTIC = "elastic"
+    KIBANA = "kibana"
+
+
 def get_variable(key: str) -> Optional[str]:
     """ Get Airflow Variable by iterating over all Secret Backends.
 
@@ -77,3 +111,74 @@ def change_task_log_level(new_levels: Union[List, int]) -> list:
         else:
             handler.setLevel(new_levels[count])
     return old_levels
+
+
+def list_connections(source):
+    """ Get a list of data source connections with name starting with <source>_, e.g., wos_curtin.
+
+    :param source: Data source (conforming to name convention) as a string, e.g., 'wos'.
+    :return: A list of connection id strings with the prefix <source>_, e.g., ['wos_curtin', 'wos_auckland'].
+    """
+    with create_session() as session:
+        query = session.query(Connection)
+        query = query.filter(Connection.conn_id.like(f'{source}_%'))
+        return query.all()
+
+
+def check_variables(*variables):
+    """ Checks whether all given airflow variables exist.
+
+    :param variables: name of airflow variable
+    :return: True if all variables are valid
+    """
+    is_valid = True
+    for name in variables:
+        try:
+            AirflowVariable.get(name)
+        except KeyError:
+            logging.error(f"Airflow variable '{name}' not set.")
+            is_valid = False
+    return is_valid
+
+
+def check_connections(*connections):
+    """ Checks whether all given airflow connections exist.
+
+    :param connections: name of airflow connection
+    :return: True if all connections are valid
+    """
+    is_valid = True
+    for name in connections:
+        try:
+            BaseHook.get_connection(name)
+        except KeyError:
+            logging.error(f"Airflow connection '{name}' not set.")
+            is_valid = False
+    return is_valid
+
+
+def create_slack_webhook(comments: str = "", project_id: str = "?", **kwargs) -> SlackWebhookHook:
+    """
+    Creates a slack webhook using the token in the slack airflow connection.
+    :param comments: Additional comments in slack message
+    :param project_id: The google cloud project id that will be displayed in the slack message
+    :param kwargs: the context passed from the PythonOperator. See
+    https://airflow.apache.org/docs/stable/macros-ref.html for a list of the keyword arguments that are passed to
+    this  argument.
+    :return: slack webhook
+    """
+    ti: TaskInstance = kwargs['ti']
+
+    message = """
+    :red_circle: Task Alert. 
+    *Task*: {task}  
+    *Dag*: {dag} 
+    *Execution Time*: {exec_date}  
+    *Log Url*: {log_url} 
+    *Project id*: {project_id}
+    *Comments*: {comments}
+    """.format(task=ti.task_id, dag=ti.dag_id, ti=ti, exec_date=kwargs['execution_date'], log_url=ti.log_url,
+               comments=comments, project_id=project_id)
+    slack_conn = BaseHook.get_connection(AirflowConns.SLACK)
+    slack_hook = SlackWebhookHook(http_conn_id=slack_conn.conn_id, webhook_token=slack_conn.password, message=message)
+    return slack_hook
diff --git a/observatory-platform/observatory/platform/utils/config_utils.py b/observatory-platform/observatory/platform/utils/config_utils.py
index c16817f4b..96a12f9ee 100644
--- a/observatory-platform/observatory/platform/utils/config_utils.py
+++ b/observatory-platform/observatory/platform/utils/config_utils.py
@@ -14,56 +14,21 @@
 
 # Author: James Diprose, Aniek Roelofs
 
+"""
+Utility functions for creating the config files that are used with the local development and/or terraform environment.
+"""
 
 import glob
 import importlib
 import logging
 import os
 import pathlib
-from enum import Enum
 from typing import Union
 
-import airflow
 import pendulum
-from airflow.hooks.base_hook import BaseHook
-from airflow.models import Connection
-from airflow.utils.db import create_session
 from natsort import natsorted
 from pendulum import Pendulum
 
-from observatory.platform.utils.airflow_utils import AirflowVariable
-
-# The path where data is saved on the system
-data_path = None
-test_data_path_val_ = None
-
-
-class AirflowVars:
-    """ Common Airflow Variable names used with the Observatory Platform """
-
-    TEST_DATA_PATH = "test_data_path"
-    DATA_PATH = "data_path"
-    ENVIRONMENT = "environment"
-    PROJECT_ID = "project_id"
-    DATA_LOCATION = "data_location"
-    DOWNLOAD_BUCKET = "download_bucket"
-    TRANSFORM_BUCKET = "transform_bucket"
-    TERRAFORM_ORGANIZATION = "terraform_organization"
-    DAGS_MODULE_NAMES = "dags_module_names"
-    KIBANA_SPACES = "kibana_spaces"
-
-
-class AirflowConns:
-    """ Common Airflow Connection names used with the Observatory Platform """
-
-    CROSSREF = "crossref"
-    MAG_RELEASES_TABLE = "mag_releases_table"
-    MAG_SNAPSHOTS_CONTAINER = "mag_snapshots_container"
-    TERRAFORM = "terraform"
-    SLACK = "slack"
-    ELASTIC = "elastic"
-    KIBANA = "kibana"
-
 
 def module_file_path(module_path: str, nav_back_steps: int = -1) -> str:
     """ Get the file path of a module, given the Python import path to the module.
@@ -104,8 +69,8 @@ def terraform_credentials_path() -> str:
     return os.path.join(pathlib.Path.home(), '.terraform.d/credentials.tfrc.json')
 
 
-def find_schema(path: str, table_name: str, release_date: Pendulum, prefix: str = '', ver: str = '') \
-        -> Union[str, None]:
+def find_schema(path: str, table_name: str, release_date: Pendulum, prefix: str = '', ver: str = None) -> Union[
+    str, None]:
     """ Finds a schema file on a given path, with a particular table name, release date and optional prefix.
 
     If no version string is sepcified, the most recent schema with a date less than or equal to the release date of the
@@ -138,8 +103,12 @@ def find_schema(path: str, table_name: str, release_date: Pendulum, prefix: str
     :return: the path to the schema or None if no schema was found.
     """
 
+    logging.info(f'Looking for schema with search parameters: analysis_schema_path={path}, '
+                  f'prefix={prefix}, table_name={table_name}, release_date={release_date}, '
+                  f'version={ver}')
+
     # Make search path for schemas
-    if ver != '':
+    if ver:
         search_path = os.path.join(path, f'{prefix}{table_name}_{ver}_*.json')
     else:
         search_path = os.path.join(path, f'{prefix}{table_name}*.json')
@@ -150,10 +119,11 @@ def find_schema(path: str, table_name: str, release_date: Pendulum, prefix: str
 
     # No schemas were found
     if len(schema_paths) == 0:
+        logging.error('No schema found.')
         return None
 
     # Deal with versioned schema first since it's simpler. Return most recent for versioned schema.
-    if ver != '':
+    if ver:
         return schema_paths[-1]
 
     # Get schemas with dates <= release date
@@ -174,95 +144,5 @@ def find_schema(path: str, table_name: str, release_date: Pendulum, prefix: str
         return selected_paths[-1]
 
     # No schemas were found
+    logging.error('No schema found.')
     return None
-
-
-def check_variables(*variables):
-    """ Checks whether all given airflow variables exist.
-
-    :param variables: name of airflow variable
-    :return: True if all variables are valid
-    """
-    is_valid = True
-    for name in variables:
-        try:
-            AirflowVariable.get(name)
-        except KeyError:
-            logging.error(f"Airflow variable '{name}' not set.")
-            is_valid = False
-    return is_valid
-
-
-def check_connections(*connections):
-    """ Checks whether all given airflow connections exist.
-
-    :param connections: name of airflow connection
-    :return: True if all connections are valid
-    """
-    is_valid = True
-    for name in connections:
-        try:
-            BaseHook.get_connection(name)
-        except KeyError:
-            logging.error(f"Airflow connection '{name}' not set.")
-            is_valid = False
-    return is_valid
-
-
-def list_connections(source):
-    """ Get a list of data source connections with name starting with <source>_, e.g., wos_curtin.
-
-    :param source: Data source (conforming to name convention) as a string, e.g., 'wos'.
-    :return: A list of connection id strings with the prefix <source>_, e.g., ['wos_curtin', 'wos_auckland'].
-    """
-    with create_session() as session:
-        query = session.query(Connection)
-        query = query.filter(Connection.conn_id.like(f'{source}_%'))
-        return query.all()
-
-
-class SubFolder(Enum):
-    """ The type of subfolder to create for telescope data """
-
-    downloaded = 'download'
-    extracted = 'extract'
-    transformed = 'transform'
-
-
-def telescope_path(sub_folder: SubFolder, name: str) -> str:
-    """ Return a path for saving telescope data. Create it if it doesn't exist.
-
-    :param sub_folder: the name of the sub folder for the telescope
-    :param name: the name of the telescope.
-    :return: the path.
-    """
-
-    # To avoid hitting the airflow database and the secret backend unnecessarily, data path is stored as a global
-    # variable and only requested once.
-    global data_path
-    if data_path is None:
-        logging.info('telescope_path: requesting data_path variable')
-        data_path = airflow.models.Variable.get(AirflowVars.DATA_PATH)
-
-    # Create telescope path
-    path = os.path.join(data_path, 'telescopes', sub_folder.value, name)
-    if not os.path.exists(path):
-        os.makedirs(path, exist_ok=True)
-
-    return path
-
-
-def test_data_path() -> str:
-    """ Return the path for the test data.
-
-    :return: the path.
-    """
-
-    # To avoid hitting the airflow database and the secret backend unnecessarily, data path is stored as a global
-    # variable and only requested once.
-    global test_data_path_val_
-    if test_data_path_val_ is None:
-        logging.info('test_data_path: requesting test_data_path variable')
-        test_data_path_val_ = airflow.models.Variable.get(AirflowVars.TEST_DATA_PATH)
-
-    return test_data_path_val_
diff --git a/observatory-platform/observatory/platform/utils/data_utils.py b/observatory-platform/observatory/platform/utils/data_utils.py
index 3938cc8d1..25a758c56 100644
--- a/observatory-platform/observatory/platform/utils/data_utils.py
+++ b/observatory-platform/observatory/platform/utils/data_utils.py
@@ -47,15 +47,15 @@
 from __future__ import division
 from __future__ import print_function
 
-import hashlib
 import os
 import shutil
+import sys
 import tarfile
 import zipfile
 from contextlib import closing
 
 import six
-import sys
+from observatory.platform.utils.file_utils import validate_file
 from six.moves.urllib.error import HTTPError
 from six.moves.urllib.error import URLError
 from six.moves.urllib.request import urlopen
@@ -140,8 +140,7 @@ def _extract_archive(file_path, path='.', archive_format='auto'):
             with open_fn(file_path) as archive:
                 try:
                     archive.extractall(path)
-                except (tarfile.TarError, RuntimeError,
-                        KeyboardInterrupt):
+                except (tarfile.TarError, RuntimeError, KeyboardInterrupt):
                     if os.path.exists(path):
                         if os.path.isfile(path):
                             os.remove(path)
@@ -152,16 +151,8 @@ def _extract_archive(file_path, path='.', archive_format='auto'):
     return False
 
 
-def get_file(fname,
-             origin,
-             untar=False,
-             md5_hash=None,
-             file_hash=None,
-             cache_subdir='datasets',
-             hash_algorithm='auto',
-             extract=False,
-             archive_format='auto',
-             cache_dir=None):
+def get_file(fname, origin, untar=False, md5_hash=None, file_hash=None, cache_subdir='datasets', hash_algorithm='auto',
+             extract=False, archive_format='auto', cache_dir=None):
     """Downloads a file from a URL if it not already in the cache.
 
     By default the file at the url `origin` is downloaded to the
@@ -274,59 +265,3 @@ def dl_progress(count, block_size, total_size):
         _extract_archive(fpath, datadir, archive_format)
 
     return fpath, download
-
-
-def _hash_file(fpath, algorithm='sha256', chunk_size=65535):
-    """Calculates a file sha256 or md5 hash.
-
-    # Example
-
-    ```python
-        >>> from keras.utils.data_utils import _hash_file
-        >>> _hash_file('/path/to/file.zip')
-        'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
-    ```
-
-    # Arguments
-        fpath: path to the file being validated
-        algorithm: hash algorithm, one of 'auto', 'sha256', or 'md5'.
-            The default 'auto' detects the hash algorithm in use.
-        chunk_size: Bytes to read at a time, important for large files.
-
-    # Returns
-        The file hash
-    """
-    if (algorithm == 'sha256') or (algorithm == 'auto' and len(hash) == 64):
-        hasher = hashlib.sha256()
-    else:
-        hasher = hashlib.md5()
-
-    with open(fpath, 'rb') as fpath_file:
-        for chunk in iter(lambda: fpath_file.read(chunk_size), b''):
-            hasher.update(chunk)
-
-    return hasher.hexdigest()
-
-
-def validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535):
-    """Validates a file against a sha256 or md5 hash.
-    # Arguments
-        fpath: path to the file being validated
-        file_hash:  The expected hash string of the file.
-            The sha256 and md5 hash algorithms are both supported.
-        algorithm: Hash algorithm, one of 'auto', 'sha256', or 'md5'.
-            The default 'auto' detects the hash algorithm in use.
-        chunk_size: Bytes to read at a time, important for large files.
-    # Returns
-        Whether the file is valid
-    """
-    if ((algorithm == 'sha256') or
-            (algorithm == 'auto' and len(file_hash) == 64)):
-        hasher = 'sha256'
-    else:
-        hasher = 'md5'
-
-    if str(_hash_file(fpath, hasher, chunk_size)) == str(file_hash):
-        return True
-    else:
-        return False
diff --git a/observatory-platform/observatory/platform/utils/file_utils.py b/observatory-platform/observatory/platform/utils/file_utils.py
new file mode 100644
index 000000000..d17223ad4
--- /dev/null
+++ b/observatory-platform/observatory/platform/utils/file_utils.py
@@ -0,0 +1,142 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: James Diprose
+
+"""
+File utility functions
+"""
+
+import codecs
+import hashlib
+import os
+import re
+import subprocess
+from subprocess import Popen
+from typing import List
+
+from google_crc32c import Checksum as Crc32cChecksum
+from observatory.platform.utils.proc_utils import wait_for_process
+
+
+def list_files(path: str, regex: str = None) -> List[str]:
+    """ Recursively list files on a path. Optionally only return files that match a regular expression.
+    :param path: the path to recursively list files on.
+    :param regex: a regular expression, if a file matches, then it is included.
+    :return: a list of full paths to the files.
+    """
+
+    paths = []
+    for root, dirs, files in os.walk(path):
+        for file in files:
+            if regex is None or re.match(regex, file):
+                paths.append(os.path.join(root, file))
+    return paths
+
+
+def _hash_file(fpath, algorithm='sha256', chunk_size=65535):
+    """Calculates a file sha256 or md5 hash.
+
+    # Example
+
+    ```python
+        >>> from keras.utils.data_utils import _hash_file
+        >>> _hash_file('/path/to/file.zip')
+        'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
+    ```
+
+    # Arguments
+        fpath: path to the file being validated
+        algorithm: hash algorithm, one of 'auto', 'sha256', or 'md5'.
+            The default 'auto' detects the hash algorithm in use.
+        chunk_size: Bytes to read at a time, important for large files.
+
+    # Returns
+        The file hash
+    """
+    if (algorithm == 'sha256') or (algorithm == 'auto' and len(hash) == 64):
+        hasher = hashlib.sha256()
+    else:
+        hasher = hashlib.md5()
+
+    with open(fpath, 'rb') as fpath_file:
+        for chunk in iter(lambda: fpath_file.read(chunk_size), b''):
+            hasher.update(chunk)
+
+    return hasher.hexdigest()
+
+
+def validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535):
+    """Validates a file against a sha256 or md5 hash.
+    # Arguments
+        fpath: path to the file being validated
+        file_hash:  The expected hash string of the file.
+            The sha256 and md5 hash algorithms are both supported.
+        algorithm: Hash algorithm, one of 'auto', 'sha256', or 'md5'.
+            The default 'auto' detects the hash algorithm in use.
+        chunk_size: Bytes to read at a time, important for large files.
+    # Returns
+        Whether the file is valid
+    """
+    if ((algorithm == 'sha256') or (algorithm == 'auto' and len(file_hash) == 64)):
+        hasher = 'sha256'
+    else:
+        hasher = 'md5'
+
+    if str(_hash_file(fpath, hasher, chunk_size)) == str(file_hash):
+        return True
+    else:
+        return False
+
+
+def gzip_file_crc(file_path: str) -> str:
+    """ Get the crc of a gzip file.
+
+    :param file_path: the path to the file.
+    :return: the crc.
+    """
+
+    proc: Popen = subprocess.Popen(['gzip', '-vl', file_path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    output, error = wait_for_process(proc)
+    return output.splitlines()[1].split(' ')[1].strip()
+
+
+def hex_to_base64_str(hex_str: bytes) -> str:
+    """ Covert a hexadecimal string into a base64 encoded string. Removes trailing newline character.
+
+    :param hex_str: the hexadecimal encoded string.
+    :return: the base64 encoded string.
+    """
+
+    string = codecs.decode(hex_str, 'hex')
+    base64 = codecs.encode(string, 'base64')
+    return base64.decode('utf8').rstrip('\n')
+
+
+def crc32c_base64_hash(file_path: str, chunk_size: int = 8 * 1024) -> str:
+    """ Create a base64 crc32c checksum of a file.
+
+    :param file_path: the path to the file.
+    :param chunk_size: the size of each chunk to check.
+    :return: the checksum.
+    """
+
+    hash_alg = Crc32cChecksum()
+
+    with open(file_path, 'rb') as f:
+        chunk = f.read(chunk_size)
+        while chunk:
+            hash_alg.update(chunk)
+            chunk = f.read(chunk_size)
+    return hex_to_base64_str(hash_alg.hexdigest())
diff --git a/observatory-platform/observatory/platform/utils/gc_utils.py b/observatory-platform/observatory/platform/utils/gc_utils.py
index 564fcc14f..fb1469a6b 100644
--- a/observatory-platform/observatory/platform/utils/gc_utils.py
+++ b/observatory-platform/observatory/platform/utils/gc_utils.py
@@ -12,81 +12,36 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Author: James Diprose
+# Author: James Diprose, Aniek Roelofs
+
+""" General google cloud utility functions (independent of telescope usage) """
 
-import codecs
 import json
 import logging
 import multiprocessing
 import os
 import re
-import subprocess
 import time
-from concurrent.futures import as_completed, ProcessPoolExecutor
+from concurrent.futures import ProcessPoolExecutor, as_completed
 from enum import Enum
 from multiprocessing import BoundedSemaphore, cpu_count
-from subprocess import Popen
 from typing import List, Union
 
 import pendulum
-from google_crc32c import Checksum as Crc32cChecksum
-from google.api_core.exceptions import Conflict, BadRequest
-from google.cloud import storage, bigquery
-from google.cloud.bigquery import SourceFormat, LoadJobConfig, LoadJob, QueryJob
+from google.api_core.exceptions import BadRequest, Conflict, Forbidden
+from google.cloud import bigquery, storage
+from google.cloud.bigquery import LoadJob, LoadJobConfig, QueryJob, SourceFormat
 from google.cloud.exceptions import NotFound
 from google.cloud.storage import Blob
 from googleapiclient import discovery as gcp_api
+from observatory.platform.utils.file_utils import crc32c_base64_hash
 from pendulum import Pendulum
 from requests.exceptions import ChunkedEncodingError
 
-from observatory.platform.utils.proc_utils import wait_for_process
-
 # The chunk size to use when uploading / downloading a blob in multiple parts, must be a multiple of 256 KB.
 DEFAULT_CHUNK_SIZE = 256 * 1024 * 4
 
 
-def gzip_file_crc(file_path: str) -> str:
-    """ Get the crc of a gzip file.
-
-    :param file_path: the path to the file.
-    :return: the crc.
-    """
-
-    proc: Popen = subprocess.Popen(['gzip', '-vl', file_path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    output, error = wait_for_process(proc)
-    return output.splitlines()[1].split(' ')[1].strip()
-
-
-def hex_to_base64_str(hex_str: bytes) -> str:
-    """ Covert a hexadecimal string into a base64 encoded string. Removes trailing newline character.
-
-    :param hex_str: the hexadecimal encoded string.
-    :return: the base64 encoded string.
-    """
-
-    string = codecs.decode(hex_str, 'hex')
-    base64 = codecs.encode(string, 'base64')
-    return base64.decode('utf8').rstrip('\n')
-
-
-def crc32c_base64_hash(file_path: str, chunk_size: int = 8 * 1024) -> str:
-    """ Create a base64 crc32c checksum of a file.
-
-    :param file_path: the path to the file.
-    :param chunk_size: the size of each chunk to check.
-    :return: the checksum.
-    """
-
-    hash_alg = Crc32cChecksum()
-
-    with open(file_path, 'rb') as f:
-        chunk = f.read(chunk_size)
-        while chunk:
-            hash_alg.update(chunk)
-            chunk = f.read(chunk_size)
-    return hex_to_base64_str(hash_alg.hexdigest())
-
-
 def table_name_from_blob(blob_name: str, file_extension: str):
     """ Make a BigQuery table name from a blob name.
 
@@ -176,12 +131,12 @@ def create_bigquery_dataset(project_id: str, dataset_id: str, location: str, des
         logging.warning(f"{func_name}: dataset already exists dataset_ref={dataset_ref}, exception={e}")
 
 
-def load_bigquery_table(uri: str, dataset_id: str, location: str, table: str, schema_file_path: str,
-                        source_format: str, csv_field_delimiter: str = ',', csv_quote_character: str = '"',
+def load_bigquery_table(uri: str, dataset_id: str, location: str, table: str, schema_file_path: str, source_format: str,
+                        csv_field_delimiter: str = ',', csv_quote_character: str = '"',
                         csv_allow_quoted_newlines: bool = False, csv_skip_leading_rows: int = 0,
                         partition: bool = False, partition_field: Union[None, str] = None,
                         partition_type: str = bigquery.TimePartitioningType.DAY, require_partition_filter=True,
-                        write_disposition: str = bigquery.WriteDisposition.WRITE_TRUNCATE) -> bool:
+                        write_disposition: str = bigquery.WriteDisposition.WRITE_TRUNCATE, description: str = '') -> bool:
     """ Load a BigQuery table from an object on Google Cloud Storage.
 
     :param uri: the uri of the object to load from Google Cloud Storage into BigQuery.
@@ -199,6 +154,7 @@ def load_bigquery_table(uri: str, dataset_id: str, location: str, table: str, sc
     :param partition_type: the type of partitioning.
     :param require_partition_filter: whether the partition filter is required or not when querying the table.
     :param write_disposition: whether to append, overwrite or throw an error when data already exists in the table.
+    :param description: the description of the table.
     Default is to overwrite.
     :return:
     """
@@ -220,6 +176,7 @@ def load_bigquery_table(uri: str, dataset_id: str, location: str, table: str, sc
     job_config.source_format = source_format
     job_config.schema = client.schema_from_json(schema_file_path)
     job_config.write_disposition = write_disposition
+    job_config.destination_table_description = description
 
     # Set CSV options
     if source_format == SourceFormat.CSV:
@@ -230,19 +187,12 @@ def load_bigquery_table(uri: str, dataset_id: str, location: str, table: str, sc
 
     # Set partitioning settings
     if partition:
-        job_config.time_partitioning = bigquery.TimePartitioning(
-            type_=partition_type,
-            field=partition_field,
-            require_partition_filter=require_partition_filter
-        )
+        job_config.time_partitioning = bigquery.TimePartitioning(type_=partition_type, field=partition_field,
+                                                                 require_partition_filter=require_partition_filter)
 
     try:
-        load_job: LoadJob = client.load_table_from_uri(
-            uri,
-            dataset.table(table),
-            location=location,
-            job_config=job_config
-        )
+        load_job: LoadJob = client.load_table_from_uri(uri, dataset.table(table), location=location,
+                                                       job_config=job_config)
         result = load_job.result()
         state = result.state == 'DONE'
 
@@ -270,18 +220,26 @@ def run_bigquery_query(query: str) -> List:
     return list(rows)
 
 
-def copy_bigquery_table(source_table_id: str, destination_table_id: str, data_location: str) -> bool:
+def copy_bigquery_table(source_table_id: Union[str, list], destination_table_id: str, data_location: str,
+                        write_disposition: bigquery.WriteDisposition = bigquery.WriteDisposition.WRITE_TRUNCATE) -> \
+        bool:
     """ Copy a BigQuery table.
 
     :param source_table_id: the id of the source table, including the project name and dataset id.
     :param destination_table_id: the id of the destination table, including the project name and dataset id.
     :param data_location: the location of the datasets.
+    :param write_disposition: whether to append, overwrite or throw an error when data already exists in the table.
     :return: whether the table was copied successfully or not.
     """
+    func_name = copy_bigquery_table.__name__
+    msg = f'source_table_ids={source_table_id}, destination_table_id={destination_table_id}, location={data_location}'
+    logging.info(f"{func_name}: copying bigquery table {msg}")
 
     client = bigquery.Client()
     job_config = bigquery.CopyJobConfig()
-    job_config.write_disposition = "WRITE_TRUNCATE"
+
+    job_config.write_disposition = write_disposition
+
     job = client.copy_table(source_table_id, destination_table_id, location=data_location, job_config=job_config)
     result = job.result()
     return result.done()
@@ -306,9 +264,8 @@ def create_bigquery_view(project_id: str, dataset_id: str, view_name: str, query
 
 
 def create_bigquery_table_from_query(sql: str, project_id: str, dataset_id: str, table_id: str, location: str,
-                                     description: str = '', labels=None,
-                                     query_parameters=None,
-                                     partition: bool = False, partition_field: Union[None, str] = None,
+                                     description: str = '', labels=None, query_parameters=None, partition: bool = False,
+                                     partition_field: Union[None, str] = None,
                                      partition_type: str = bigquery.TimePartitioningType.DAY,
                                      require_partition_filter=True, cluster: bool = False,
                                      clustering_fields=None) -> bool:
@@ -355,23 +312,15 @@ def create_bigquery_table_from_query(sql: str, project_id: str, dataset_id: str,
     dataset.location = location
     dataset.description = description
 
-    job_config = bigquery.QueryJobConfig(
-        allow_large_results=True,
-        destination=dataset.table(table_id),
-        description=description,
-        labels=labels,
-        use_legacy_sql=False,
-        query_parameters=query_parameters,
-        write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE
-    )
+    job_config = bigquery.QueryJobConfig(allow_large_results=True, destination=dataset.table(table_id),
+                                         description=description, labels=labels, use_legacy_sql=False,
+                                         query_parameters=query_parameters,
+                                         write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE)
 
     # Set partitioning settings
     if partition:
-        job_config.time_partitioning = bigquery.TimePartitioning(
-            type_=partition_type,
-            field=partition_field,
-            require_partition_filter=require_partition_filter,
-        )
+        job_config.time_partitioning = bigquery.TimePartitioning(type_=partition_type, field=partition_field,
+                                                                 require_partition_filter=require_partition_filter, )
 
     if cluster:
         job_config.clustering_fields = clustering_fields
@@ -383,6 +332,19 @@ def create_bigquery_table_from_query(sql: str, project_id: str, dataset_id: str,
     return success
 
 
+def storage_bucket_exists(bucket_name: str):
+    """ Check if a storage bucket exists.
+    :param bucket_name: The bucket name.
+    :return: None.
+    """
+    client = storage.Client()
+    try:
+        bucket = client.get_bucket(bucket_name)
+    except Forbidden:
+        bucket = None
+    return True if bucket else None
+
+
 def download_blob_from_cloud_storage(bucket_name: str, blob_name: str, file_path: str, retries: int = 3,
                                      connection_sem: BoundedSemaphore = None,
                                      chunk_size: int = DEFAULT_CHUNK_SIZE) -> bool:
@@ -628,8 +590,7 @@ def upload_file_to_cloud_storage(bucket_name: str, blob_name: str, file_path: st
 
 def azure_to_google_cloud_storage_transfer(azure_storage_account_name: str, azure_sas_token: str, azure_container: str,
                                            include_prefixes: List[str], gc_project_id: str, gc_bucket: str,
-                                           description: str, start_date: Pendulum = pendulum.utcnow()) \
-        -> bool:
+                                           description: str, start_date: Pendulum = pendulum.utcnow()) -> bool:
     """ Transfer files from an Azure blob container to a Google Cloud Storage bucket.
 
     :param azure_storage_account_name: the name of the Azure Storage account that holds the Azure blob container.
@@ -724,22 +685,3 @@ def azure_to_google_cloud_storage_transfer(azure_storage_account_name: str, azur
         time.sleep(wait_time)
 
     return status == TransferStatus.success
-
-
-def upload_telescope_file_list(bucket_name: str, inst_id: str, telescope_path: str, file_list: List[str]) -> List[str]:
-    """ Upload list of files to cloud storage.
-
-    :param bucket_name: Name of storage bucket.
-    :param inst_id: institution id from airflow connection id.
-    :param telescope_path: Path to upload telescope data.
-    :param file_list: List of files to upload.
-    :return: List of location paths in the cloud.
-    """
-
-    blob_list = list()
-    for file in file_list:
-        file_name = os.path.basename(file)
-        blob_name = f'{telescope_path}/{inst_id}/{file_name}'
-        blob_list.append(blob_name)
-        upload_file_to_cloud_storage(bucket_name, blob_name, file_path=file)
-    return blob_list
diff --git a/observatory-platform/observatory/platform/utils/telescope_utils.py b/observatory-platform/observatory/platform/utils/telescope_utils.py
index 99dfdbcc8..52687fb1d 100644
--- a/observatory-platform/observatory/platform/utils/telescope_utils.py
+++ b/observatory-platform/observatory/platform/utils/telescope_utils.py
@@ -12,25 +12,100 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Author: Tuan Chien
+# Author: Tuan Chien, Aniek Roelofs, James Diprose
 
+""" Utility functions that support specific telescope(s) """
 
 import calendar
 import gzip
+import io
 import json
 import logging
 import os
+import re
 import shutil
+import sys
 from collections import deque
 from dataclasses import dataclass
+from math import ceil
 from pathlib import Path
-from typing import List, Tuple, Any, Type
+from typing import Any, List, Tuple, Type
 
 import jsonlines
 import pendulum
-import sys
 from airflow.models.taskinstance import TaskInstance
-from math import ceil
+from observatory.dags.config import workflow_sql_templates_path
+from observatory.platform.utils.gc_utils import upload_file_to_cloud_storage
+from observatory.platform.utils.jinja2_utils import (make_jinja2_filename, render_template)
+
+
+def list_to_jsonl_gz(file_path: str, list_of_dicts: List[dict]):
+    """ Takes a list of dictionaries and writes this to a gzipped jsonl file.
+    :param file_path: Path to the .jsonl.gz file
+    :param list_of_dicts: A list containing dictionaries that can be written out with jsonlines
+    :return: None.
+    """
+    with io.BytesIO() as bytes_io:
+        with gzip.GzipFile(fileobj=bytes_io, mode='w') as gzip_file:
+            with jsonlines.Writer(gzip_file) as writer:
+                writer.write_all(list_of_dicts)
+
+        with open(file_path, 'wb') as jsonl_gzip_file:
+            jsonl_gzip_file.write(bytes_io.getvalue())
+
+
+def args_list(args) -> list:
+    return args
+
+
+def write_boto_config(s3_host: str, aws_access_key_id: str, aws_secret_access_key: str, boto_config_path: str):
+    """ Write a boto3 configuration file, created by using a template and given info.
+    :param s3_host: Host of the s3 bucket.
+    :param aws_access_key_id: Access key id to bucket.
+    :param aws_secret_access_key: Secret access key to bucket.
+    :param boto_config_path: Path to write the boto config file to.
+    :return: None.
+    """
+    logging.info(f'Writing boto config file to {boto_config_path}')
+
+    template_path = os.path.join(workflow_sql_templates_path(), make_jinja2_filename('boto'))
+    rendered = render_template(template_path, s3_host=s3_host, aws_access_key_id=aws_access_key_id,
+                               aws_secret_access_key=aws_secret_access_key)
+    with open(boto_config_path, 'w') as f:
+        f.write(rendered)
+
+
+def convert(k: str) -> str:
+    """ Convert a key name.
+    BigQuery specification for field names: Fields must contain only letters, numbers, and underscores, start with a
+    letter or underscore, and be at most 128 characters long.
+    :param k: Key.
+    :return: Converted key.
+    """
+    # Trim special characters at start:
+    k = re.sub('^[^A-Za-z0-9]+', "", k)
+    # Replace other special characters (except '_') in remaining string:
+    k = re.sub('\W+', '_', k)
+    return k
+
+
+def change_keys(obj, convert):
+    """ Recursively goes through the dictionary obj and replaces keys with the convert function.
+    :param obj: Dictionary object.
+    :param convert: Convert function.
+    :return: Updated dictionary object.
+    """
+    if isinstance(obj, (str, int, float)):
+        return obj
+    if isinstance(obj, dict):
+        new = obj.__class__()
+        for k, v in list(obj.items()):
+            new[convert(k)] = change_keys(v, convert)
+    elif isinstance(obj, (list, set, tuple)):
+        new = obj.__class__(change_keys(v, convert) for v in obj)
+    else:
+        return obj
+    return new
 
 
 def build_schedule(sched_start_date, sched_end_date):
@@ -365,3 +440,22 @@ def optimise(max_per_call: int, max_per_query: int, historic_counts: List[Type[P
 
         schedule = ScheduleOptimiser.extract_schedule(historic_counts, moves)
         return schedule, min_calls[-1]
+
+
+def upload_telescope_file_list(bucket_name: str, inst_id: str, telescope_path: str, file_list: List[str]) -> List[str]:
+    """ Upload list of files to cloud storage.
+
+    :param bucket_name: Name of storage bucket.
+    :param inst_id: institution id from airflow connection id.
+    :param telescope_path: Path to upload telescope data.
+    :param file_list: List of files to upload.
+    :return: List of location paths in the cloud.
+    """
+
+    blob_list = list()
+    for file in file_list:
+        file_name = os.path.basename(file)
+        blob_name = f'{telescope_path}/{inst_id}/{file_name}'
+        blob_list.append(blob_name)
+        upload_file_to_cloud_storage(bucket_name, blob_name, file_path=file)
+    return blob_list
diff --git a/observatory-platform/observatory/platform/utils/template_utils.py b/observatory-platform/observatory/platform/utils/template_utils.py
new file mode 100644
index 000000000..9a70bbc11
--- /dev/null
+++ b/observatory-platform/observatory/platform/utils/template_utils.py
@@ -0,0 +1,369 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs
+
+""" Utility functions that support (almost) all telescopes/the template """
+
+import logging
+import os
+import pathlib
+import traceback
+from enum import Enum
+from typing import List, Tuple
+
+import pendulum
+import six
+from airflow import AirflowException
+from airflow.utils.dates import cron_presets
+from croniter import croniter
+from datetime import timedelta
+from google.cloud import bigquery
+from google.cloud.bigquery import SourceFormat
+from observatory.dags.config import schema_path, workflow_sql_templates_path
+from observatory.platform.observatory_config import Environment
+from observatory.platform.utils.airflow_utils import AirflowVariable, AirflowVars, create_slack_webhook
+from observatory.platform.utils.config_utils import find_schema
+from observatory.platform.utils.gc_utils import bigquery_partitioned_table_id, \
+    copy_bigquery_table, \
+    create_bigquery_dataset, \
+    load_bigquery_table, \
+    run_bigquery_query, \
+    upload_files_to_cloud_storage
+from observatory.platform.utils.jinja2_utils import make_sql_jinja2_filename, render_template
+
+# To avoid hitting the airflow database and the secret backend unnecessarily, some variables are stored as a global
+# variable and only requested once
+data_path = None
+test_data_path_val_ = None
+project_id = None
+bucket_name = None
+data_location = None
+environment = None
+
+
+def telescope_path(*subdirs) -> str:
+    """ Return a path for saving telescope data. Create it if it doesn't exist.
+    :param subdirs: the subdirectories.
+    :return: the path.
+    """
+    global data_path
+    if data_path is None:
+        logging.info('telescope_path: requesting data_path variable')
+        data_path = AirflowVariable.get(AirflowVars.DATA_PATH)
+
+    subdirs = [subdir.value if isinstance(subdir, Enum) else subdir for subdir in subdirs]
+
+    # Create telescope path
+    path = os.path.join(data_path, 'telescopes', *subdirs)
+    if not os.path.exists(path):
+        os.makedirs(path, exist_ok=True)
+    return path
+
+
+def test_data_path() -> str:
+    """ Return the path for the test data.
+
+    :return: the path.
+    """
+    global test_data_path_val_
+    if test_data_path_val_ is None:
+        logging.info('test_data_path: requesting test_data_path variable')
+        test_data_path_val_ = AirflowVariable.get(AirflowVars.TEST_DATA_PATH)
+
+    return test_data_path_val_
+
+
+def blob_name(path: str) -> str:
+    """ Convert a file path into the full path of the Blob, excluding the bucket name.
+    :param path: the path to the file.
+    :return: the full path of the Blob.
+    """
+
+    # Remove everything before 'telescopes' and make sure that we are using posix file paths
+    # Which use / forward slashes rather than backslashes
+    sub_paths = pathlib.Path(os.path.relpath(path, telescope_path())).parts[1:]
+    relative_path = pathlib.Path(*sub_paths).as_posix()
+
+    return pathlib.Path(os.path.join('telescopes', relative_path)).as_posix()
+
+
+def upload_files_from_list(files_list: List[str], bucket_name: str) -> bool:
+    """ Upload all files in a list to the google cloud download bucket.
+    :param files_list: List of full path of files that will be uploaded
+    :param bucket_name: The name of the google cloud bucket
+    :return: True if upload was successful, else False.
+    """
+    blob_names = []
+    for file_path in files_list:
+        blob_names.append(blob_name(file_path))
+
+    success = upload_files_to_cloud_storage(bucket_name, blob_names, files_list)
+    return success
+
+
+def table_ids_from_path(transform_path: str) -> Tuple[str, str]:
+    """
+    To create the table ids the basename of the transform path is taken and file extensions are stripped.
+    E.g.: /opt/observatory/
+    :param transform_path: The full path of a file in the transform folder
+    :return: Main table id and partition table id
+    """
+    logging.info(f'Creating table ids from path: {transform_path}')
+
+    main_table_id = os.path.splitext(pathlib.Path(transform_path).stem)[0]
+    partition_table_id = f'{main_table_id}_partitions'
+
+    logging.info(f'Table id: {main_table_id}, partition table id: {partition_table_id}')
+    return main_table_id, partition_table_id
+
+
+def prepare_bq_load(dataset_id: str, table_id: str, release_date: pendulum.Pendulum, prefix: str,
+                    schema_version: str) -> [str, str, str, str]:
+    """
+    Prepare to load data into BigQuery. This will:
+     - create the dataset if it does not exist yet
+     - get the path to the schema
+     - return values of project id, bucket name, and data location
+    :param dataset_id: Dataset id.
+    :param table_id: Table id.
+    :param release_date: The release date used for schema lookup.
+    :param prefix: The prefix for the schema.
+    :param schema_version: Schema version.
+    :return: The project id, bucket name, data location and schema path
+    """
+    global project_id
+    if project_id is None:
+        logging.info('requesting project_id variable')
+        project_id = AirflowVariable.get(AirflowVars.PROJECT_ID)
+
+    global bucket_name
+    if bucket_name is None:
+        logging.info('requesting transform_bucket variable')
+        bucket_name = AirflowVariable.get(AirflowVars.TRANSFORM_BUCKET)
+
+    global data_location
+    if data_location is None:
+        logging.info('requesting data_location variable')
+        data_location = AirflowVariable.get(AirflowVars.DATA_LOCATION)
+
+    # Create dataset
+    dataset_id = dataset_id
+    create_bigquery_dataset(project_id, dataset_id, data_location)
+
+    # Select schema file based on release date
+    analysis_schema_path = schema_path()
+    schema_file_path = find_schema(analysis_schema_path, table_id, release_date, prefix, schema_version)
+    if schema_file_path is None:
+        exit(os.EX_CONFIG)
+    return project_id, bucket_name, data_location, schema_file_path
+
+
+def bq_load_shard(release_date: pendulum.Pendulum, transform_blob: str, dataset_id: str, table_id: str,
+                  prefix: str = '', schema_version: str = None, description: str = None):
+    """ Load data from a specific file (blob) in the transform bucket to a BigQuery shard.
+    :param release_date: Release date.
+    :param transform_blob: Name of the transform blob.
+    :param dataset_id: Dataset id.
+    :param table_id: Table id.
+    :param prefix: The prefix for the schema.
+    :param schema_version: Schema version.
+    :param description: The description for the dataset
+    :return: None.
+    """
+    _, bucket_name, data_location, schema_file_path = prepare_bq_load(dataset_id, table_id, release_date, prefix,
+                                                                      schema_version)
+
+    # Create table id
+    table_id = bigquery_partitioned_table_id(table_id, release_date)
+
+    # Load BigQuery table
+    uri = f"gs://{bucket_name}/{transform_blob}"
+    logging.info(f"URI: {uri}")
+    success = load_bigquery_table(uri, dataset_id, data_location, table_id, schema_file_path,
+                                  SourceFormat.NEWLINE_DELIMITED_JSON, description=description)
+    if not success:
+        raise AirflowException()
+
+
+def bq_load_partition(end_date: pendulum.Pendulum, transform_blob: str, dataset_id: str, main_table_id: str,
+                      partition_table_id: str, prefix: str = '', schema_version: str = None, description: str = None):
+    """ Load data from a specific file (blob) in the transform bucket to a partition. Since no partition field is
+    given it will automatically partition by ingestion datetime.
+    :param end_date: End date.
+    :param transform_blob: Name of the transform blob.
+    :param dataset_id: Dataset id.
+    :param main_table_id: Main table id.
+    :param partition_table_id: Partition table id (should include date as data in table is overwritten).
+    :param prefix: The prefix for the schema.
+    :param schema_version: Schema version.
+    :param description: The description for the dataset
+    :return: None.
+    """
+    _, bucket_name, data_location, schema_file_path = prepare_bq_load(dataset_id, main_table_id, end_date, prefix,
+                                                                      schema_version)
+
+    uri = f"gs://{bucket_name}/{transform_blob}"
+
+    success = load_bigquery_table(uri, dataset_id, data_location, partition_table_id, schema_file_path,
+                                  SourceFormat.NEWLINE_DELIMITED_JSON, partition=True,
+                                  require_partition_filter=False, description=description)
+    if not success:
+        raise AirflowException()
+
+
+def bq_delete_old(start_date: pendulum.Pendulum, end_date: pendulum.Pendulum, dataset_id: str, main_table_id: str,
+                  partition_table_id: str, merge_partition_field: str, updated_date_field: str):
+    """ Will run a BigQuery query that deletes rows from the main table that are matched with rows from
+    specific partitions of the partition table.
+    The query is created from a template and the given info.
+    :param start_date: Start date.
+    :param end_date: End date.
+    :param dataset_id: Dataset id.
+    :param main_table_id: Main table id.
+    :param partition_table_id: Partition table id.
+    :param merge_partition_field: Merge partition field.
+    :param updated_date_field: Updated date field.
+    :return: None.
+    """
+    # include end date in period
+    start_date = start_date.strftime("%Y-%m-%d")
+    end_date = (end_date+timedelta(days=1)).strftime("%Y-%m-%d")
+    # Get merge variables
+    dataset_id = dataset_id
+    main_table = main_table_id
+    partitioned_table = partition_table_id
+
+    merge_condition_field = merge_partition_field
+    updated_date_field = updated_date_field
+
+    template_path = os.path.join(workflow_sql_templates_path(), make_sql_jinja2_filename('merge_delete_matched'))
+    query = render_template(template_path, dataset=dataset_id, main_table=main_table,
+                            partitioned_table=partitioned_table, merge_condition_field=merge_condition_field,
+                            start_date=start_date, end_date=end_date, updated_date_field=updated_date_field)
+    run_bigquery_query(query)
+
+
+def bq_append_from_partition(start_date: pendulum.Pendulum, end_date: pendulum.Pendulum, dataset_id: str,
+                             main_table_id: str, partition_table_id: str, prefix: str = '', schema_version: str = None):
+    """ Appends rows to the main table by coping specific partitions from the partition table to the main table.
+    :param start_date: Start date.
+    :param end_date: End date.
+    :param dataset_id: Dataset id.
+    :param main_table_id: Main table id.
+    :param partition_table_id: Partition table id.
+    :param prefix: The prefix for the schema.
+    :param schema_version: Schema version.
+    :return: None.
+    """
+    project_id, bucket_name, data_location, schema_file_path = prepare_bq_load(dataset_id, main_table_id, end_date,
+                                                                               prefix, schema_version)
+    # include end date in period
+    period = pendulum.period(start_date, end_date+timedelta(days=1))
+    logging.info(f'Getting table partitions: ')
+    source_table_ids = []
+    for dt in period:
+        table_id = f"{project_id}.{dataset_id}.{partition_table_id}${dt.strftime('%Y%m%d')}"
+        source_table_ids.append(table_id)
+        logging.info(f'Adding table_id: {table_id}')
+    destination_table_id = f"{project_id}.{dataset_id}.{main_table_id}"
+    success = copy_bigquery_table(source_table_ids, destination_table_id, data_location,
+                                  bigquery.WriteDisposition.WRITE_APPEND)
+    if not success:
+        raise AirflowException()
+
+
+def bq_append_from_file(end_date: pendulum.Pendulum, transform_blob: str, dataset_id: str, main_table_id: str,
+                        prefix: str = '', schema_version: str = None, description: str = None):
+    """ Appends rows to the main table by loading data from a specific file (blob) in the transform bucket.
+    :param end_date: End date.
+    :param transform_blob: Name of the transform blob.
+    :param dataset_id: Dataset id.
+    :param main_table_id: Main table id.
+    :param prefix: The prefix for the schema.
+    :param schema_version: Schema version.
+    :param description: The description.
+    :return: None.
+    """
+    project_id, bucket_name, data_location, schema_file_path = prepare_bq_load(dataset_id, main_table_id, end_date,
+                                                                               prefix, schema_version)
+
+    # Load BigQuery table
+    uri = f"gs://{bucket_name}/{transform_blob}"
+    logging.info(f"URI: {uri}")
+
+    # Append to table table
+    table_id = main_table_id
+    success = load_bigquery_table(uri, dataset_id, data_location, table_id, schema_file_path,
+                                  SourceFormat.NEWLINE_DELIMITED_JSON,
+                                  write_disposition=bigquery.WriteDisposition.WRITE_APPEND, description=description)
+    if not success:
+        raise AirflowException()
+
+
+def on_failure_callback(kwargs):
+    """
+    Function that is called on failure of an airflow task. Will create a slack webhook and send a notification.
+    :param kwargs:
+    :return: None.
+    """
+    global environment
+    if environment is None:
+        logging.info('requesting environment variable')
+        environment = AirflowVariable.get(AirflowVars.ENVIRONMENT)
+
+    global project_id
+    if project_id is None:
+        logging.info('requesting project_id variable')
+        project_id = AirflowVariable.get(AirflowVars.PROJECT_ID)
+
+    if environment == Environment.develop:
+        logging.info('Not sending slack notification in develop environment.')
+    else:
+        exception = kwargs.get('exception')
+        formatted_exception = ''.join(
+            traceback.format_exception(etype=type(exception), value=exception, tb=exception.__traceback__)).strip()
+        comments = f'Task failed, exception:\n{formatted_exception}'
+        slack_hook = create_slack_webhook(comments, project_id, **kwargs)
+        slack_hook.execute()
+
+
+def valid_cron_expression(field, value, error):
+    if not croniter.is_valid(normalize_schedule_interval(value)):
+        error(field, "Must be a valid cron expression")
+
+
+def normalize_schedule_interval(schedule_interval: str):
+    """
+    Returns Normalized Schedule Interval. This is used internally by the Scheduler to
+    schedule DAGs.
+    1. Converts Cron Preset to a Cron Expression (e.g ``@monthly`` to ``0 0 1 * *``)
+    2. If Schedule Interval is "@once" return "None"
+    3. If not (1) or (2) returns schedule_interval
+    """
+    if isinstance(schedule_interval, six.string_types) and schedule_interval in cron_presets:
+        _schedule_interval = cron_presets.get(schedule_interval)
+    elif schedule_interval == '@once':
+        _schedule_interval = None
+    else:
+        _schedule_interval = schedule_interval
+    return _schedule_interval
+
+
+class SubFolder(Enum):
+    """ The type of subfolder to create for telescope data """
+
+    downloaded = 'download'
+    extracted = 'extract'
+    transformed = 'transform'


Test patch:
diff --git a/tests/fixtures/vcr_cassettes/grid_2015-09-22.yaml b/tests/fixtures/vcr_cassettes/grid_2015-09-22.yaml
index 8d47809d7..b4289bef4 100644
--- a/tests/fixtures/vcr_cassettes/grid_2015-09-22.yaml
+++ b/tests/fixtures/vcr_cassettes/grid_2015-09-22.yaml
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4453196253710aa616da0a607ace2ddcfbcf4279d44cf376bd97389239be082f
-size 51455438
+oid sha256:443bc3570246ee0a763d27b4912b6eb08e21c999bb7e917f4df7a3074c39a1fc
+size 46848977
diff --git a/tests/fixtures/vcr_cassettes/grid_2020-03-15.yaml b/tests/fixtures/vcr_cassettes/grid_2020-03-15.yaml
index 12b66ef38..d534aba84 100644
--- a/tests/fixtures/vcr_cassettes/grid_2020-03-15.yaml
+++ b/tests/fixtures/vcr_cassettes/grid_2020-03-15.yaml
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b78d4c04ed5f0e80e75b5eee00b6f54de569a8b9c9ef8c6ce6236615baaa264c
+oid sha256:200365835ae887b66218ef558e22c0c1d371d6dd7b0da52e4606e8f2c0be80c7
 size 53598103
diff --git a/tests/fixtures/vcr_cassettes/list_grid_releases.yaml b/tests/fixtures/vcr_cassettes/list_grid_releases.yaml
index e2f235eda..2fa18f157 100644
--- a/tests/fixtures/vcr_cassettes/list_grid_releases.yaml
+++ b/tests/fixtures/vcr_cassettes/list_grid_releases.yaml
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ef646f95186356ac873b99160ba9a2efc36ca3eca974eebf2757e3b3b061c461
-size 5030
+oid sha256:76bb831c7566d40cbaafc991d64b6d131e8fb80c05fc1ef38bb14d6e147ef78f
+size 5342
diff --git a/tests/fixtures/vcr_cassettes/oapen_metadata_2021-02-19.yaml b/tests/fixtures/vcr_cassettes/oapen_metadata_2021-02-19.yaml
new file mode 100644
index 000000000..e463b568e
--- /dev/null
+++ b/tests/fixtures/vcr_cassettes/oapen_metadata_2021-02-19.yaml
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91c6dd84b5b11d62a4ad5c616c5df05565832451c99f0c76fa4ecdf386207f2e
+size 45416595
diff --git a/tests/observatory/dags/telescopes/test_crossref_metadata.py b/tests/observatory/dags/telescopes/test_crossref_metadata.py
index adf4cb946..e05619ed9 100644
--- a/tests/observatory/dags/telescopes/test_crossref_metadata.py
+++ b/tests/observatory/dags/telescopes/test_crossref_metadata.py
@@ -25,15 +25,13 @@
 import vcr
 from click.testing import CliRunner
 from natsort import natsorted
+from observatory.dags.telescopes.crossref_metadata import (CrossrefMetadataRelease,
+                                                           CrossrefMetadataTelescope,
+                                                           extract_release,
+                                                           transform_release)
+from observatory.platform.utils.file_utils import _hash_file
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
 
-from observatory.dags.telescopes.crossref_metadata import (
-    CrossrefMetadataRelease,
-    CrossrefMetadataTelescope,
-    extract_release,
-    transform_release
-)
-from observatory.platform.utils.config_utils import telescope_path, SubFolder
-from observatory.platform.utils.data_utils import _hash_file
 from tests.observatory.test_utils import test_fixtures_path
 
 
@@ -71,7 +69,7 @@ def __init__(self, *args, **kwargs):
         logging.basicConfig()
         logging.getLogger().setLevel(logging.WARNING)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_list_releases(self, mock_variable_get):
         """ Test that list releases returns a list of string with urls.
 
@@ -92,7 +90,7 @@ def test_list_releases(self, mock_variable_get):
                 release = CrossrefMetadataRelease(self.year, self.month)
                 self.assertFalse(release.exists())
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_download_path(self, mock_variable_get):
         """ Test that path of downloaded file is correct for given url.
 
@@ -109,7 +107,7 @@ def test_download_path(self, mock_variable_get):
             path = telescope_path(SubFolder.downloaded, CrossrefMetadataTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.download_file_name), release.download_path)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_extract_path(self, mock_variable_get):
         """ Test that path of decompressed/extracted file is correct for given url.
 
@@ -126,7 +124,7 @@ def test_extract_path(self, mock_variable_get):
             path = telescope_path(SubFolder.extracted, CrossrefMetadataTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.extract_folder), release.extract_path)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_transform_path(self, mock_variable_get):
         """ Test that path of transformed file is correct for given url.
 
@@ -144,7 +142,7 @@ def test_transform_path(self, mock_variable_get):
                                 self.transform_folder)
             self.assertEqual(path, release.transform_path)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_extract_release(self, mock_variable_get):
         """ Test that the release is decompressed as expected.
 
@@ -171,7 +169,7 @@ def test_extract_release(self, mock_variable_get):
             for md5sum, file_path in zip(self.extract_hashes, file_paths):
                 self.assertEqual(md5sum, _hash_file(file_path, algorithm='md5'))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_transform_release(self, mock_variable_get):
         """ Test that the release is transformed as expected.
 
diff --git a/tests/observatory/dags/telescopes/test_fundref.py b/tests/observatory/dags/telescopes/test_fundref.py
index 8b113edc1..bc167c2e2 100644
--- a/tests/observatory/dags/telescopes/test_fundref.py
+++ b/tests/observatory/dags/telescopes/test_fundref.py
@@ -19,27 +19,24 @@
 import shutil
 import unittest
 import xml.etree.ElementTree as ET
-from typing import List, Dict
+from typing import Dict, List
 from unittest.mock import patch
 
 import pendulum
 import vcr
 from click.testing import CliRunner
+from observatory.dags.telescopes.fundref import (FundrefRelease,
+                                                 FundrefTelescope,
+                                                 add_funders_relationships,
+                                                 download_release,
+                                                 extract_release,
+                                                 list_releases,
+                                                 parse_fundref_registry_rdf,
+                                                 recursive_funders,
+                                                 transform_release)
+from observatory.platform.utils.file_utils import _hash_file, gzip_file_crc
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
 
-from observatory.dags.telescopes.fundref import (
-    FundrefRelease,
-    FundrefTelescope,
-    add_funders_relationships,
-    extract_release,
-    list_releases,
-    parse_fundref_registry_rdf,
-    recursive_funders,
-    transform_release,
-    download_release
-)
-from observatory.platform.utils.config_utils import telescope_path, SubFolder
-from observatory.platform.utils.data_utils import _hash_file
-from observatory.platform.utils.gc_utils import gzip_file_crc
 from tests.observatory.test_utils import test_fixtures_path
 
 
@@ -62,8 +59,9 @@ def __init__(self, *args, **kwargs):
         self.fundref_test_path = os.path.join(test_fixtures_path(), 'telescopes', 'fundref.tar.gz')
         self.fundref_test_url = FundrefTelescope.TELESCOPE_DEBUG_URL
         self.fundref_nested_element = ET.Element(
-            '{http://data.crossref.org/fundingdata/xml/schema/grant/grant-1.2/}country',
-            attrib={'{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource': 'http://sws.geonames.org/6252001/'})
+            '{http://data.crossref.org/fundingdata/xml/schema/grant/grant-1.2/}country', attrib={
+                '{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource': 'http://sws.geonames.org/6252001/'
+            })
         self.fundref_nested_element.tail = '\n      '
         self.fundref_test_date = pendulum.datetime(year=3000, month=1, day=1)
         self.fundref_test_download_file_name = 'fundref_3000_01_01.tar.gz'
@@ -79,7 +77,7 @@ def __init__(self, *args, **kwargs):
         logging.basicConfig()
         logging.getLogger().setLevel(logging.WARNING)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_list_releases(self, mock_variable_get):
         """ Test that list releases returns a dictionary with releases.
 
@@ -89,9 +87,7 @@ def test_list_releases(self, mock_variable_get):
         # Mock data variable
         data_path = 'data'
         mock_variable_get.return_value = data_path
-        cassette_path = os.path.join(test_fixtures_path(),
-                                     'vcr_cassettes',
-                                     'list_fundref_releases.yaml')
+        cassette_path = os.path.join(test_fixtures_path(), 'vcr_cassettes', 'list_fundref_releases.yaml')
 
         with CliRunner().isolated_filesystem():
             with vcr.use_cassette(cassette_path):
@@ -101,14 +97,12 @@ def test_list_releases(self, mock_variable_get):
                     self.assertIsInstance(release, FundrefRelease)
                 self.assertEqual(39, len(releases))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_download_release(self, mock_variable_get):
         # Mock data variable
         data_path = 'data'
         mock_variable_get.return_value = data_path
-        cassette_path = os.path.join(test_fixtures_path(),
-                                     'vcr_cassettes',
-                                     'fundref_download_release.yaml')
+        cassette_path = os.path.join(test_fixtures_path(), 'vcr_cassettes', 'fundref_download_release.yaml')
         url = 'https://gitlab.com/crossref/open_funder_registry/-/archive/v1.30/open_funder_registry-v1.30.tar.gz'
         date = pendulum.datetime(year=2020, month=1, day=14, hour=14, minute=51, second=51)
         expected_crc = '87235a1d'
@@ -119,7 +113,7 @@ def test_download_release(self, mock_variable_get):
                 path = download_release(release)
                 self.assertEqual(expected_crc, gzip_file_crc(path))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_download(self, mock_variable_get):
         """ Test that path of downloaded file is correct for given url.
 
@@ -137,7 +131,7 @@ def test_filepath_download(self, mock_variable_get):
             path = telescope_path(SubFolder.downloaded, FundrefTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.fundref_test_download_file_name), file_path_download)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_extract(self, mock_variable_get):
         """ Test that path of decompressed/extracted file is correct for given url.
 
@@ -155,7 +149,7 @@ def test_filepath_extract(self, mock_variable_get):
             path = telescope_path(SubFolder.extracted, FundrefTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.fundref_test_decompress_file_name), file_path_extract)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_transform(self, mock_variable_get):
         """ Test that path of transformed file is correct for given url.
 
@@ -173,7 +167,7 @@ def test_filepath_transform(self, mock_variable_get):
             path = telescope_path(SubFolder.transformed, FundrefTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.fundref_test_transform_file_name), file_path_transform)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_download_release_date(self, mock_variable_get):
         """ Test that the test url contains the correct date.
 
@@ -188,7 +182,7 @@ def test_download_release_date(self, mock_variable_get):
             release = FundrefRelease(self.fundref_test_url, self.fundref_test_date)
             self.assertEqual(self.fundref_test_date, release.date)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_extract_release(self, mock_variable_get):
         """ Test that the release is decompressed as expected.
 
@@ -211,7 +205,7 @@ def test_extract_release(self, mock_variable_get):
             self.assertEqual(self.fundref_test_decompress_file_name, decompress_file_name)
             self.assertEqual(self.fundref_test_decompress_hash, _hash_file(decompress_file_path, algorithm='md5'))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_transform_release(self, mock_variable_get):
         """ Test that the release is transformed as expected.
 
@@ -236,7 +230,7 @@ def test_transform_release(self, mock_variable_get):
             self.assertEqual(self.fundref_test_transform_file_name, transform_file_name)
             self.assertEqual(self.fundref_test_transform_hash, gzip_file_crc(transform_file_path))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_parse_fundref_registry_rdf(self, mock_variable_get):
         """ Test that correct funders list and dictionary are returned when parsing funders registry.
 
@@ -257,61 +251,100 @@ def test_parse_fundref_registry_rdf(self, mock_variable_get):
             funders, funders_by_key = parse_fundref_registry_rdf(decompress_file_path)
 
             self.assertIsInstance(funders, List)
-            self.assertListEqual(funders, [
-                {'funder': 'http://dx.doi.org/10.13039/100000001', 'pre_label': 'National Science Foundation',
-                 'alt_label': ['NSF', 'US NSF', 'USA NSF'],
-                 'narrower': ['http://dx.doi.org/10.13039/100005447', 'http://dx.doi.org/10.13039/100014591',
-                              'http://dx.doi.org/10.13039/100014073', 'http://dx.doi.org/10.13039/100000081',
-                              'http://dx.doi.org/10.13039/100000076', 'http://dx.doi.org/10.13039/100015815',
-                              'http://dx.doi.org/10.13039/100005716', 'http://dx.doi.org/10.13039/100014072',
-                              'http://dx.doi.org/10.13039/100014411', 'http://dx.doi.org/10.13039/100010608',
-                              'http://dx.doi.org/10.13039/100000088', 'http://dx.doi.org/10.13039/100005441',
-                              'http://dx.doi.org/10.13039/100000179', 'http://dx.doi.org/10.13039/100000085',
-                              'http://dx.doi.org/10.13039/100000084', 'http://dx.doi.org/10.13039/100000083',
-                              'http://dx.doi.org/10.13039/100000086', 'http://dx.doi.org/10.13039/100014074',
-                              'http://dx.doi.org/10.13039/100014071'], 'broader': [],
-                 'modified': '2020-02-28T20:45:11.000000', 'created': '2009-07-06T18:53:11.000000',
-                 'funding_body_type': 'National government', 'funding_body_sub_type': 'gov', 'region': 'Americas',
-                 'country': None, 'country_code': 'usa', 'state': None, 'tax_id': None, 'continuation_of': [],
-                 'renamed_as': [], 'replaces': [], 'affil_with': [], 'merged_with': [], 'incorporated_into': [],
-                 'is_replaced_by': [], 'incorporates': [], 'split_into': [], 'status': None, 'merger_of': [],
-                 'split_from': None, 'formly_known_as': None, 'notation': None},
-                {'funder': 'http://dx.doi.org/10.13039/100000002', 'pre_label': 'National Institutes of Health',
-                 'alt_label': ['NIH'],
-                 'narrower': ['http://dx.doi.org/10.13039/100006545', 'http://dx.doi.org/10.13039/100012893',
-                              'http://dx.doi.org/10.13039/100000066', 'http://dx.doi.org/10.13039/100000053',
-                              'http://dx.doi.org/10.13039/100006086', 'http://dx.doi.org/10.13039/100000092',
-                              'http://dx.doi.org/10.13039/100000052', 'http://dx.doi.org/10.13039/100000135',
-                              'http://dx.doi.org/10.13039/100008460', 'http://dx.doi.org/10.13039/100000093',
-                              'http://dx.doi.org/10.13039/100000097', 'http://dx.doi.org/10.13039/100000065',
-                              'http://dx.doi.org/10.13039/100000118', 'http://dx.doi.org/10.13039/100000072',
-                              'http://dx.doi.org/10.13039/100000060', 'http://dx.doi.org/10.13039/100000061',
-                              'http://dx.doi.org/10.13039/100006084', 'http://dx.doi.org/10.13039/100009633',
-                              'http://dx.doi.org/10.13039/100000055', 'http://dx.doi.org/10.13039/100006108',
-                              'http://dx.doi.org/10.13039/100000071', 'http://dx.doi.org/10.13039/100000069',
-                              'http://dx.doi.org/10.13039/100000054', 'http://dx.doi.org/10.13039/100000056',
-                              'http://dx.doi.org/10.13039/100006955', 'http://dx.doi.org/10.13039/100000064',
-                              'http://dx.doi.org/10.13039/100000049', 'http://dx.doi.org/10.13039/100000098',
-                              'http://dx.doi.org/10.13039/100000057', 'http://dx.doi.org/10.13039/100000026',
-                              'http://dx.doi.org/10.13039/100000025', 'http://dx.doi.org/10.13039/100005440',
-                              'http://dx.doi.org/10.13039/100000096', 'http://dx.doi.org/10.13039/100000124',
-                              'http://dx.doi.org/10.13039/100000027', 'http://dx.doi.org/10.13039/100000070',
-                              'http://dx.doi.org/10.13039/100000051', 'http://dx.doi.org/10.13039/100006085',
-                              'http://dx.doi.org/10.13039/100000062', 'http://dx.doi.org/10.13039/100000050'],
-                 'broader': ['http://dx.doi.org/10.13039/100000016'], 'modified': '2020-03-12T22:59:31.000000',
-                 'created': '2009-07-06T18:53:11.000000', 'funding_body_type': 'National government',
-                 'funding_body_sub_type': 'gov', 'region': 'Americas', 'country': None, 'country_code': 'usa',
-                 'state': None, 'tax_id': None, 'continuation_of': [], 'renamed_as': [], 'replaces': [],
-                 'affil_with': [], 'merged_with': [], 'incorporated_into': [], 'is_replaced_by': [], 'incorporates': [],
-                 'split_into': [], 'status': None, 'merger_of': [], 'split_from': None, 'formly_known_as': None,
-                 'notation': None}])
+            self.assertListEqual(funders, [{
+                'funder': 'http://dx.doi.org/10.13039/100000001',
+                'pre_label': 'National Science Foundation',
+                'alt_label': ['NSF', 'US NSF', 'USA NSF'],
+                'narrower': ['http://dx.doi.org/10.13039/100005447', 'http://dx.doi.org/10.13039/100014591',
+                             'http://dx.doi.org/10.13039/100014073', 'http://dx.doi.org/10.13039/100000081',
+                             'http://dx.doi.org/10.13039/100000076', 'http://dx.doi.org/10.13039/100015815',
+                             'http://dx.doi.org/10.13039/100005716', 'http://dx.doi.org/10.13039/100014072',
+                             'http://dx.doi.org/10.13039/100014411', 'http://dx.doi.org/10.13039/100010608',
+                             'http://dx.doi.org/10.13039/100000088', 'http://dx.doi.org/10.13039/100005441',
+                             'http://dx.doi.org/10.13039/100000179', 'http://dx.doi.org/10.13039/100000085',
+                             'http://dx.doi.org/10.13039/100000084', 'http://dx.doi.org/10.13039/100000083',
+                             'http://dx.doi.org/10.13039/100000086', 'http://dx.doi.org/10.13039/100014074',
+                             'http://dx.doi.org/10.13039/100014071'],
+                'broader': [],
+                'modified': '2020-02-28T20:45:11.000000',
+                'created': '2009-07-06T18:53:11.000000',
+                'funding_body_type': 'National government',
+                'funding_body_sub_type': 'gov',
+                'region': 'Americas',
+                'country': None,
+                'country_code': 'usa',
+                'state': None,
+                'tax_id': None,
+                'continuation_of': [],
+                'renamed_as': [],
+                'replaces': [],
+                'affil_with': [],
+                'merged_with': [],
+                'incorporated_into': [],
+                'is_replaced_by': [],
+                'incorporates': [],
+                'split_into': [],
+                'status': None,
+                'merger_of': [],
+                'split_from': None,
+                'formly_known_as': None,
+                'notation': None
+            }, {
+                'funder': 'http://dx.doi.org/10.13039/100000002',
+                'pre_label': 'National Institutes of Health',
+                'alt_label': ['NIH'],
+                'narrower': ['http://dx.doi.org/10.13039/100006545', 'http://dx.doi.org/10.13039/100012893',
+                             'http://dx.doi.org/10.13039/100000066', 'http://dx.doi.org/10.13039/100000053',
+                             'http://dx.doi.org/10.13039/100006086', 'http://dx.doi.org/10.13039/100000092',
+                             'http://dx.doi.org/10.13039/100000052', 'http://dx.doi.org/10.13039/100000135',
+                             'http://dx.doi.org/10.13039/100008460', 'http://dx.doi.org/10.13039/100000093',
+                             'http://dx.doi.org/10.13039/100000097', 'http://dx.doi.org/10.13039/100000065',
+                             'http://dx.doi.org/10.13039/100000118', 'http://dx.doi.org/10.13039/100000072',
+                             'http://dx.doi.org/10.13039/100000060', 'http://dx.doi.org/10.13039/100000061',
+                             'http://dx.doi.org/10.13039/100006084', 'http://dx.doi.org/10.13039/100009633',
+                             'http://dx.doi.org/10.13039/100000055', 'http://dx.doi.org/10.13039/100006108',
+                             'http://dx.doi.org/10.13039/100000071', 'http://dx.doi.org/10.13039/100000069',
+                             'http://dx.doi.org/10.13039/100000054', 'http://dx.doi.org/10.13039/100000056',
+                             'http://dx.doi.org/10.13039/100006955', 'http://dx.doi.org/10.13039/100000064',
+                             'http://dx.doi.org/10.13039/100000049', 'http://dx.doi.org/10.13039/100000098',
+                             'http://dx.doi.org/10.13039/100000057', 'http://dx.doi.org/10.13039/100000026',
+                             'http://dx.doi.org/10.13039/100000025', 'http://dx.doi.org/10.13039/100005440',
+                             'http://dx.doi.org/10.13039/100000096', 'http://dx.doi.org/10.13039/100000124',
+                             'http://dx.doi.org/10.13039/100000027', 'http://dx.doi.org/10.13039/100000070',
+                             'http://dx.doi.org/10.13039/100000051', 'http://dx.doi.org/10.13039/100006085',
+                             'http://dx.doi.org/10.13039/100000062', 'http://dx.doi.org/10.13039/100000050'],
+                'broader': ['http://dx.doi.org/10.13039/100000016'],
+                'modified': '2020-03-12T22:59:31.000000',
+                'created': '2009-07-06T18:53:11.000000',
+                'funding_body_type': 'National government',
+                'funding_body_sub_type': 'gov',
+                'region': 'Americas',
+                'country': None,
+                'country_code': 'usa',
+                'state': None,
+                'tax_id': None,
+                'continuation_of': [],
+                'renamed_as': [],
+                'replaces': [],
+                'affil_with': [],
+                'merged_with': [],
+                'incorporated_into': [],
+                'is_replaced_by': [],
+                'incorporates': [],
+                'split_into': [],
+                'status': None,
+                'merger_of': [],
+                'split_from': None,
+                'formly_known_as': None,
+                'notation': None
+            }])
             self.assertIsInstance(funders_by_key, Dict)
             self.assertEqual(23411, len(funders_by_key))
             for key in funders_by_key:
                 self.assertTrue(key.startswith('http://dx.doi.org/10.13039/'))
                 self.assertIsInstance(funders_by_key[key], Dict)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_recursive_funders(self, mock_variable_get):
         """ Test that correct children/parents list and depth are returned.
 
@@ -344,7 +377,7 @@ def test_recursive_funders(self, mock_variable_get):
             for parent in parents:
                 self.assertIn('parent', parent)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_add_funders_relationships(self, mock_variable_get):
         """ Test that children are added to each funder dictionary.
 
@@ -366,117 +399,341 @@ def test_add_funders_relationships(self, mock_variable_get):
             # add funders relationships
             funders = add_funders_relationships(funders, funders_by_key)
 
-            self.assertListEqual(funders, [
-                {'funder': 'http://dx.doi.org/10.13039/100000001', 'pre_label': 'National Science Foundation',
-                 'alt_label': ['NSF', 'US NSF', 'USA NSF'],
-                 'narrower': ['http://dx.doi.org/10.13039/100005447', 'http://dx.doi.org/10.13039/100014591',
-                              'http://dx.doi.org/10.13039/100014073', 'http://dx.doi.org/10.13039/100000081',
-                              'http://dx.doi.org/10.13039/100000076', 'http://dx.doi.org/10.13039/100015815',
-                              'http://dx.doi.org/10.13039/100005716', 'http://dx.doi.org/10.13039/100014072',
-                              'http://dx.doi.org/10.13039/100014411', 'http://dx.doi.org/10.13039/100010608',
-                              'http://dx.doi.org/10.13039/100000088', 'http://dx.doi.org/10.13039/100005441',
-                              'http://dx.doi.org/10.13039/100000179', 'http://dx.doi.org/10.13039/100000085',
-                              'http://dx.doi.org/10.13039/100000084', 'http://dx.doi.org/10.13039/100000083',
-                              'http://dx.doi.org/10.13039/100000086', 'http://dx.doi.org/10.13039/100014074',
-                              'http://dx.doi.org/10.13039/100014071'], 'broader': [],
-                 'modified': '2020-02-28T20:45:11.000000', 'created': '2009-07-06T18:53:11.000000',
-                 'funding_body_type': 'National government', 'funding_body_sub_type': 'gov', 'region': 'Americas',
-                 'country': None, 'country_code': 'usa', 'state': None, 'tax_id': None, 'continuation_of': [],
-                 'renamed_as': [], 'replaces': [], 'affil_with': [], 'merged_with': [], 'incorporated_into': [],
-                 'is_replaced_by': [], 'incorporates': [], 'split_into': [], 'status': None, 'merger_of': [],
-                 'split_from': None, 'formly_known_as': None, 'notation': None,
-                 'children': [{'funder': 'http://dx.doi.org/10.13039/100005447', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100014591', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100014073', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000081', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000076', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100015815', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100005716', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100014072', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100014411', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100010608', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000088', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100005441', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000179', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000085', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000084', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000083', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000086', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100014074', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100014071', 'name': None, 'children': []}],
-                 'bottom': True, 'parents': [], 'top': False},
-                {'funder': 'http://dx.doi.org/10.13039/100000002', 'pre_label': 'National Institutes of Health',
-                 'alt_label': ['NIH'],
-                 'narrower': ['http://dx.doi.org/10.13039/100006545', 'http://dx.doi.org/10.13039/100012893',
-                              'http://dx.doi.org/10.13039/100000066', 'http://dx.doi.org/10.13039/100000053',
-                              'http://dx.doi.org/10.13039/100006086', 'http://dx.doi.org/10.13039/100000092',
-                              'http://dx.doi.org/10.13039/100000052', 'http://dx.doi.org/10.13039/100000135',
-                              'http://dx.doi.org/10.13039/100008460', 'http://dx.doi.org/10.13039/100000093',
-                              'http://dx.doi.org/10.13039/100000097', 'http://dx.doi.org/10.13039/100000065',
-                              'http://dx.doi.org/10.13039/100000118', 'http://dx.doi.org/10.13039/100000072',
-                              'http://dx.doi.org/10.13039/100000060', 'http://dx.doi.org/10.13039/100000061',
-                              'http://dx.doi.org/10.13039/100006084', 'http://dx.doi.org/10.13039/100009633',
-                              'http://dx.doi.org/10.13039/100000055', 'http://dx.doi.org/10.13039/100006108',
-                              'http://dx.doi.org/10.13039/100000071', 'http://dx.doi.org/10.13039/100000069',
-                              'http://dx.doi.org/10.13039/100000054', 'http://dx.doi.org/10.13039/100000056',
-                              'http://dx.doi.org/10.13039/100006955', 'http://dx.doi.org/10.13039/100000064',
-                              'http://dx.doi.org/10.13039/100000049', 'http://dx.doi.org/10.13039/100000098',
-                              'http://dx.doi.org/10.13039/100000057', 'http://dx.doi.org/10.13039/100000026',
-                              'http://dx.doi.org/10.13039/100000025', 'http://dx.doi.org/10.13039/100005440',
-                              'http://dx.doi.org/10.13039/100000096', 'http://dx.doi.org/10.13039/100000124',
-                              'http://dx.doi.org/10.13039/100000027', 'http://dx.doi.org/10.13039/100000070',
-                              'http://dx.doi.org/10.13039/100000051', 'http://dx.doi.org/10.13039/100006085',
-                              'http://dx.doi.org/10.13039/100000062', 'http://dx.doi.org/10.13039/100000050'],
-                 'broader': ['http://dx.doi.org/10.13039/100000016'], 'modified': '2020-03-12T22:59:31.000000',
-                 'created': '2009-07-06T18:53:11.000000', 'funding_body_type': 'National government',
-                 'funding_body_sub_type': 'gov', 'region': 'Americas', 'country': None, 'country_code': 'usa',
-                 'state': None, 'tax_id': None, 'continuation_of': [], 'renamed_as': [], 'replaces': [],
-                 'affil_with': [], 'merged_with': [], 'incorporated_into': [], 'is_replaced_by': [], 'incorporates': [],
-                 'split_into': [], 'status': None, 'merger_of': [], 'split_from': None, 'formly_known_as': None,
-                 'notation': None,
-                 'children': [{'funder': 'http://dx.doi.org/10.13039/100006545', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100012893', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000066', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000053', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100006086', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000092', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000052', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000135', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100008460', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000093', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000097', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000065', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000118', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000072', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000060', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000061', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100006084', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100009633', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000055', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100006108', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000071', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000069', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000054', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000056', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100006955', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000064', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000049', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000098', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000057', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000026', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000025', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100005440', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000096', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000124', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000027', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000070', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000051', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100006085', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000062', 'name': None, 'children': []},
-                              {'funder': 'http://dx.doi.org/10.13039/100000050', 'name': None, 'children': []}],
-                 'bottom': True,
-                 'parents': [{'funder': 'http://dx.doi.org/10.13039/100000016', 'name': None, 'parent': []}],
-                 'top': True}])
+            self.assertListEqual(funders, [{
+                'funder': 'http://dx.doi.org/10.13039/100000001',
+                'pre_label': 'National Science Foundation',
+                'alt_label': ['NSF', 'US NSF', 'USA NSF'],
+                'narrower': ['http://dx.doi.org/10.13039/100005447', 'http://dx.doi.org/10.13039/100014591',
+                             'http://dx.doi.org/10.13039/100014073', 'http://dx.doi.org/10.13039/100000081',
+                             'http://dx.doi.org/10.13039/100000076', 'http://dx.doi.org/10.13039/100015815',
+                             'http://dx.doi.org/10.13039/100005716', 'http://dx.doi.org/10.13039/100014072',
+                             'http://dx.doi.org/10.13039/100014411', 'http://dx.doi.org/10.13039/100010608',
+                             'http://dx.doi.org/10.13039/100000088', 'http://dx.doi.org/10.13039/100005441',
+                             'http://dx.doi.org/10.13039/100000179', 'http://dx.doi.org/10.13039/100000085',
+                             'http://dx.doi.org/10.13039/100000084', 'http://dx.doi.org/10.13039/100000083',
+                             'http://dx.doi.org/10.13039/100000086', 'http://dx.doi.org/10.13039/100014074',
+                             'http://dx.doi.org/10.13039/100014071'],
+                'broader': [],
+                'modified': '2020-02-28T20:45:11.000000',
+                'created': '2009-07-06T18:53:11.000000',
+                'funding_body_type': 'National government',
+                'funding_body_sub_type': 'gov',
+                'region': 'Americas',
+                'country': None,
+                'country_code': 'usa',
+                'state': None,
+                'tax_id': None,
+                'continuation_of': [],
+                'renamed_as': [],
+                'replaces': [],
+                'affil_with': [],
+                'merged_with': [],
+                'incorporated_into': [],
+                'is_replaced_by': [],
+                'incorporates': [],
+                'split_into': [],
+                'status': None,
+                'merger_of': [],
+                'split_from': None,
+                'formly_known_as': None,
+                'notation': None,
+                'children': [{
+                                 'funder': 'http://dx.doi.org/10.13039/100005447',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100014591',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100014073',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000081',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000076',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100015815',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100005716',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100014072',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100014411',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100010608',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000088',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100005441',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000179',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000085',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000084',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000083',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000086',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100014074',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100014071',
+                                 'name': None,
+                                 'children': []
+                             }],
+                'bottom': True,
+                'parents': [],
+                'top': False
+            }, {
+                'funder': 'http://dx.doi.org/10.13039/100000002',
+                'pre_label': 'National Institutes of Health',
+                'alt_label': ['NIH'],
+                'narrower': ['http://dx.doi.org/10.13039/100006545', 'http://dx.doi.org/10.13039/100012893',
+                             'http://dx.doi.org/10.13039/100000066', 'http://dx.doi.org/10.13039/100000053',
+                             'http://dx.doi.org/10.13039/100006086', 'http://dx.doi.org/10.13039/100000092',
+                             'http://dx.doi.org/10.13039/100000052', 'http://dx.doi.org/10.13039/100000135',
+                             'http://dx.doi.org/10.13039/100008460', 'http://dx.doi.org/10.13039/100000093',
+                             'http://dx.doi.org/10.13039/100000097', 'http://dx.doi.org/10.13039/100000065',
+                             'http://dx.doi.org/10.13039/100000118', 'http://dx.doi.org/10.13039/100000072',
+                             'http://dx.doi.org/10.13039/100000060', 'http://dx.doi.org/10.13039/100000061',
+                             'http://dx.doi.org/10.13039/100006084', 'http://dx.doi.org/10.13039/100009633',
+                             'http://dx.doi.org/10.13039/100000055', 'http://dx.doi.org/10.13039/100006108',
+                             'http://dx.doi.org/10.13039/100000071', 'http://dx.doi.org/10.13039/100000069',
+                             'http://dx.doi.org/10.13039/100000054', 'http://dx.doi.org/10.13039/100000056',
+                             'http://dx.doi.org/10.13039/100006955', 'http://dx.doi.org/10.13039/100000064',
+                             'http://dx.doi.org/10.13039/100000049', 'http://dx.doi.org/10.13039/100000098',
+                             'http://dx.doi.org/10.13039/100000057', 'http://dx.doi.org/10.13039/100000026',
+                             'http://dx.doi.org/10.13039/100000025', 'http://dx.doi.org/10.13039/100005440',
+                             'http://dx.doi.org/10.13039/100000096', 'http://dx.doi.org/10.13039/100000124',
+                             'http://dx.doi.org/10.13039/100000027', 'http://dx.doi.org/10.13039/100000070',
+                             'http://dx.doi.org/10.13039/100000051', 'http://dx.doi.org/10.13039/100006085',
+                             'http://dx.doi.org/10.13039/100000062', 'http://dx.doi.org/10.13039/100000050'],
+                'broader': ['http://dx.doi.org/10.13039/100000016'],
+                'modified': '2020-03-12T22:59:31.000000',
+                'created': '2009-07-06T18:53:11.000000',
+                'funding_body_type': 'National government',
+                'funding_body_sub_type': 'gov',
+                'region': 'Americas',
+                'country': None,
+                'country_code': 'usa',
+                'state': None,
+                'tax_id': None,
+                'continuation_of': [],
+                'renamed_as': [],
+                'replaces': [],
+                'affil_with': [],
+                'merged_with': [],
+                'incorporated_into': [],
+                'is_replaced_by': [],
+                'incorporates': [],
+                'split_into': [],
+                'status': None,
+                'merger_of': [],
+                'split_from': None,
+                'formly_known_as': None,
+                'notation': None,
+                'children': [{
+                                 'funder': 'http://dx.doi.org/10.13039/100006545',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100012893',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000066',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000053',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100006086',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000092',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000052',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000135',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100008460',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000093',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000097',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000065',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000118',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000072',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000060',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000061',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100006084',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100009633',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000055',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100006108',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000071',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000069',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000054',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000056',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100006955',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000064',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000049',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000098',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000057',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000026',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000025',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100005440',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000096',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000124',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000027',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000070',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000051',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100006085',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000062',
+                                 'name': None,
+                                 'children': []
+                             }, {
+                                 'funder': 'http://dx.doi.org/10.13039/100000050',
+                                 'name': None,
+                                 'children': []
+                             }],
+                'bottom': True,
+                'parents': [{
+                                'funder': 'http://dx.doi.org/10.13039/100000016',
+                                'name': None,
+                                'parent': []
+                            }],
+                'top': True
+            }])
 
             for funder in funders:
                 self.assertIsInstance(funder, Dict)
diff --git a/tests/observatory/dags/telescopes/test_geonames.py b/tests/observatory/dags/telescopes/test_geonames.py
index 45e024cb5..a057e72f9 100644
--- a/tests/observatory/dags/telescopes/test_geonames.py
+++ b/tests/observatory/dags/telescopes/test_geonames.py
@@ -23,17 +23,14 @@
 import pendulum
 import vcr
 from click.testing import CliRunner
-
-from observatory.dags.telescopes.geonames import (
-    GeonamesRelease,
-    GeonamesTelescope,
-    transform_release,
-    first_sunday_of_month
-)
+from observatory.dags.telescopes.geonames import GeonamesRelease, \
+    GeonamesTelescope, \
+    first_sunday_of_month, \
+    transform_release
 from observatory.dags.telescopes.geonames import fetch_release_date
-from observatory.platform.utils.config_utils import telescope_path, SubFolder
-from observatory.platform.utils.data_utils import _hash_file
-from observatory.platform.utils.gc_utils import gzip_file_crc
+from observatory.platform.utils.file_utils import _hash_file, gzip_file_crc
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
+
 from tests.observatory.test_utils import test_fixtures_path
 
 
@@ -92,7 +89,7 @@ def test_fetch_release_date(self):
             date = fetch_release_date()
             self.assertEqual(date, pendulum.datetime(year=2020, month=7, day=16, hour=1, minute=22, second=15))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_download(self, mock_variable_get):
         """ Test that path of downloaded file is correct.
 
@@ -110,7 +107,7 @@ def test_filepath_download(self, mock_variable_get):
             path = telescope_path(SubFolder.downloaded, GeonamesTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.geonames_test_download_file_name), file_path_download)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_decompress(self, mock_variable_get):
         """ Test that path of downloaded file is correct.
 
@@ -128,7 +125,7 @@ def test_filepath_decompress(self, mock_variable_get):
             path = telescope_path(SubFolder.extracted, GeonamesTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.geonames_test_decompress_file_name), file_path_decompress)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_extract(self, mock_variable_get):
         """ Test that path of decompressed/extracted file is correct for given url.
 
@@ -146,7 +143,7 @@ def test_filepath_extract(self, mock_variable_get):
             path = telescope_path(SubFolder.extracted, GeonamesTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.geonames_test_decompress_file_name), file_path_extract)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_transform(self, mock_variable_get):
         """ Test that path of transformed file is correct for given url.
 
@@ -164,7 +161,7 @@ def test_filepath_transform(self, mock_variable_get):
             path = telescope_path(SubFolder.transformed, GeonamesTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.geonames_test_transform_file_name), file_path_transform)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_get_blob_name(self, mock_variable_get):
         """ Test get_blob_name.
 
@@ -178,12 +175,11 @@ def test_get_blob_name(self, mock_variable_get):
 
         with CliRunner().isolated_filesystem():
             release = GeonamesRelease(self.geonames_test_date)
-            self.assertEqual(release.get_blob_name(SubFolder.downloaded),
-                             'telescopes/geonames/geonames_3000_01_01.zip')
+            self.assertEqual(release.get_blob_name(SubFolder.downloaded), 'telescopes/geonames/geonames_3000_01_01.zip')
             self.assertEqual(release.get_blob_name(SubFolder.transformed),
                              'telescopes/geonames/geonames_3000_01_01.csv.gz')
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_transform_release(self, mock_variable_get):
         """ Test that the release is transformed as expected.
 
diff --git a/tests/observatory/dags/telescopes/test_grid.py b/tests/observatory/dags/telescopes/test_grid.py
index 07b70b603..d85ff9b23 100644
--- a/tests/observatory/dags/telescopes/test_grid.py
+++ b/tests/observatory/dags/telescopes/test_grid.py
@@ -12,28 +12,57 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Author: James Diprose
+# Author: James Diprose, Aniek Roelofs
 
-import glob
 import logging
 import os
 import unittest
-from typing import List, Dict
+from datetime import datetime
+from unittest.mock import patch
 
+import pendulum
 import vcr
 from click.testing import CliRunner
+from observatory.dags.telescopes.grid import (GridRelease, GridTelescope, list_grid_records, )
+from observatory.platform.utils.file_utils import _hash_file, gzip_file_crc
 
-from observatory.dags.telescopes.grid import (list_grid_releases, download_grid_release, extract_grid_release,
-                                              transform_grid_release)
-from observatory.platform.utils.data_utils import _hash_file
-from observatory.platform.utils.gc_utils import gzip_file_crc
 from tests.observatory.test_utils import test_fixtures_path
 
 
+class MockTaskInstance:
+    def __init__(self, records):
+        """ Construct a MockTaskInstance. This mocks the airflow TaskInstance and is passed as a keyword arg to the
+        make_release function.
+        :param records: List of record info, returned as value during xcom_pull
+        """
+        self.records = records
+
+    def xcom_pull(self, key: str, task_ids: str, include_prior_dates: bool):
+        """ Mock xcom_pull method of airflow TaskInstance.
+        :param key: -
+        :param task_ids: -
+        :param include_prior_dates: -
+        :return: Records list
+        """
+        return self.records
+
+
+def side_effect(arg):
+    values = {
+        'project_id': 'project',
+        'download_bucket_name': 'download-bucket',
+        'transform_bucket_name': 'transform-bucket',
+        'data_path': 'data',
+        'data_location': 'US'
+    }
+    return values[arg]
+
+
+@patch('observatory.platform.utils.template_utils.AirflowVariable.get')
 class TestGrid(unittest.TestCase):
     """ Tests for the functions used by the GRID telescope """
 
-    def __init__(self, *args, **kwargs):
+    def __init__(self, *args, **kwargs, ):
         """ Constructor which sets up variables used by tests.
 
         :param args: arguments.
@@ -41,108 +70,138 @@ def __init__(self, *args, **kwargs):
         """
 
         super(TestGrid, self).__init__(*args, **kwargs)
+        # Telescope instance
+        self.grid = GridTelescope()
 
         # Paths
         self.vcr_cassettes_path = os.path.join(test_fixtures_path(), 'vcr_cassettes')
-        self.list_grid_releases_path = os.path.join(self.vcr_cassettes_path, 'list_grid_releases.yaml')
-        self.list_grid_releases_hash = 'f08806feb7215f78e9fe9292e6319144'
-        self.work_dir = ''
-
-        # GRID Release  2015-09-22
-        self.grid_2015_09_22_article_id = 1553267
-        self.grid_2015_09_22_title = 'GRID release 2015-09-22 in JSON'
-        self.grid_2015_09_22_path = os.path.join(self.vcr_cassettes_path, 'grid_2015-09-22.yaml')
-        self.grid_2015_09_22_hash = 'f8afd953b530bd1a8b6bb0e74ea1991b'
-
-        # GRID Release 2020-03-15
-        self.grid_2020_03_15_article_id = 12022722
-        self.grid_2020_03_15_title = 'GRID release 2020-03-15'
-        self.grid_2020_03_15_path = os.path.join(self.vcr_cassettes_path, 'grid_2020-03-15.yaml')
-        self.grid_2020_03_15_hash = 'c80d8e456597e196c87881371dd80eda'
-        self.grid_2020_03_15_download_expected_hash = '3d300affce1666ac50b8d945c6ca4c5a'
-        self.grid_2020_03_15_transform_version = 'release_2020_03_15'
-        self.grid_2020_03_15_transform_file_name = 'grid_2020_03_15.jsonl.gz'
-        self.grid_2020_03_15_transform_crc = '77bc8585'
+        self.list_grid_records_path = os.path.join(self.vcr_cassettes_path, 'list_grid_releases.yaml')
+
+        # Contains GRID releases 2015-09-22 and 2015-10-09 (format for both is .csv and .json files)
+        with patch('observatory.platform.utils.template_utils.AirflowVariable.get') as mock_variable_get:
+            mock_variable_get.side_effect = side_effect
+            self.grid_run_2015_10_18 = {
+                'start_date': datetime(2015, 10, 11),
+                'end_date': datetime(2015, 10, 18),
+                'records': [{
+                                'article_ids': [1570968, 1570967],
+                                'release_date': pendulum.parse('2015-10-09T00:00:00+00:00')
+                            }, {
+                                'article_ids': [1553266, 1553267],
+                                'release_date': pendulum.parse('2015-09-22T00:00:00+00:00')
+                            }],
+                # there are 2 releases in this run, but use only 1 for testing
+                'release': GridRelease(self.grid.dag_id, ['1553266', '1553267'],
+                                       pendulum.parse('2015-09-22T00:00:00+00:00')),
+                'download_path': os.path.join(self.vcr_cassettes_path, 'grid_2015-09-22.yaml'),
+                'download_hash': 'c6fd33fd31b6699a2f19622f0283f4f1',
+                'extract_hash': 'c6fd33fd31b6699a2f19622f0283f4f1',
+                'transform_crc': 'eb66ae78'
+            }
+
+            # Contains GRID release 2020-03-15 (format is a .zip file, which is more common)
+            self.grid_run_2020_03_27 = {
+                'start_date': datetime(2020, 3, 20),
+                'end_date': datetime(2020, 3, 27),
+                'records': [{
+                                'article_ids': [12022722],
+                                'release_date': pendulum.parse('2020-03-15T00:00:00+00:00')
+                            }],
+                'release': GridRelease(self.grid.dag_id, ['12022722'], pendulum.parse('2020-03-15T00:00:00+00:00')),
+                'download_path': os.path.join(self.vcr_cassettes_path, 'grid_2020-03-15.yaml'),
+                'download_hash': '3d300affce1666ac50b8d945c6ca4c5a',
+                'extract_hash': '5aff68e9bf72e846a867e91c1fa206a0',
+                'transform_crc': '77bc8585'
+            }
+
+        self.grid_runs = [self.grid_run_2015_10_18, self.grid_run_2020_03_27]
 
         # Turn logging to warning because vcr prints too much at info level
         logging.basicConfig()
         logging.getLogger().setLevel(logging.WARNING)
 
-    def test_list_grid_releases(self):
-        """ Check that list grid releases returns a list of dictionaries with keys that we use.
+    def test_list_grid_records(self, mock_variable_get):
+        """ Check that list grid records returns a list of dictionaries with records in the correct format.
 
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
         :return: None.
         """
+        with vcr.use_cassette(self.list_grid_records_path):
+            start_date = self.grid_run_2015_10_18['start_date']
+            end_date = self.grid_run_2015_10_18['end_date']
+            records = list_grid_records(start_date, end_date, GridTelescope.GRID_DATASET_URL)
+            self.assertEqual(self.grid_run_2015_10_18['records'], records)
 
-        with vcr.use_cassette(self.list_grid_releases_path):
-            releases = list_grid_releases()
-            self.assertIsInstance(releases, List)
-            for release in releases:
-                self.assertIsInstance(release, Dict)
-                self.assertIn('id', release)
-                self.assertIn('title', release)
-
-    def test_download_grid_release(self):
-        """ Download a specific GRID release and check that it has expected md5 sum.
+    def test_make_release(self, mock_variable_get):
+        """ Check that make_release returns a list of GridRelease instances.
 
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
         :return: None.
         """
+        mock_variable_get.side_effect = side_effect
 
-        with CliRunner().isolated_filesystem():
-            with vcr.use_cassette(self.grid_2020_03_15_path):
-                files = download_grid_release(self.work_dir, self.grid_2020_03_15_article_id,
-                                              self.grid_2020_03_15_title)
-                # Check that returned downloads has correct length
-                self.assertEqual(1, len(files))
+        for run in self.grid_runs:
+            records = run['records']
 
-                # Check that file has expected hash
-                file_path = files[0]
-                self.assertTrue(os.path.exists(file_path))
-                self.assertEqual(self.grid_2020_03_15_download_expected_hash, _hash_file(file_path, algorithm='md5'))
+            releases = self.grid.make_release(ti=MockTaskInstance(records))
+            self.assertIsInstance(releases, list)
+            for release in releases:
+                self.assertIsInstance(release, GridRelease)
 
-    def test_extract_grid_release(self):
-        """ Test with a GRID release that is an unzipped JSON file (some of the earlier releases).
+    def test_download_release(self, mock_variable_get):
+        """ Download two specific GRID releases and check they have the expected md5 sum.
 
-        :return: None.
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
+        :return:
         """
+        mock_variable_get.side_effect = side_effect
 
         with CliRunner().isolated_filesystem():
-            with vcr.use_cassette(self.grid_2015_09_22_path):
-                files = download_grid_release(self.work_dir, self.grid_2015_09_22_article_id,
-                                              self.grid_2015_09_22_title)
-                release_extracted_path = extract_grid_release(files[0], self.work_dir)
-                self.assertTrue(os.path.exists(release_extracted_path))
-                file_paths = glob.glob(os.path.join(release_extracted_path, '*.json'))
-                self.assertTrue(len(file_paths))
-                self.assertTrue(os.path.isfile(file_paths[0]))
-
-        # Test with GRID release that is a .zip file (more common)
+            for run in self.grid_runs:
+                release = run['release']
+                with vcr.use_cassette(run['download_path']):
+                    downloads = release.download()
+                    # Check that returned downloads has correct length
+                    self.assertEqual(1, len(downloads))
+
+                    # Check that file has expected hash
+                    file_path = downloads[0]
+                    self.assertTrue(os.path.exists(file_path))
+                    self.assertEqual(run['download_hash'], _hash_file(file_path, algorithm='md5'))
+
+    def test_extract_release(self, mock_variable_get):
+        """ Test that the GRID releases are extracted as expected, both for an unzipped json file and a zip file.
+
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
+        :return: None.
+        """
+        mock_variable_get.side_effect = side_effect
+
         with CliRunner().isolated_filesystem():
-            with vcr.use_cassette(self.grid_2020_03_15_path):
-                files = download_grid_release(self.work_dir, self.grid_2020_03_15_article_id,
-                                              self.grid_2020_03_15_title)
-                release_extracted_path = extract_grid_release(files[0], self.work_dir)
-                self.assertTrue(os.path.exists(release_extracted_path))
-                self.assertTrue(os.path.isfile(os.path.join(release_extracted_path, 'grid.json')))
+            for run in self.grid_runs:
+                release = run['release']
+                with vcr.use_cassette(run['download_path']):
+                    release.download()
+                    release.extract()
+
+                    self.assertEqual(1, len(release.extract_files))
+                    self.assertEqual(run['extract_hash'], _hash_file(release.extract_files[0], algorithm='md5'))
 
-    def test_transform_grid_release(self):
-        """ Test that the GRID release is transformed as expected.
+    def test_transform_release(self, mock_variable_get):
+        """ Test that the GRID releases are transformed as expected.
 
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
         :return: None.
         """
+        mock_variable_get.side_effect = side_effect
 
         with CliRunner().isolated_filesystem():
-            with vcr.use_cassette(self.grid_2020_03_15_path):
-                # Get data
-                files = download_grid_release(self.work_dir, self.grid_2020_03_15_article_id,
-                                              self.grid_2020_03_15_title)
-                release_extracted_path = extract_grid_release(files[0], self.work_dir)
-
-                # Transform and check data
-                release_json_file = os.path.join(release_extracted_path, 'grid.json')
-                version, file_name, file_path = transform_grid_release(release_json_file, self.work_dir)
-                self.assertEqual(self.grid_2020_03_15_transform_version, version)
-                self.assertEqual(self.grid_2020_03_15_transform_file_name, file_name)
-                self.assertTrue(os.path.exists(file_path))
-                gzip_crc = gzip_file_crc(file_path)
-                self.assertEqual(self.grid_2020_03_15_transform_crc, gzip_crc)
+            for run in self.grid_runs:
+                release = run['release']
+                with vcr.use_cassette(run['download_path']):
+                    release.download()
+                    release.extract()
+                    release.transform()
+
+                    self.assertEqual(1, len(release.transform_files))
+                    self.assertEqual(run['transform_crc'], gzip_file_crc(release.transform_files[0]))
diff --git a/tests/observatory/dags/telescopes/test_mag.py b/tests/observatory/dags/telescopes/test_mag.py
index fd4d6a63d..e73404f86 100644
--- a/tests/observatory/dags/telescopes/test_mag.py
+++ b/tests/observatory/dags/telescopes/test_mag.py
@@ -26,11 +26,13 @@
 from google.cloud import bigquery
 from google.cloud import storage
 from google.cloud.storage import Blob
-
-from observatory.dags.telescopes.mag import (list_mag_release_files, transform_mag_file, transform_mag_release,
-                                             db_load_mag_release)
-from observatory.platform.utils.data_utils import _hash_file
+from observatory.dags.telescopes.mag import (db_load_mag_release,
+                                             list_mag_release_files,
+                                             transform_mag_file,
+                                             transform_mag_release)
+from observatory.platform.utils.file_utils import _hash_file
 from observatory.platform.utils.gc_utils import upload_files_to_cloud_storage
+
 from tests.observatory.test_utils import random_id, test_fixtures_path
 
 
@@ -82,8 +84,8 @@ def __init__(self, *args, **kwargs):
                     'PaperUrls.txt', 'Papers.txt']
         self.nlp = ['PaperAbstractsInvertedIndex.txt.1', 'PaperAbstractsInvertedIndex.txt.2',
                     'PaperCitationContexts.txt']
-        self.samples = ['CreateDatabase.usql', 'CreateFunctions.usql', 'HIndexDatabricksSample.py',
-                        'ReadMe.pdf', 'ReleaseNote.txt']
+        self.samples = ['CreateDatabase.usql', 'CreateFunctions.usql', 'HIndexDatabricksSample.py', 'ReadMe.pdf',
+                        'ReleaseNote.txt']
 
     def test_list_mag_release_files(self):
         """ Test that list_mag_release_files lists all files in the MAG releases folder.
@@ -238,8 +240,7 @@ def test_bq_load_mag_release(self):
 
                 # Load release into BigQuery
                 result = db_load_mag_release(self.gc_project_id, self.gc_bucket_name, self.gc_data_location,
-                                             release_path,
-                                             self.release_date, dataset_id=dataset_id)
+                                             release_path, self.release_date, dataset_id=dataset_id)
 
                 # Check that all tables have loaded
                 self.assertTrue(result)
diff --git a/tests/observatory/dags/telescopes/test_oapen_metadata.py b/tests/observatory/dags/telescopes/test_oapen_metadata.py
new file mode 100644
index 000000000..71cc41deb
--- /dev/null
+++ b/tests/observatory/dags/telescopes/test_oapen_metadata.py
@@ -0,0 +1,152 @@
+# Copyright 2020 Curtin University
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Author: Aniek Roelofs
+
+import logging
+import os
+import unittest
+from unittest.mock import patch
+
+import pendulum
+import vcr
+from click.testing import CliRunner
+from observatory.dags.telescopes.oapen_metadata import (OapenMetadataRelease, OapenMetadataTelescope)
+from observatory.platform.utils.file_utils import _hash_file, gzip_file_crc
+
+from tests.observatory.test_utils import test_fixtures_path
+
+
+class MockTaskInstance:
+    def __init__(self, start_date: pendulum.Pendulum, end_date: pendulum.Pendulum, first_release: bool):
+        """ Construct a MockTaskInstance. This mocks the airflow TaskInstance and is passed as a keyword arg to the
+        make_release function.
+        :param start_date: Start date of dag run
+        :param end_date: End date of dag run
+        :param first_release: Whether first time a release is processed
+        """
+        self.start_date = start_date
+        self.end_date = end_date
+        self.first_release = first_release
+
+    def xcom_pull(self, key: str, include_prior_dates: bool):
+        """ Mock xcom_pull method of airflow TaskInstance.
+        :param key: -
+        :param include_prior_dates: -
+        :return: Records list
+        """
+        return self.start_date, self.end_date, self.first_release
+
+
+def side_effect(arg):
+    values = {
+        'project_id': 'project',
+        'download_bucket_name': 'download-bucket',
+        'transform_bucket_name': 'transform-bucket',
+        'data_path': 'data',
+        'data_location': 'US'
+    }
+    return values[arg]
+
+
+@patch('observatory.platform.utils.template_utils.AirflowVariable.get')
+class TestOapenMetadata(unittest.TestCase):
+    """ Tests for the functions used by the OapenMetadata telescope """
+
+    def __init__(self, *args, **kwargs, ):
+        """ Constructor which sets up variables used by tests.
+
+        :param args: arguments.
+        :param kwargs: keyword arguments.
+        """
+
+        super(TestOapenMetadata, self).__init__(*args, **kwargs)
+        # Paths
+        self.vcr_cassettes_path = os.path.join(test_fixtures_path(), 'vcr_cassettes')
+        self.download_path = os.path.join(self.vcr_cassettes_path, 'oapen_metadata_2021-02-19.yaml')
+
+        # Telescope instance
+        self.oapen_metadata = OapenMetadataTelescope()
+
+        # Dag run info
+        self.start_date = pendulum.parse('2021-02-12')
+        self.end_date = pendulum.parse('2021-02-19')
+        self.download_hash = '4c261bbfaceafde1854e102d31fcbc0e'
+        self.transform_crc = '34ac5707'
+
+        # Create release instance that is used to test download/transform
+        with patch('observatory.platform.utils.template_utils.AirflowVariable.get') as mock_variable_get:
+            mock_variable_get.side_effect = side_effect
+
+            self.release = OapenMetadataRelease(self.oapen_metadata.dag_id, self.start_date, self.end_date,
+                                                first_release=True)
+
+        # Turn logging to warning because vcr prints too much at info level
+        logging.basicConfig()
+        logging.getLogger().setLevel(logging.WARNING)
+
+    def test_make_release(self, mock_variable_get):
+        """ Check that make_release returns a list of GridRelease instances.
+
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
+        :return: None.
+        """
+        mock_variable_get.side_effect = side_effect
+
+        first_release = self.oapen_metadata.make_release(ti=MockTaskInstance(self.start_date, self.end_date, True))
+        later_release = self.oapen_metadata.make_release(ti=MockTaskInstance(self.start_date, self.end_date, False))
+        for release in [first_release, later_release]:
+            self.assertIsInstance(release, OapenMetadataRelease)
+        self.assertTrue(first_release.first_release)
+        self.assertFalse(later_release.first_release)
+
+    def test_download_release(self, mock_variable_get):
+        """ Download release and check it has the expected md5 sum.
+
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
+        :return:
+        """
+        mock_variable_get.side_effect = side_effect
+
+        with CliRunner().isolated_filesystem():
+            with vcr.use_cassette(self.download_path):
+                success = self.release.download()
+
+                # Check that download is successful
+                self.assertTrue(success)
+
+                # Check that file has expected hash
+                self.assertEqual(1, len(self.release.download_files))
+                file_path = self.release.download_files[0]
+                self.assertTrue(os.path.exists(file_path))
+                self.assertEqual(self.download_hash, _hash_file(file_path, algorithm='md5'))
+
+    def test_transform_release(self, mock_variable_get):
+        """ Test that the release is transformed as expected.
+
+        :param mock_variable_get: Mock result of airflow's Variable.get() function
+        :return: None.
+        """
+        mock_variable_get.side_effect = side_effect
+
+        with CliRunner().isolated_filesystem():
+            with vcr.use_cassette(self.download_path):
+                self.release.download()
+                self.release.transform()
+
+                # Check that file has expected crc
+                self.assertEqual(1, len(self.release.transform_files))
+                file_path = self.release.transform_files[0]
+                self.assertTrue(os.path.exists(file_path))
+                self.assertEqual(self.transform_crc, gzip_file_crc(file_path))
diff --git a/tests/observatory/dags/telescopes/test_open_citations.py b/tests/observatory/dags/telescopes/test_open_citations.py
index 53050965f..ca4d8f6a7 100644
--- a/tests/observatory/dags/telescopes/test_open_citations.py
+++ b/tests/observatory/dags/telescopes/test_open_citations.py
@@ -38,7 +38,7 @@ def setUp(self) -> None:
                                                                'vcr_cassettes',
                                                                'fetch_open_citations_versions.yaml')
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_open_citations_release(self, mock_variable_get):
         """ Test that the getter properties in OpenCitationsRelease work as expected.
 
diff --git a/tests/observatory/dags/telescopes/test_unpaywall.py b/tests/observatory/dags/telescopes/test_unpaywall.py
index 7d9caed96..9d67fb1d1 100644
--- a/tests/observatory/dags/telescopes/test_unpaywall.py
+++ b/tests/observatory/dags/telescopes/test_unpaywall.py
@@ -24,16 +24,14 @@
 import pendulum
 import vcr
 from click.testing import CliRunner
+from observatory.dags.telescopes.unpaywall import (UnpaywallRelease,
+                                                   UnpaywallTelescope,
+                                                   extract_release,
+                                                   list_releases,
+                                                   transform_release)
+from observatory.platform.utils.file_utils import _hash_file
+from observatory.platform.utils.template_utils import SubFolder, telescope_path
 
-from observatory.dags.telescopes.unpaywall import (
-    UnpaywallRelease,
-    UnpaywallTelescope,
-    extract_release,
-    list_releases,
-    transform_release
-)
-from observatory.platform.utils.config_utils import telescope_path, SubFolder
-from observatory.platform.utils.data_utils import _hash_file
 from tests.observatory.test_utils import test_fixtures_path
 
 
@@ -57,7 +55,8 @@ def __init__(self, *args, **kwargs):
         # Unpaywall test release
         self.unpaywall_test_path = os.path.join(test_fixtures_path(), 'telescopes', 'unpaywall.jsonl.gz')
         self.unpaywall_test_file = 'unpaywall_snapshot_3000-01-27T153236.jsonl.gz'
-        self.unpaywall_test_url = 'https://unpaywall-data-snapshots.s3-us-west-2.amazonaws.com/unpaywall_snapshot_3000-01-27T153236.jsonl.gz'
+        self.unpaywall_test_url = 'https://unpaywall-data-snapshots.s3-us-west-2.amazonaws.com' \
+                                  '/unpaywall_snapshot_3000-01-27T153236.jsonl.gz'
         self.unpaywall_test_date = pendulum.datetime(year=3000, month=1, day=27)
         self.unpaywall_test_download_file_name = 'unpaywall_3000_01_27.jsonl.gz'
         self.unpaywall_test_decompress_file_name = 'unpaywall_3000_01_27.jsonl'
@@ -72,7 +71,7 @@ def __init__(self, *args, **kwargs):
         logging.basicConfig()
         logging.getLogger().setLevel(logging.WARNING)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_list_releases(self, mock_variable_get):
         """ Test that list releases returns a list of string with urls.
 
@@ -99,7 +98,7 @@ def test_parse_release_date(self):
         release_date = UnpaywallRelease.parse_release_date(self.unpaywall_test_file)
         self.assertEqual(self.unpaywall_test_date, release_date)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_download(self, mock_variable_get):
         """ Test that path of downloaded file is correct for given url.
 
@@ -116,7 +115,7 @@ def test_filepath_download(self, mock_variable_get):
             path = telescope_path(SubFolder.downloaded, UnpaywallTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.unpaywall_test_download_file_name), release.filepath_download)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_extract(self, mock_variable_get):
         """ Test that path of decompressed/extracted file is correct for given url.
 
@@ -133,7 +132,7 @@ def test_filepath_extract(self, mock_variable_get):
             path = telescope_path(SubFolder.extracted, UnpaywallTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.unpaywall_test_decompress_file_name), release.filepath_extract)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_filepath_transform(self, mock_variable_get):
         """ Test that path of transformed file is correct for given url.
 
@@ -150,7 +149,7 @@ def test_filepath_transform(self, mock_variable_get):
             path = telescope_path(SubFolder.transformed, UnpaywallTelescope.DAG_ID)
             self.assertEqual(os.path.join(path, self.unpaywall_test_transform_file_name), release.filepath_transform)
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_extract_release(self, mock_variable_get):
         """ Test that the release is decompressed as expected.
 
@@ -173,7 +172,7 @@ def test_extract_release(self, mock_variable_get):
             self.assertEqual(self.unpaywall_test_decompress_file_name, decompress_file_name)
             self.assertEqual(self.unpaywall_test_decompress_hash, _hash_file(decompress_file_path, algorithm='md5'))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_transform_release(self, mock_variable_get):
         """ Test that the release is transformed as expected.
 
diff --git a/tests/observatory/platform/utils/test_config_utils.py b/tests/observatory/platform/utils/test_config_utils.py
index 1585852d4..96351e5b3 100644
--- a/tests/observatory/platform/utils/test_config_utils.py
+++ b/tests/observatory/platform/utils/test_config_utils.py
@@ -21,17 +21,13 @@
 
 import pendulum
 from click.testing import CliRunner
+from observatory.platform.utils.config_utils import (find_schema,
+                                                     module_file_path,
+                                                     observatory_home,
+                                                     terraform_credentials_path)
+from observatory.platform.utils.template_utils import SubFolder, telescope_path, test_data_path
 
 import tests.observatory.platform.utils as platform_utils_tests
-from observatory.platform.utils.config_utils import (
-    SubFolder,
-    find_schema,
-    observatory_home,
-    telescope_path,
-    module_file_path,
-    terraform_credentials_path,
-    test_data_path
-)
 from tests.observatory.test_utils import test_fixtures_path
 
 
@@ -145,7 +141,7 @@ def test_find_schema(self):
         self.assertIsNotNone(result)
         self.assertTrue(result.endswith(expected_schema))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_telescope_path(self, mock_variable_get):
         runner = CliRunner()
         with runner.isolated_filesystem():
@@ -175,7 +171,7 @@ def test_telescope_path(self, mock_variable_get):
             self.assertEqual(expected, path_transformed)
             self.assertTrue(os.path.exists(path_transformed))
 
-    @patch('observatory.platform.utils.config_utils.airflow.models.Variable.get')
+    @patch('observatory.platform.utils.template_utils.AirflowVariable.get')
     def test_test_data_path(self, mock_variable_get):
         # Mock test data path variable
         expected_path = '/tmp/test_data'
diff --git a/tests/observatory/platform/utils/test_gc_utils.py b/tests/observatory/platform/utils/test_gc_utils.py
index eaf3a38d1..c866d23e1 100644
--- a/tests/observatory/platform/utils/test_gc_utils.py
+++ b/tests/observatory/platform/utils/test_gc_utils.py
@@ -19,19 +19,26 @@
 from typing import Optional
 
 import pendulum
-from azure.storage.blob import BlobServiceClient, BlobClient
+from azure.storage.blob import BlobClient, BlobServiceClient
 from click.testing import CliRunner
-from google.cloud import storage, bigquery
+from google.cloud import bigquery, storage
 from google.cloud.bigquery import SourceFormat
-
-from observatory.platform.utils.gc_utils import (hex_to_base64_str, crc32c_base64_hash, bigquery_partitioned_table_id,
-                                                 azure_to_google_cloud_storage_transfer, create_bigquery_dataset,
-                                                 load_bigquery_table, upload_file_to_cloud_storage,
-                                                 download_blob_from_cloud_storage, upload_files_to_cloud_storage,
+from observatory.platform.utils.file_utils import crc32c_base64_hash, hex_to_base64_str
+from observatory.platform.utils.gc_utils import (azure_to_google_cloud_storage_transfer,
+                                                 bigquery_partitioned_table_id,
+                                                 bigquery_table_exists,
+                                                 copy_bigquery_table,
+                                                 create_bigquery_dataset,
+                                                 create_bigquery_table_from_query,
+                                                 create_bigquery_view,
+                                                 download_blob_from_cloud_storage,
                                                  download_blobs_from_cloud_storage,
-                                                 table_name_from_blob, run_bigquery_query,
-                                                 copy_bigquery_table, create_bigquery_view, bigquery_table_exists,
-                                                 create_bigquery_table_from_query)
+                                                 load_bigquery_table,
+                                                 run_bigquery_query,
+                                                 table_name_from_blob,
+                                                 upload_file_to_cloud_storage,
+                                                 upload_files_to_cloud_storage)
+
 from tests.observatory.test_utils import random_id
 from tests.observatory.test_utils import test_fixtures_path
 
@@ -197,8 +204,8 @@ def test_load_bigquery_table(self):
             table_name = random_id()
             result = load_bigquery_table(uri, dataset_id, self.gc_bucket_location, table_name,
                                          schema_file_path=schema_path,
-                                         source_format=SourceFormat.NEWLINE_DELIMITED_JSON,
-                                         partition=True, partition_field='dob')
+                                         source_format=SourceFormat.NEWLINE_DELIMITED_JSON, partition=True,
+                                         partition_field='dob')
             self.assertTrue(result)
             self.assertTrue(bigquery_table_exists(self.gc_project_id, dataset_id, table_name))
         finally:
@@ -281,9 +288,9 @@ def test_create_bigquery_table_from_query(self):
 
         try:
             create_bigquery_dataset(self.gc_project_id, dataset_id, data_location)
-            success = create_bigquery_table_from_query(query, self.gc_project_id, dataset_id,
-                                                       table_name, data_location, partition=True,
-                                                       partition_field='date', cluster=True, clustering_fields=['date'])
+            success = create_bigquery_table_from_query(query, self.gc_project_id, dataset_id, table_name, data_location,
+                                                       partition=True, partition_field='date', cluster=True,
+                                                       clustering_fields=['date'])
             self.assertTrue(success)
             self.assertTrue(bigquery_table_exists(self.gc_project_id, dataset_id, table_name))
         finally:
@@ -393,16 +400,12 @@ def test_azure_to_google_cloud_storage_transfer(self):
             az_blob.upload_blob(self.data)
 
             # Transfer data
-            transfer = azure_to_google_cloud_storage_transfer(
-                self.az_storage_account_name,
-                self.az_container_sas_token,
-                self.az_container_name,
-                include_prefixes=[blob_name],
-                gc_project_id=self.gc_project_id,
-                gc_bucket=self.gc_bucket_name,
-                description=f'Test Azure to Google Cloud Storage Transfer '
-                            f'{pendulum.datetime.utcnow().to_datetime_string()}'
-            )
+            transfer = azure_to_google_cloud_storage_transfer(self.az_storage_account_name, self.az_container_sas_token,
+                                                              self.az_container_name, include_prefixes=[blob_name],
+                                                              gc_project_id=self.gc_project_id,
+                                                              gc_bucket=self.gc_bucket_name,
+                                                              description=f'Test Azure to Google Cloud Storage Transfer '
+                                                                          f'{pendulum.datetime.utcnow().to_datetime_string()}')
 
             # Check that transfer was successful
             self.assertTrue(transfer)