Package: mag-archiver
MAG ArchiverMAG Archiver is an Azure Function App that automatically archives Microsoft Academic Graph (MAG) releases so that they
can be transferred to other cloud services.StatusThis is a proof of concept; the functionality for archiving and compressing each MAG release has not been implemented yet.SetupThe following instructions explain how to setup Mag Archiver.DependenciesInstall Azure CLIInstall Azure Functions Core ToolsCreate an Azure Storage AccountRegion: choose an Azure region that is close to the other cloud provider that you want to transfer the data to.Access tier: hot (need to be able to delete containers without cold storage deletion fees)Create container: mag-snapshotsUnder Blob Service > Lifecycle Management > Code view: add the life-cycle rules from lifecycle-rules.jsonThese rules move blobs to the cold tier after 30 days and delete the blobs after 61 days.Create an Azure Function AppTake note of your function app name, you will need it later.Under Settings > Configuration > Application settings, add the following Application settings (name: value):STORAGE_ACCOUNT_NAME: the name of your storage account.STORAGE_ACCOUNT_KEY: they key for your storage account.TARGET_CONTAINER: mag-snapshotsSubscribe to Microsoft Academic Graph on Azure storageDeploy to AzureTo deploy mag-archiver follow the instructions below.Setup Azure accountMake sure that the Azure account that your Function App is deployed to is set as the default.To do this, list your accounts and copy the id of the account that should be the default account:azaccountlistSet the account to the Azure account that your Function App is deployed to:azaccountset-s<insertyouraccountidhere>Check that the correct account is set, you should see your account show up:azaccountshowDeploy the Function AppClone the project:gitclonegit@github.com:The-Academic-Observatory/mag-archiver.gitEnter the function app folder:cdmag-archiverDeploy the function:funcazurefunctionapppublish<yourfunctionappname>--pythonArchitectureThe architecture of MAG Archiver is illustrated via the deployment and process view diagrams below.Process ViewThe MAG subscription adds each new MAG release as a new Azure Blob storage container in the user's Azure Storage
account.AnAzure Function Appruns every 10 minutes and checks to
see if any new MAG release containers have been added.Metadata about which MAG releases have been discovered and processed are stored in anAzure Table Storagetable calledMagReleases.
TheMagReleasestable is also used used to enable the Apache Airflow MAG workflow to query and find out what MAG
releases have finished processing and where on the Azure blob storage container they can be downloaded from. Ashare access signature (SAS)with read only privileges is used to provide the Apache Airflow MAG workflow with access to the table.When the Function App finds a new MAG release, it copies the files from the container onto a shared container calledmag-snapshotsunder a folder with the same name as the container it was copied from. After copying the files, the
original container is deleted.The Function App copies the MAG files to a shared container so that the Apache Airflow MAG workflow only needs to
hold a single SAS token, one for the shared container. In the future the copying of files by the Cloud Function can be
replaced by a service that compresses the files, as shown in the diagram above.A total of two SAS tokens are shared: one for theMagReleasestable and one for themag-snapshotscontainer.Deployment ViewThe deployment view below shows what services are used and where they are deployed.