# Visualize MongoDB Atlas Database Audit Logs

MongoDB Atlas has advanced security capabilities, and audit logs are one of them. Simply put, enabling audit logs in an Atlas cluster allows you to track what happened in the database by whom and when. 

In this blog post, I’ll walk you through how you can visualize MongoDB Atlas Database Audit Logs with MongoDB Atlas Charts.

## High level architecture

1. In Atlas App Services Values, Atlas Admin API public and private keys and AWS API access key id and secret access have been defined. 
2. aws-sdk node package has been added as a dependency to Atlas Functions.
3. Atlas Data Federation has been configured to query the data in a cloud storage - Amazon S3 of Microsoft Blob Storage bucket. 
4. Atlas Function retrieves both Atlas Admin API and AWS API credentials.
5. Atlas Function calls the Atlas Admin API with the credentials and other relevant parameters (time interval for the audit logs) and fetches the compressed audit logs. 
6. Atlas Function uploads the compressed audit logs as a zip file into a cloud object storage bucket where Atlas has read access. 
7. Atlas Charts visualize the data in S3 through Atlas Data Federation. 
## Prerequisites

The following items must be completed before working on the steps.

* Provision an Atlas cluster where the tier is at least M10. The reason for this is auditing is not supported by free (M0) and shared tier (M2, M5) clusters.
* You need to set up database auditing in the Atlas cluster where you want to track activities.
    * Under the Security section on the left hand menu on the main dashboard, select Advanced. Then toggle Database Auditing and click Audit Filter Settings.
* For the sake of simplicity, check All actions to be tracked in Audit Configuration as shown in the below screenshot. 

* If you don’t have your own load generator to generate load in the database in order to visualize through MongoDB Charts later, you can review this load generator in the Github repository of this blog post.
* Create an app in Atlas App Services that will implement our functions inside it. If you haven’t created an app in Atlas App Services before, please follow this tutorial.
* Create an AWS account along with the following credentials — AWS Access Key and AWS Secret Access Secret.
* Set an AWS IAM Role that has privilege to write into the cloud object storage bucket.
    * Later, Atlas will assume this role to make write operations inside S3 bucket.
## Step 1: configuring credentials
Atlas Admin API allows you to automate your Atlas environment. With a REST client, you can execute a wide variety of management activities such as retrieving audit logs. 

In order to utilize Atlas Admin API, we need to create keys and use these keys later in Atlas Functions. Follow the instructions to create an API key for your project. 

### Creating app services values and app services secrets
After you’ve successfully created public and private keys for the Atlas project, we can store the Atlas Admin API keys and AWS credentials in App Services Values and Secrets.

App Services Values and App Services Secrets are static, server-side constants that you can access or link to from other components of your application. 

In the following part, we’ll create four App Services Values and two App Services Secrets to manage both MongoDB Atlas Admin API and AWS credentials. In order to create App Services Values and Secrets, navigate your App Services app, and on the left hand menu, select **Values**. This will bring you to a page showing the secrets and values available in your App Services app.
#### Setting up Atlas Admin API credentials

In this section, we’ll create two App Services Values and one App Services Secrets to store Atlas Admin API Credentials.

**Value 1: AtlasAdminAPIPublicKey**

This Atlas App Services value keeps the value of the public key of Atlas Admin API. Values should be wrapped in double quotes as shown in the following example.

**Secret 1: AtlasAdminAPIPrivateKey**

This Atlas App Services Secret keeps the value of the private key of Atlas Admin API. You should not wrap the secret in quotation marks. 

**Value 2: AtlasAdminAPIPrivateKeyLinkToSecret**

We can’t directly access secrets in our Atlas Functions. That’s why we have to create a new value and link it to the secret containing our private key. 

Until now, we’ve defined necessary App Services Values and Atlas App Services Secrets to access Atlas Admin API from an App Services App.

In order to access our S3 bucket, we need to utilize AWS SDK. Therefore, we need to do a similar configuration for AWS SDK keys.

### Setting up AWS credentials

In this section, we’ll create two App Services Values and one App Services Secret to store AWS Credentials. Learn how to get your AWS Credentials.

**Value 3: AWSAccessKeyId**

This Atlas App Services Value keeps the value of the access key id of AWS SDK.

**Secret 2: AWSSecretAccessKey**

This Atlas App Services Secret keeps the value of the secret access key of AWS SDK.

**Value 4: AWSSecretAccessKeyLinkToSecret**

This Atlas App Services Value keeps the link of Atlas App Services Secret that keeps the secret key of AWS SDK.

And after you have all these values and secrets as shown below, you can deploy the changes to make it permanent.

## Step 2: adding an external dependency 

An external dependency is an external library that includes logic you'd rather not implement yourself, such as string parsing, convenience functions for array manipulations, and data structure or algorithm implementations. You can upload external dependencies from the npm repository to App Services and then import those libraries into your functions with a `require('external-module')` statement.

In order to work with AWS S3, we will add the official aws-sdk npm package.

In your App Services app, on the left-side menu, navigate to Functions. And then, navigate to the Dependencies pane in this page. 

In order to work with AWS S3, we will add the official **aws-sdk** npm package.

In your App Services app, on the left-side menu, navigate to **Functions**. And then, navigate to the **Dependencies** pane in this page. 

Click **Add Dependency**.

Provide **aws-sdk** as the package name and keep the package version empty. That will install the latest version of aws-sdk node package.

Now, the **aws-sdk** package is ready to be used in our Atlas App Services App.

## Step 3: configuring Atlas Data Federation to consume cloud object storage data through MongoDB Query Language (MQL)

In this tutorial, we’ll not go through all the steps to create a federated database instance in Atlas. Please check out our Atlas Data Federation resources to go through all steps to create Atlas Data Federated Instance.

As an output of this step, we’d expect a ready Federated Database Instance as shown  below.

I have already added the S3 bucket (the name of the bucket is **fuat-sungur-bucket**) that I own into this Federated Database Instance as a data source and I created the collection **auditlogscollection** inside the database **auditlogs** in this Federated Database Instance. 

Now, if I have the files in this S3 bucket (fuat-sungur-bucket), I’ll be able to query it using the MongoDB aggregation framework or Atlas SQL. 

## Step 4: creating an Atlas function to retrieve credentials from Atlas App Services Values and Secrets

Let’s create an Atlas function, give it the name **RetrieveAndUploadAuditLogs**, and choose **System** for authentication.

We also provide the following piece of code in the **Function Editor** and **Run** the function. We’ll see the credentials have been printed out in the console.

```
exports = async function(){
    const atlasAdminAPIPublicKey = context.values.get("AtlasAdminAPIPublicKey");
   const atlasAdminAPIPrivateKey = context.values.get("AtlasAdminAPIPrivateKeyLinkToSecret");
  
   const awsAccessKeyID = context.values.get("AWSAccessKeyID")
   const awsSecretAccessKey = context.values.get("AWSSecretAccessKeyLinkToSecret")
  
   console.log(`Atlas Public + Private Keys: ${atlasAdminAPIPublicKey}, ${atlasAdminAPIPrivateKey}`)
   console.log(`AWS Access Key ID + Secret Access Key: ${awsAccessKeyID}, ${awsSecretAccessKey}`)}

```

## Step 5: retrieving audit logs in the Atlas function 

We now continue to enhance our existing Atlas function, **RetrieveAndUploadAuditLogs**. Now, we’ll execute the HTTP/S request to retrieve audit logs into the Atlas function.

Following piece of code generates an HTTP GET request,calls the relevant Atlas Admin API resource to retrieve audit logs within 1440 minutes, and converts this compressed audit data to the Buffer class in JavaScript.

```
exports = async function(){
    const atlasAdminAPIPublicKey = context.values.get("AtlasAdminAPIPublicKey");
    const atlasAdminAPIPrivateKey = context.values.get("AtlasAdminAPIPrivateKeyLinkToSecret");
  
   const awsAccessKeyID = context.values.get("AWSAccessKeyID")
   const awsSecretAccessKey = context.values.get("AWSSecretAccessKeyLinkToSecret")
  
   console.log(`Atlas Public + Private Keys: ${atlasAdminAPIPublicKey}, ${atlasAdminAPIPrivateKey}`)
   console.log(`AWS Access Key ID + Secret Access Key: ${awsAccessKeyID}, ${awsSecretAccessKey}`)
  
    //////////////////////////////////////////////////////////////////////////////////////////////////
  
   // Atlas Cluster information
   const groupId = '5ca48430014b76f34448bbcf';
   const host = "exchangedata-shard-00-01.5tka5.mongodb.net";
   const logType = "mongodb-audit-log"; // the other option is "mongodb" -> that allows you to download database logs
    // defining startDate and endDate of Audit Logs
   const endDate = new Date();
   const durationInMinutes = 20;
   const durationInMilliSeconds = durationInMinutes * 60 * 1000
   const startDate = new Date(endDate.getTime()-durationInMilliSeconds)
  
   const auditLogsArguments = {
     scheme: 'https',
     host: 'cloud.mongodb.com',
     path: `api/atlas/v1.0/groups/${groupId}/clusters/${host}/logs/${logType}.gz`,
     username: atlasAdminAPIPublicKey,
     password: atlasAdminAPIPrivateKey,
     headers: {'Content-Type': 'application/json'], 'Accept-Encoding': ['application/gzip']},
     digestAuth:true,
     query: {
     "startDate": [Math.round(startDate / 1000).toString()],
     "endDate": [Math.round(endDate / 1000).toString()]
   }
   };
  
   console.log(`Arguments:${JSON.stringify(auditLogsArguments)}`)
  
   const response = await context.http.get(auditLogsArguments)
   auditData = response.body;
   console.log("AuditData:"+(auditData))
   console.log("JS Type:" + typeof auditData)
    // convert it to base64 and then Buffer
   var bufferAuditData = Buffer.from(auditData.toBase64(),'base64')
   console.log("Buffered Audit Data" + bufferAuditData)
 }

```

## Step 6: uploading audit data into the S3 bucket 

Until now, in our Atlas function, we retrieved the audit logs based on the given interval, and now we’ll upload this data into the S3 bucket as a zip file. 

Firstly, we import **aws-sdk** NodeJS library and then configure the credentials for AWS S3. We have already retrieved the AWS credentials from App Services Values and App Services Secrets and assigned those into function variables. 

After that, we configure S3-related parameters, bucket name, key (folder and filename), and body (actual payload that is our audit zip file stored in a Buffer Javascript data type). And finally, we run our upload command [(S3.putObject()).

Here you can find the entire function code:

```
exports = async function(){
    const atlasAdminAPIPublicKey = context.values.get("AtlasAdminAPIPublicKey");
   const atlasAdminAPIPrivateKey = context.values.get("AtlasAdminAPIPrivateKeyLinkToSecret");
  
   const awsAccessKeyID = context.values.get("AWSAccessKeyID")
   const awsSecretAccessKey = context.values.get("AWSSecretAccessKeyLinkToSecret")
  
   console.log(`Atlas Public + Private Keys: ${atlasAdminAPIPublicKey}, ${atlasAdminAPIPrivateKey}`)
   console.log(`AWS Access Key ID + Secret Access Key: ${awsAccessKeyID}, ${awsSecretAccessKey}`)
  
    //////////////////////////////////////////////////////////////////////////////////////////////////
  
   // Atlas Cluster information
   const groupId = '5ca48430014b76f34448bbcf';
   const host = "exchangedata-shard-00-01.5tka5.mongodb.net";
   const logType = "mongodb-audit-log"; // the other option is "mongodb" -> that allows you to download database logs
    // defining startDate and endDate of Audit Logs
   const endDate = new Date();
   const durationInMinutes = 20;
   const durationInMilliSeconds = durationInMinutes * 60 * 1000
   const startDate = new Date(endDate.getTime()-durationInMilliSeconds)
  
   const auditLogsArguments = {
     scheme: 'https',
     host: 'cloud.mongodb.com',
     path: `api/atlas/v1.0/groups/${groupId}/clusters/${host}/logs/${logType}.gz`,
     username: atlasAdminAPIPublicKey,
     password: atlasAdminAPIPrivateKey,
     headers: {'Content-Type': 'application/json'], 'Accept-Encoding': ['application/gzip']},
     digestAuth:true,
     query: {
     "startDate": [Math.round(startDate / 1000).toString()],
     "endDate": [Math.round(endDate / 1000).toString()]
   }
   };
  
   console.log(`Arguments:${JSON.stringify(auditLogsArguments)}`)
  
   const response = await context.http.get(auditLogsArguments)
   auditData = response.body;
   console.log("AuditData:"+(auditData))
   console.log("JS Type:" + typeof auditData)
    // convert it to base64 and then Buffer
   var bufferAuditData = Buffer.from(auditData.toBase64(),'base64')
   console.log("Buffered Audit Data" + bufferAuditData)
    // uploading into S3 
  
   const AWS = require('aws-sdk');
  
   // configure AWS credentials
   const config = {
     accessKeyId: awsAccessKeyID,
     secretAccessKey: awsSecretAccessKey
   };
  
   // configure S3 parameters 
   const fileName= `auditlogs/auditlog-${new Date().getTime()}.gz`
   const S3params = {
     Bucket: "fuat-sungur-bucket",
     Key: fileName,
     Body: bufferAuditData
   };
   const S3 = new AWS.S3(config);
    // create the promise object
   const s3Promise = S3.putObject(S3params).promise();
  
   s3Promise.then(function(data) {
     console.log('Put Object Success');
     return { success: true }
   }).catch(function(err) {
     console.log(err);
     return { success: false, failure: err }
   });
  };

```

After we run the Atlas function, we can check out the S3 bucket and verify that the compressed audit file has been uploaded.

![A folder in an S3 bucket where we store the audit logs

You can find the entire code of the Atlas functions in the dedicated Github repository.

## Step 7: visualizing audit data in MongoDB Charts

First, we need to add our Federated Database Instance that we created in Step 4 into our Charts application that we created in the prerequisites section as a data source. This Federated Database Instance allows us to run queries with the MongoDB aggregation framework on the data that is in the cloud object storage (that is S3, in this case). 

Before doing anything with Atlas Charts, let’s connect to Federated Database Instance and query the audit logs to make sure we have already established the data pipeline correctly. 

```bash 
$ mongosh "mongodb://federateddatabaseinstance0-5tka5.a.query.mongodb.net/myFirstDatabase" --tls --authenticationDatabase admin --username main_user
Enter password: *********
Current Mongosh Log ID: 63066b8cef5a94f0eb34f561
Connecting to:
mongodb://@federateddatabaseinstance0-5tka5.a.query.mongodb.net/myFirstDatabase?directConnection=true&tls=true&authSource=admin&appName=mongosh+1.5.4
Using MongoDB: 5.2.0
Using Mongosh: 1.5.4

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

AtlasDataFederation myFirstDatabase> show dbs
auditlogs  0 B
AtlasDataFederation myFirstDatabase> use auditlogs
switched to db auditlogs
AtlasDataFederation auditlogs> show collections
Auditlogscollection

```

Now, we can get a record from the **auditlogscollection**.

```bash
AtlasDataFederation auditlogs> db.auditlogscollection.findOne()
{
  atype: 'authCheck',
  ts: ISODate("2022-08-24T17:42:44.435Z"),
  uuid: UUID("f5fb1c0a-399b-4308-b67e-732254828d17"),
  local: { ip: '192.168.248.180', port: 27017 },
  remote: { ip: '192.168.248.180', port: 44072 },
  users:  { user: 'mms-automation', db: 'admin' } ],
  roles: [
    { role: 'backup', db: 'admin' },
    { role: 'clusterAdmin', db: 'admin' },
    { role: 'dbAdminAnyDatabase', db: 'admin' },
    { role: 'readWriteAnyDatabase', db: 'admin' },
    { role: 'restore', db: 'admin' },
    { role: 'userAdminAnyDatabase', db: 'admin' }
  ],
  param: {
    command: 'find',
    ns: 'local.clustermanager',
    args: {
      find: 'clustermanager',
      filter: {},
      limit: Long("1"),
      singleBatch: true,
      sort: {},
      lsid: { id: UUID("f49d243f-9c09-4a37-bd81-9ff5a2994f05") },
      '$clusterTime': {
        clusterTime: Timestamp({ t: 1661362964, i: 1 }),
        signature: {
          hash: Binary(Buffer.from("1168fff7240bc852e17c04e9b10ceb78c63cd398", "hex"), 0),
          keyId: Long("7083075402444308485")
        }
      },
      '$db': 'local',
      '$readPreference': { mode: 'primaryPreferred' }
    }
  },
  result: 0
}
AtlasDataFederation auditlogs>

```

Let’s check the audit log of an update operation.

```bash
AtlasDataFederation auditlogs> db.auditlogscollection.findOne({atype:"authCheck", "param.command":"update", "param.ns": "audit_test.orders"})
{
  atype: 'authCheck',
  ts: ISODate("2022-08-24T17:42:44.757Z"),
  uuid: UUID("b7115a0a-c44c-4d6d-b007-a67d887eaea6"),
  local: { ip: '192.168.248.180', port: 27017 },
  remote: { ip: '91.75.0.56', port: 22787 },
  users: [ { user: 'main_user', db: 'admin' } ],
  roles: [
    { role: 'atlasAdmin', db: 'admin' },
    { role: 'backup', db: 'admin' },
    { role: 'clusterMonitor', db: 'admin' },
    { role: 'dbAdminAnyDatabase', db: 'admin' },
    { role: 'enableSharding', db: 'admin' },
    { role: 'readWriteAnyDatabase', db: 'admin' }
  ],
  param: {
    command: 'update',
    ns: 'audit_test.orders',
    args: {
      update: 'orders',
      updates: [
        { q: { _id: 3757 }, u: { '$set': { location: '7186a' } } }
      ],
      ordered: true,
      writeConcern: { w: 'majority' },
      lsid: { id: UUID("a3ace80b-5907-4bf4-a917-be5944ec5a83") },
      txnNumber: Long("509"),
      '$clusterTime': {
        clusterTime: Timestamp({ t: 1661362964, i: 2 }),
        signature: {
          hash: Binary(Buffer.from("1168fff7240bc852e17c04e9b10ceb78c63cd398", "hex"), 0),
          keyId: Long("7083075402444308485")
        }
      },
      '$db': 'audit_test'
    }
  },
  result: 0
}

```

If we are able to see some records, that’s great. Now we can build our [dashboard in Atlas Charts.

You can import this dashboard into your Charts application. You might need to configure the data source name for the Charts for your convenience. In the given dashboard, the datasource was a collection with the name **auditlogscollection** in the database **auditlogs** in the Atlas Federated Database Instance with the name **FederatedDatabaseInstance0**, as shown below.

## Caveats

The following topics can be considered for more effective and efficient audit log analysis.

* You could retrieve logs from all the hosts rather than one node.
    * Therefore you can track the data modifications even in the case of primary node failures.
* You might consider tracking only the relevant activities rather than tracking all the activities in the database. Tracking all the activities in the database might impact the performance.
* You can schedules your triggers.
    * The Atlas function in the example runs once manually, but it can be scheduled via Atlas scheduled triggers. 
        * Then, date intervals (start and end time) for the audit logs for each execution need to be calculated properly. 
* You could improve read efficiency.
    * You might consider partitioning in the S3 bucket by most frequently filtered fields. For further information, please check the docs on optimizing query performance.

## Summary

MongoDB Atlas is not only a database as a service platform to run your MongoDB workloads. It also provides a wide variety of components to build end-to-end solutions.  In this blog post, we explored some of the capabilities of Atlas such as App Services, Atlas Charts, and Atlas Data Federation, and observed how we utilized them to build a real-world scenario. 

Questions on this tutorial? Thoughts, comments? Join the conversation over at the MongoDB Community Forums!