Capture clickstream data with Azure Application Insights
By: Kris Peeters
Tags: Azure, Analytics, Clickstream, JavaScript

Raw clickstream data is a valuable data source in almost any analytics project. But it’s not always easy to capture. Free tools like Google Analytics often don’t expose raw clicks. And enterprise tools like Google Analytics premium or Adobe Analytics can be expensive, and in large companies often require a procurement process.

We discovered a nice feature on Azure where you can get these raw clickstreams in near-real-time, at an affordable price. It is part of a service called Application Insights, which can be used for much more than just website click data. But that is out-of-scope for this blog. It is a standard Azure service and thus part of your monthly cloud cost and so typically doesn’t require a separate procurement process. In this blog, we go through the steps needed to set this up. You can find more information in this quickstart from Microsoft. and on the github page for the javascript SDK of Application Insights.

Set up Application Insights

Well, that’s really simple. Go to your Azure Portal, search for Application Insights, and click New. You get something like this.

The important thing is the Instrumentation Key. That is unique. Next, you change the `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx` in the following code snippet with that Instrumentation Key:



var appInsights=window.appInsights||function(a){

function b(a){c[a]=function(){var b=arguments;c.queue.push(function(){c[a].apply(c,b)})}}var c={config:a},d=document,e=window;setTimeout(function(){var b=d.createElement("script");b.src=a.url||"

}({

instrumentationKey: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"

}); var appInsights=window.appInsights||function(a){function b(a){c[a]=function(){var b=arguments;c.queue.push(function(){c[a].apply(c,b)})}}var c={config:a},d=document,e=window;setTimeout(function(){var b=d.createElement("script");b.src=a.url||" <a href="https://az416426.vo.msecnd.net/scripts/a/ai.0.js" class="fc hm" rel="noopener nofollow">https://az416426.vo.msecnd.net/scripts/a/ai.0.js</a> ",d.getElementsByTagName("script")[0].parentNode.appendChild(b)});try{c.cookie=d.cookie}catch(a){}c.queue=[];for(var f=["Event","Exception","Metric","PageView","Trace","Dependency"];f.length;)b("track"+f.pop());if(b("setAuthenticatedUserContext"),b("clearAuthenticatedUserContext"),b("startTrackEvent"),b("stopTrackEvent"),b("startTrackPage"),b("stopTrackPage"),b("flush"),!a.disableExceptionTracking){f="onerror",b("_"+f);var g=e[f];e[f]=function(a,b,d,e,h){var i=g&&g(a,b,d,e,h);return!0!==i&&c["_"+f](a,b,d,e,h),i}}return c}({instrumentationKey: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"}); window.appInsights=appInsights,appInsights.queue&&0===appInsights.queue.length&&appInsights.trackPageView();

</script>

Once you add this script to your website, you should see data coming in. Eg, on the side-panel, go to Users. You should see something like this:

Export raw data to blob storage

On that same side panel, you also see the item ‘Continuous Export’. There, you can, as you guessed, configure where to store all the raw data:

The important data types to export are Page View and Custom Event, if you use those. The result is a whole bunch of .blob files in blob storage:

Each blob is an ndjson file which means it’s very verbose and slow to query. but it does contain all the raw information you need. Plus, you can attach custom dimensions if you want to track additional information. The first thing we do, is turn all those thousands of blob files into a few parquet files. Parquet files take only very little storage space, have strongly typed fields and are super efficient to query.

Automate with Terraform

Clicking around on web UIs to build infrastructure is not cool anymore. We love Infrastructure as Code, so we’ve also automated the creation of Application Insights:

resource "azurerm_application_insights" "clickstream" {

application_type = "Web"

location = "${data.azurerm_storage_account.storage_account.location}"

name = "${var.resource_prefix}-app-insights-clickstream-${var.env}"

resource_group_name = "${var.resource_group_name}"

}

And then we output the instrumentation key so it can be used when building the website:

output "clickstream_instrumentation_key" {

value = "${azurerm_application_insights.clickstream.instrumentation_key}"



}

Note we didn’t automate the “Continuous Export” in Terraform, because it’s not available yet. Terraform support is not perfect on Azure. But we do prefer it over the very verbose ARM templates. Just like we prefer Terraform over AWS CloudFormation. Of course, in the end, you choose the tools that work best for your particular use case.

What are the costs?

To be honest, that was a bit unclear when we started it. But apparently, it costs 2.13 EUR/GB. For a relatively large Belgian website, that results in a monthly cost of about EUR 250. That’s not super cheap, but it is definitely not as expensive as some of the alternatives, which can be well over 100K per year.

But what if we’re not on Azure?

That is OK. We love all the clouds. We often import Google Analytics data from the Google Cloud into other clouds. You could do the same with Azure Application Insights. Just run this in Azure, and copy the blobs from Blob Storage to your favorite cloud provider. There is a cost to getting data out, but it shouldn’t be significant. That cost is often over-estimated. If it does become a concern, you can always zip all blobs before sending them over, or even already turn them into Parquet files on Azure.