Repository: apify/crawlee
Pull request #2960: feat: Add Segment analytics [internal]
Author: login: HonzaTuron
Labels: name: t-web
description: Issues with this label are in the ownership of the web team.
Created: 2025-05-12T07:55:04Z | Merged: 2025-05-13T11:36:11Z

Adds segment analytics plugin to Crawlee website.

Allows analytics of conversion from Crawlee website to Apify console.


Diff:
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index 38af5d78683d..ed5b432dd87b 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -41,6 +41,7 @@ jobs:
                     yarn build
                 env:
                     APIFY_SIGNING_TOKEN: ${{ secrets.APIFY_SIGNING_TOKEN }}
+                    SEGMENT_TOKEN: ${{ secrets.SEGMENT_TOKEN }}
 
             -   name: Set up GitHub Pages
                 uses: actions/configure-pages@v5
diff --git a/.github/workflows/test-ci.yml b/.github/workflows/test-ci.yml
index af6d32a2ae1a..68a3b39e1a37 100644
--- a/.github/workflows/test-ci.yml
+++ b/.github/workflows/test-ci.yml
@@ -122,6 +122,7 @@ jobs:
                     yarn build
                 env:
                     APIFY_SIGNING_TOKEN: ${{ secrets.APIFY_SIGNING_TOKEN }}
+                    SEGMENT_TOKEN: ${{ secrets.SEGMENT_TOKEN }}
 
     lint:
         name: Lint
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index a4a70d68d0bd..7797a1f31ec4 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -1,4 +1,5 @@
 /* eslint-disable global-require */
+const path = require('path');
 const { externalLinkProcessor } = require('./tools/utils/externalLink');
 
 const packages = [
@@ -209,6 +210,13 @@ module.exports = {
                 },
             };
         },
+        [
+            path.resolve(__dirname, 'src/plugins/docusaurus-plugin-segment'),
+            {
+                writeKey: process.env.SEGMENT_TOKEN,
+                allowedInDev: false,
+            },
+        ],
     ],
     themeConfig:
     /** @type {import('@docusaurus/preset-classic').ThemeConfig} */ ({
diff --git a/website/src/plugins/docusaurus-plugin-segment/index.js b/website/src/plugins/docusaurus-plugin-segment/index.js
new file mode 100644
index 000000000000..9a9929a5267d
--- /dev/null
+++ b/website/src/plugins/docusaurus-plugin-segment/index.js
@@ -0,0 +1,39 @@
+const path = require('path');
+
+module.exports = function (context, options) {
+  const { writeKey, allowedInDev = false } = options;
+
+  return {
+    name: 'docusaurus-plugin-segment',
+
+    getClientModules() {
+      return [path.resolve(__dirname, './segment')];
+    },
+
+    injectHtmlTags() {
+      if (!writeKey) {
+        throw new Error(
+          'You need to specify a Segment writeKey in the plugin options'
+        );
+      }
+
+      if (process.env.NODE_ENV !== 'production' && !allowedInDev) {
+        return {};
+      }
+
+      return {
+        headTags: [
+          {
+            tagName: 'script',
+            innerHTML: `
+            !function(){var i="analytics",analytics=window[i]=window[i]||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","debug","page","screen","once","off","on","addSourceMiddleware","addIntegrationMiddleware","setAnonymousId","addDestinationMiddleware","register"];analytics.factory=function(e){return function(){if(window[i].initialized)return window[i][e].apply(window[i],arguments);var n=Array.prototype.slice.call(arguments);if(["track","screen","alias","group","page","identify"].indexOf(e)>-1){var c=document.querySelector("link[rel='canonical']");n.push({__t:"bpc",c:c&&c.getAttribute("href")||void 0,p:location.pathname,u:location.href,s:location.search,t:document.title,r:document.referrer})}n.unshift(e);analytics.push(n);return analytics}};for(var n=0;n<analytics.methods.length;n++){var key=analytics.methods[n];analytics[key]=analytics.factory(key)}analytics.load=function(key,n){var t=document.createElement("script");t.type="text/javascript";t.async=!0;t.setAttribute("data-global-segment-analytics-key",i);t.src="https://cdn.segment.com/analytics.js/v1/" + key + "/analytics.min.js";var r=document.getElementsByTagName("script")[0];r.parentNode.insertBefore(t,r);analytics._loadOptions=n};analytics._writeKey="${writeKey}";;analytics.SNIPPET_VERSION="5.2.0";
+            analytics.load("${writeKey}", { integrations: { "Segment.io": { apiHost: "analytics.apify.com/v1" } } });
+            analytics.page({ app: 'crawlee', pageType: 'CRAWLEE_PAGE', page_type: 'CRAWLEE_PAGE' });
+            }}();
+            `,
+          },
+        ],
+      };
+    },
+  };
+};
diff --git a/website/src/plugins/docusaurus-plugin-segment/segment.js b/website/src/plugins/docusaurus-plugin-segment/segment.js
new file mode 100644
index 000000000000..8c16cd6e857c
--- /dev/null
+++ b/website/src/plugins/docusaurus-plugin-segment/segment.js
@@ -0,0 +1,14 @@
+import ExecutionEnvironment from '@docusaurus/ExecutionEnvironment';
+
+export default ExecutionEnvironment.canUseDOM ? {
+    onRouteUpdate({ location }) {
+      // Don't track page views on development
+      if (process.env.NODE_ENV === 'production' && window.analytics) {
+        window.analytics.page({
+          path: location.pathname,
+          url: location.href,
+          search: location.search,
+        });
+      }
+    },
+} : null;


Reviews:
user: login: barjin
body: I suppose you want to pass the secret in [test-ci.yml](https://github.com/apify/crawlee/blob/6d636c03c7e2073ddcfc4929eecfd2a598653903/.github/workflows/test-ci.yml#L124) too, that's why the checks are failing. 

We could make `docs.yml` into a `workflow_call` workflow (and call it from `test-ci`), so we don't have to maintain two separate yamls... wdyt @B4nan , is there a reason this is like this?

Other than that, I don't see any issue... thanks @HonzaTuron !
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T08:10:14Z
user: login: B4nan
body: just one nit, otherwise all good
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-12T10:11:20Z
user: login: B4nan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T11:22:13Z
user: login: HonzaTuron
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T11:34:14Z

Review comments:
diff_hunk: @@ -0,0 +1,19 @@
+import ExecutionEnvironment from '@docusaurus/ExecutionEnvironment';
+
+export default (function () {
+  if (!ExecutionEnvironment.canUseDOM) {
+    return null;
+  }
+
+  return {
+    onRouteUpdate({ location }) {
+      // Don't track page views on development
+      if (process.env.NODE_ENV === 'production' && window.analytics) {
+        window.analytics.page({
+          path: location.pathname,
+          url: location.href,
+        });
+      }
+    },
+  };
+})();
path: website/src/plugins/docusaurus-plugin-segment/segment.js
user: login: B4nan
body: do we need the IIFE here? i dont think so, it looks quite weird this way
created_at: 2025-05-12T10:10:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,12 @@
+{
path: website/src/plugins/docusaurus-plugin-segment/package.json
user: login: B4nan
body: i dont think you need this one, since its an inline plugin, it will see the project dependencies
created_at: 2025-05-12T11:22:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,12 @@
+{
path: website/src/plugins/docusaurus-plugin-segment/package.json
user: login: HonzaTuron
body: Yes, IDE was complaining, but works even without `package.json`. Thanks!
created_at: 2025-05-12T11:34:14Z
author_association: CONTRIBUTOR

PR comments:
user: login: B4nan
created_at: 2025-05-12T08:26:17Z
body: the changes in the lock file are quite suspicious, many things are duplicated (all things babel), try running `yarn dedupe` to resolve that. 

> We could make docs.yml into a workflow_call workflow (and call it from test-ci), so we don't have to maintain two separate yamls... wdyt @B4nan , is there a reason this is like this?

docs PR checks are not running the same as the docs.yaml, one only checks if we can build the docs, the other also deploys them (so sets up GH pages environment etc). we could also only build the latest docs snapshot in the PR check to make things much faster, for the deployment workflow, we need to build it all.
author_association: MEMBER
user: login: barjin
created_at: 2025-05-12T08:44:45Z
body: To be honest, we could use `workflow_call` inputs to parametrize all these things. Also, looking closer at the diff, the `turbo` cache doesn't really do anything here, does it? Then it's all about the publishing steps, which can be easily fenced off with `if:`.

![image](https://github.com/user-attachments/assets/72d9d7a4-dc98-4067-833c-d03ac9ff9e61)

Anyway, sorry to hijack this PR, I'll make a separate issue soon.

author_association: MEMBER
user: login: B4nan
created_at: 2025-05-12T08:46:23Z
body: yeah not saying it cannot be done, its just not as simple as calling the docs workflow, since that would also deploy, and ideally we would optimize the PR docs check so its only considering the latest version and not the old docs snapshots
author_association: MEMBER
user: login: B4nan
created_at: 2025-05-12T08:53:26Z
body: looking at the `@twilio-labs/docusaurus-plugin-segment` package, it states a direct dependency on some old docusaurus v2 beta version, we might need to use yarn resolutions here to force the correct docusaurus version, now it would install two of them for no reason.

or better, just write your own tiny plugin based on it, since that one is apparently wrong (should use peer dependencies and not lock a specific version, not to mention it shouldn't be a beta version in the first place)
author_association: MEMBER
user: login: B4nan
created_at: 2025-05-14T13:21:02Z
body: duh, this is not a crawlee feature, next time please pay attention to the PR type, features will end up in a changelog
author_association: MEMBER