Repository: microsoft/ApplicationInsights-node.js

Issue:
setAutoCollectDependencies not working after upgrading from v2 to v3
Hi,

I'm working on migrating existing functions from Azure Function v3 to v4 along with node,js version upgrade. As part of this process, I've upgraded the package applicationinsights library from version "2.9" to "3.5". But after the upgrade, it seems the existing setAutoCollectDependencies false property doesn't work and it is creating hude logs for dependency logs.

  `
			import * as appInsights from 'applicationinsights';
  
  			export class AppInsights implements ILoggerService {
  				private readonly client: appInsights.TelemetryClient;
  				constructor() {
  				this.client = appInsights.defaultClient;
						appInsights
							.setup()
							.setAutoCollectDependencies(false) // seems not working.
							.setAutoCollectPerformance(false, false)
							.start();
						this.client = appInsights.defaultClient;
	
	}

	// Track Events
	/**
	 *
	 * @param trackEvent
	 */
	TrackEvent(trackEvent: IEventLog): void {
		this.client.trackEvent(trackEvent);
	}

	// Track Metric
	/**
	 *
	 * @param trackMetric
	 */

	TrackMetric(trackMetric: IMetricLog): void {
		this.client.trackMetric(trackMetric);
	}

	// Track Trace
	/**
	 *
	 * @param trackMessage
	 */
	TrackTrace(trackMessage: ITraceLog): void {
		this.client.trackTrace(trackMessage);
	}

	// Track Exception
	/**
	 *
	 * @param trackException
	 */
	TrackException(trackException: IExceptionLog): void {
		this.client.trackException(trackException);
	}

	// Track Dependency
	/**
	 *
	 * @param trackDependency
	 */
	TrackDependency(trackDependency: IDependencyLog): void {
		this.client.trackDependency(trackDependency);
	}

	// Track Request
	/**
	 *
	 * @param trackRequest
	 */
	TrackRequest(trackRequest: appInsights.Contracts.RequestTelemetry): void {
		this.client.trackRequest(trackRequest);
	}
}
`
My host.json is as follows

`
{
	"version": "2.0",
	"logging": {
		"applicationInsights": {
			"samplingSettings": {
				"isEnabled": true,
				"excludedTypes": "Request;Dependency" // added Dependency here not working 
			},
			"enableDependencyTracking": false,
			"enablePerformanceCountersCollection": false
		},
// tried with adding below key and not working 
		"logLevel": {
			"default": "Warning",
			"Function": "Warning",
			"Host.Aggregator": "Error",
			"Azure.Core": "None",
			"azure-data-tables": "None"
		}
	},
	"extensionBundle": {
		"id": "Microsoft.Azure.Functions.ExtensionBundle",
		"version": "[4.*, 5.0.0)"
	},
	"extensions": {
		"serviceBus": {
			"maxConcurrentCalls": 2,
			"maxAutoLockRenewalDuration": "00:10:00"
		}
	},
	"functionTimeout": "00:10:00"
}
`

Do I need to update this as a part of version upgrade to avoid the dependency logs? 


Discussion:
@JacksonWeber: @qb-raj-lvmh in version 3.x of the SDK that disabling of dependencies is only going to disable HTTP dependencies. What kind of dependencies are you seeing recorded? It's important to note that it only disables _automatically instrumented_ dependencies. If you manually track dependency, you're still going to see that in application insights. @qb-raj-lvmh: @JacksonWeber I'm getting logs mainly of type "InProc | Microsoft.Data.Tables" and "Azure table". When I downgraded to version 2.9.5, no logs are being generated and everything works as expected. @JacksonWeber: @qb-raj-lvmh I see, looks like disabling dependencies at the moment only disables the HTTP instrumentation's dependencies and not any other OTel instrumentations under the hood.@qb-raj-lvmh: @JacksonWeber so for the time being, I'll be using the downgraded version. Can I expect a new version upgrade with the fix ?@JacksonWeber: @qb-raj-lvmh Fix will be available in the next release.@bencefr: @JacksonWeber ,
This bug seem to cause about 8000 USD bills per month.
Granted, we didn't explicitly disabled collecting dependencies, but they were not sent before so this is quite unexpected.
The related PR doesn't seem the included in version 3.7.0, any ETA for an updated module?
Any suggested urgent workaround?
@JacksonWeber: @bencefr working on a release now https://github.com/microsoft/ApplicationInsights-node.js/pull/1442 however, you should be able to downgrade to the latest version of 2.x to mitigate the issue for now.@bencefr: @JacksonWeber , do you have an ETA releasing 3.7.1?@JacksonWeber: @bencefr it's reliant on a PR for updating this repository to using OpenTelemetry v2.x dependencies https://github.com/microsoft/ApplicationInsights-node.js/pull/1449. I'll be unavailable the next week, but will likely release the week after.@bencefr: @JacksonWeber , please.@JacksonWeber: @bencefr Testing finished, kicking off release today.@JacksonWeber: @bencefr 3.8.0 should be available as of yesterday and this fix is released.

Solution patch:
diff --git a/src/main.ts b/src/main.ts
index 3fef3f9..b2eda00 100644
--- a/src/main.ts
+++ b/src/main.ts
@@ -17,6 +17,7 @@ import { AZURE_MONITOR_STATSBEAT_FEATURES, AzureMonitorOpenTelemetryOptions } fr
 import { ApplicationInsightsConfig } from "./shared/configuration/config";
 import { LogApi } from "./shim/logsApi";
 import { StatsbeatFeature, StatsbeatInstrumentation } from "./shim/types";
+import { RequestSpanProcessor } from "./traces/requestProcessor";
 
 let autoCollectLogs: AutoCollectLogs;
 let exceptions: AutoCollectExceptions;
@@ -31,6 +32,8 @@ export function useAzureMonitor(options?: AzureMonitorOpenTelemetryOptions) {
         instrumentation: StatsbeatInstrumentation.NONE,
         feature: StatsbeatFeature.SHIM
     });
+    // Allows for full filtering of dependency/request spans
+    options.spanProcessors = [new RequestSpanProcessor(options.enableAutoCollectDependencies, options.enableAutoCollectRequests)];
     distroUseAzureMonitor(options);
     const internalConfig = new ApplicationInsightsConfig(options);
     const logApi = new LogApi(logs.getLogger("ApplicationInsightsLogger"));
diff --git a/src/shim/shim-config.ts b/src/shim/shim-config.ts
index f099c16..11f05ad 100644
--- a/src/shim/shim-config.ts
+++ b/src/shim/shim-config.ts
@@ -186,6 +186,14 @@ class Config implements IConfig {
             options.enableAutoCollectExceptions = this.enableAutoCollectExceptions;
         }
 
+        if (typeof (this.enableAutoCollectRequests) === "boolean") {
+            options.enableAutoCollectRequests = this.enableAutoCollectRequests;
+        }
+
+        if (typeof (this.enableAutoCollectDependencies) === "boolean") {
+            options.enableAutoCollectDependencies = this.enableAutoCollectDependencies;
+        }
+
         if (this.enableAutoCollectDependencies === false && this.enableAutoCollectRequests === false) {
             options.instrumentationOptions.http.enabled = false;
         }
diff --git a/src/traces/requestProcessor.ts b/src/traces/requestProcessor.ts
new file mode 100644
index 0000000..779d639
--- /dev/null
+++ b/src/traces/requestProcessor.ts
@@ -0,0 +1,44 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+import { Context, SpanKind, TraceFlags } from "@opentelemetry/api";
+import { ReadableSpan, Span, SpanProcessor } from "@opentelemetry/sdk-trace-base";
+
+/**
+ * Azure Monitor Incoming & Outgoing Request Processor.
+ * @internal
+ */
+export class RequestSpanProcessor implements SpanProcessor {
+    private _enableDependencyTelemetry: boolean;
+    private _enableRequestTelemetry: boolean;
+
+    constructor(enableDependencyTelemetry: boolean, enableRequestTelemetry: boolean) {
+        this._enableDependencyTelemetry = enableDependencyTelemetry;
+        this._enableRequestTelemetry = enableRequestTelemetry;
+    }
+
+    forceFlush(): Promise<void> {
+        return Promise.resolve();
+    }
+
+    onStart(span: Span, _context: Context): void {
+        if (this._enableDependencyTelemetry === false) {
+            if (span.kind === SpanKind.CLIENT || span.kind === SpanKind.PRODUCER) {
+                span.spanContext().traceFlags = TraceFlags.SAMPLED;
+            }
+        }
+        if (this._enableRequestTelemetry === false) {
+            if (span.kind === SpanKind.SERVER || span.kind === SpanKind.CONSUMER) {
+                span.spanContext().traceFlags = TraceFlags.SAMPLED;
+            }
+        }
+    }
+
+    onEnd(span: ReadableSpan): void {
+        return;
+    }
+
+    shutdown(): Promise<void> {
+        return Promise.resolve();
+    }
+}
\ No newline at end of file
diff --git a/src/types.ts b/src/types.ts
index cf897c1..8e3267e 100644
--- a/src/types.ts
+++ b/src/types.ts
@@ -31,6 +31,8 @@ export interface AzureMonitorOpenTelemetryOptions extends DistroOptions {
    * if true performance counters will be collected every second and sent to Azure Monitor
    */
   enableAutoCollectPerformance?: boolean;
+  enableAutoCollectDependencies?: boolean;
+  enableAutoCollectRequests?: boolean;
 }
 
 export interface InstrumentationOptions extends DistroInstrumentationOptions {



Test patch:
diff --git a/test/unitTests/agent/aksLoader.tests.ts b/test/unitTests/agent/aksLoader.tests.ts
index 52e844b..b0faa37 100644
--- a/test/unitTests/agent/aksLoader.tests.ts
+++ b/test/unitTests/agent/aksLoader.tests.ts
@@ -57,7 +57,7 @@ describe("agent/AKSLoader", () => {
         assert.equal(meterProvider["_sharedState"]["metricCollectors"][0]["_metricReader"]["_exporter"].constructor.name, "AzureMonitorMetricExporter");
         let tracerProvider = ((trace.getTracerProvider() as ProxyTracerProvider).getDelegate()) as any;
         assert.equal(tracerProvider.constructor.name, "NodeTracerProvider");
-        assert.equal(tracerProvider["_registeredSpanProcessors"][1]["_exporter"].constructor.name, "AzureMonitorTraceExporter");
+        assert.equal(tracerProvider["_registeredSpanProcessors"][2]["_exporter"].constructor.name, "AzureMonitorTraceExporter");
         let loggerProvider = logs.getLoggerProvider() as any;
         assert.equal(loggerProvider.constructor.name, "LoggerProvider");
         assert.equal(loggerProvider["_sharedState"]["registeredLogRecordProcessors"][1]["_exporter"].constructor.name, "AzureMonitorLogExporter");
diff --git a/test/unitTests/main.tests.ts b/test/unitTests/main.tests.ts
index cbc89e2..f3f616f 100644
--- a/test/unitTests/main.tests.ts
+++ b/test/unitTests/main.tests.ts
@@ -30,8 +30,8 @@ describe("ApplicationInsightsClient", () => {
 
         let tracerProvider = ((trace.getTracerProvider() as ProxyTracerProvider).getDelegate() as any);
         let spanProcessors = tracerProvider["_registeredSpanProcessors"];
-        assert.ok(spanProcessors.length == 3, "wrong number of spanProcessors");
-        otlpExporter = spanProcessors[2]["_exporter"];
+        assert.ok(spanProcessors.length == 4, "wrong number of spanProcessors");
+        otlpExporter = spanProcessors[3]["_exporter"];
         assert.ok(otlpExporter instanceof OTLPTraceExporter, "wrong exporter");
 
         let loggerProvider = ((logs.getLoggerProvider() as LoggerProvider) as any);
diff --git a/test/unitTests/shim/applicationInsights.tests.ts b/test/unitTests/shim/applicationInsights.tests.ts
index 71557cc..ac794f2 100644
--- a/test/unitTests/shim/applicationInsights.tests.ts
+++ b/test/unitTests/shim/applicationInsights.tests.ts
@@ -93,6 +93,15 @@ describe("ApplicationInsights", () => {
             assert.equal(JSON.stringify((appInsights.defaultClient["_options"].instrumentationOptions as InstrumentationOptions).winston), JSON.stringify({ enabled: false }));
         });
 
+        it("should set dependency and request collection", () => {
+            appInsights.setup(connString)
+                .setAutoCollectRequests(true)
+                .setAutoCollectDependencies(false)
+            appInsights.start();
+            assert.equal(appInsights.defaultClient["_options"].enableAutoCollectDependencies, false);
+            assert.equal(appInsights.defaultClient["_options"].enableAutoCollectRequests, true);
+        });
+
         describe("#CorrelationContext", () => {
             it("should return context once AppInsights is intialized", () => {
                 appInsights.setup(connString).start();