Repository: hpcc-systems/HPCC-Platform
Pull request #19456: HPCC-33298 JTrace support sampling configuration
Author: login: rpastrana
Created: 2025-01-24T20:16:27Z | Merged: 2025-03-06T16:47:08Z

- Adds Jtrace sampler configuration
- Adds OTel sampler initialization logic
- Updates JTrace configuration README
- Provides samples
- Jlog trace/span ids suppressed if not sampled (not sure if this is wanted)

<!-- Thank you for submitting a pull request to the HPCC project

 PLEASE READ the following before proceeding.

 This project only accepts pull requests related to open JIRA issues.
 If suggesting a new feature or change, please discuss it in a JIRA issue first.
 If fixing a bug, there should be an issue describing it with steps to reproduce.
 The title line of the pull request (and of each commit within it) should refer to the
 associated issue using the format:

 HPCC-nnnnn Short description of issue

 This will allow the Jira ticket to be automatically updated to refer to this pull request,
 and will ensure that the automatically-generated changelog is properly formatted.
 Where a pull request contains a single commit the pull request title will be set automatically,
 assuming that the commit has followed the proper guidelines.

 Please go over all the following points, and put an `x` in all the boxes that apply. You may find
 it easier to press the 'Create' button first then click on the checkboxes to edit the comment.
-->

## Type of change:
- [ ] This change is a bug fix (non-breaking change which fixes an issue).
- [x] This change is a new feature (non-breaking change which adds functionality).
- [ ] This change improves the code (refactor or other change that does not change the functionality)
- [ ] This change fixes warnings (the fix does not alter the functionality or the generated code)
- [ ] This change is a breaking change (fix or feature that will cause existing behavior to change).
- [ ] This change alters the query API (existing queries will have to be recompiled)

## Checklist:
- [x] My code follows the code style of this project.
  - [ ] My code does not create any new warnings from compiler, build system, or lint.
- [x] The commit message is properly formatted and free of typos.
  - [ ] The commit message title makes sense in a changelog, by itself.
  - [ ] The commit is signed.
- [x] My change requires a change to the documentation.
  - [x] I have updated the documentation accordingly, or...
  - [ ] I have created a JIRA ticket to update the documentation.
  - [ ] Any new interfaces or exported functions are appropriately commented.
- [ ] I have read the CONTRIBUTORS document.
- [ ] The change has been fully tested:
  - [ ] I have added tests to cover my changes.
  - [ ] All new and existing tests passed.
  - [ ] I have checked that this change does not introduce memory leaks.
  - [ ] I have used Valgrind or similar tools to check for potential issues.
- [ ] I have given due consideration to all of the following potential concerns:
  - [ ] Scalability
  - [ ] Performance
  - [ ] Security
  - [ ] Thread-safety
  - [ ] Cloud-compatibility
  - [ ] Premature optimization
  - [ ] Existing deployed queries will not be broken
  - [ ] This change fixes the problem, not just the symptom
  - [ ] The target branch of this pull request is appropriate for such a change.
- [x] There are no similar instances of the same problem that should be addressed
  - [ ] I have addressed them here
  - [ ] I have raised JIRA issues to address them separately
- [ ] This is a user interface / front-end modification
  - [ ] I have tested my changes in multiple modern browsers
  - [ ] The component(s) render as expected

## Smoketest:
- [ ] Send notifications about my Pull Request position in Smoketest queue.
- [ ] Test my draft Pull Request.

## Testing:
<!-- Please describe how this change has been tested.-->

<!-- Thank you for taking the time to submit this pull request and to answer all of the above-->



Diff:
diff --git a/helm/examples/tracing/README.md b/helm/examples/tracing/README.md
index b2c82e0c372..9ffbcf2ac3c 100644
--- a/helm/examples/tracing/README.md
+++ b/helm/examples/tracing/README.md
@@ -13,6 +13,10 @@ All configuration options detailed here are part of the HPCC Systems Helm chart,
 - optAlwaysCreateTraceIds - If true components generate trace/span ids if none are provided by the remote caller.
 - enableDefaultLogExporter - If true, creates a trace exporter outputting to the log using the default options
 - enableOTELDebugLogging - If true, OTel library logging level set to debug, otherwise warning
+- sampling - Defines head sampling strategy. Decision to sample or drop a span or trace is not made by inspecting the trace as a whole. https://opentelemetry.io/docs/concepts/sampling/
+  - type "AlwaysOff" | "AlwaysOn" | "Ratio"
+  - ratio - Required if Ratio sampling type enabled. Represents the ratio of trace/spans to sample Must be a numeric value betwen 0.0 and 1.0. 
+  - parentBased - Optional boolean. Determines if the sampling policy honors the remote root span sampled flag
 - resourceAttributes: - Defines OTel specific resource attribute configuration values
                         which are appended to the runtime OTEL_RESOURCE_ATTRIBUTES. See OTel doc: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration
   - deploymentEnvironment - Defines deployment.environment, which is used to specify
diff --git a/helm/examples/tracing/baremetal-otlp-http-localhost-sample.xml b/helm/examples/tracing/baremetal-otlp-http-localhost-sample.xml
index e430888efa5..c65d00a0b3e 100644
--- a/helm/examples/tracing/baremetal-otlp-http-localhost-sample.xml
+++ b/helm/examples/tracing/baremetal-otlp-http-localhost-sample.xml
@@ -5,6 +5,7 @@
    <exporters consoleDebug="true" endpoint="http://localhost:4318/v1/traces" type="OTLP-HTTP">
     <batch enabled="false" maxQueueSize="4095" scheduledDelayMillis="6001" maxExportBatchSize="511"/>
    </exporters>
+   <sampling type="Ratio" ratio="0.1" parentBased="true"/>
   </tracing>
  </Software>
 </Environment>
\ No newline at end of file
diff --git a/helm/examples/tracing/otlp-http-collector-default-sampled.yaml b/helm/examples/tracing/otlp-http-collector-default-sampled.yaml
new file mode 100644
index 00000000000..b81a910246b
--- /dev/null
+++ b/helm/examples/tracing/otlp-http-collector-default-sampled.yaml
@@ -0,0 +1,10 @@
+global:
+  tracing:
+    sampling:
+      type: Ratio       #Head sampling based on simple ratio
+      ratio: 0.1        #only sample 10% of traces/spans
+      parentBased: true #Sampling strategy based on parent's sampling
+    exporters:
+    - type: OTLP-HTTP
+      endpoint: "localhost:4318/v1/traces"
+      consoleDebug: true
\ No newline at end of file
diff --git a/helm/hpcc/values.schema.json b/helm/hpcc/values.schema.json
index 691015880d4..665221be3c3 100644
--- a/helm/hpcc/values.schema.json
+++ b/helm/hpcc/values.schema.json
@@ -1171,6 +1171,25 @@
           "type": "boolean",
           "description": "If true, sets OTEL library logging to debug level (otherwise set to warning)"
         },
+        "sampling": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "type": "string",
+              "enum": ["AlwaysOff", "AlwaysOn", "Ratio"],
+              "description": "Name of the Head Sampling type AlwaysOff|AlwaysOn|Ratio"
+            },
+            "ratio" : {
+              "type": "number",
+              "description": "Required if sampling type set to Ratio - Represents the sampling ratio value."
+            },
+            "parentBased" : {
+              "type": "boolean",
+              "description": "Optional - Determines if the sampling policy honors the remote root span sampled flag."
+            },
+            "additionalProperties": false
+          }
+        },
         "exporters": {
           "type": "array",
           "description": "List of trace exporters",
diff --git a/system/jlib/jtrace.cpp b/system/jlib/jtrace.cpp
index 8e68ac7cc62..2be2adb75a4 100644
--- a/system/jlib/jtrace.cpp
+++ b/system/jlib/jtrace.cpp
@@ -23,6 +23,12 @@
 #include "opentelemetry/sdk/trace/tracer_context_factory.h" //opentelemetry::sdk::trace::TracerContextFactory::Create(std::move(processors));
 #include "opentelemetry/sdk/trace/simple_processor_factory.h"
 #include "opentelemetry/sdk/trace/batch_span_processor_factory.h"
+#include <opentelemetry/sdk/trace/samplers/always_on_factory.h>
+#include <opentelemetry/sdk/trace/samplers/always_off_factory.h>
+#include <opentelemetry/sdk/trace/samplers/trace_id_ratio_factory.h>
+#include <opentelemetry/sdk/trace/samplers/trace_id_ratio.h>
+#include <opentelemetry/sdk/trace/samplers/always_off.h>
+#include <opentelemetry/sdk/trace/samplers/parent.h>
 #include "opentelemetry/exporters/ostream/span_exporter_factory.h"// auto exporter = opentelemetry::exporter::trace::OStreamSpanExporterFactory::Create();
 #include "opentelemetry/exporters/ostream/common_utils.h"
 #include "opentelemetry/exporters/memory/in_memory_span_exporter_factory.h"
@@ -1382,6 +1388,50 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            if (streq(samplerType, "AlwaysOff"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (streq(samplerType, "AlwaysOn"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (streq(samplerType,"Ratio"))
+            {
+                double ratio = samplerTree->getPropReal("@ratio", -1.0);
+                if (ratio >= 0.0 && ratio <= 1.0)
+                {
+                    sampler.reset(new opentelemetry::sdk::trace::TraceIdRatioBasedSampler(ratio));
+                }
+                else
+                {
+                    OERRLOG("JTrace invalid ratio sampling configuration. Ratio must be between 0.0 and 1.0");
+                }
+            }
+            else
+            {
+                WARNLOG("JTrace initialization: Invalid sampling type configured: '%s'", samplerType);
+            }
+
+            if (sampler && samplerTree->getPropBool("@parentBased", true))
+            {
+                return std::unique_ptr<opentelemetry::sdk::trace::ParentBasedSampler>(new opentelemetry::sdk::trace::ParentBasedSampler(std::move(sampler)));
+            }
+        }
+    }
+
+    return sampler;
+}
+
 void CTraceManager::initTracerProviderAndGlobalInternals(const IPropertyTree * traceConfig)
 {
     /*
@@ -1399,6 +1449,7 @@ void CTraceManager::initTracerProviderAndGlobalInternals(const IPropertyTree * t
         };
 
     std::vector<std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor>> processors;
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
 
     bool enableOTELDebugLogging = false;
 
@@ -1407,6 +1458,8 @@ void CTraceManager::initTracerProviderAndGlobalInternals(const IPropertyTree * t
     bool enableDefaultLogExporter = isDebugBuild();
     if (traceConfig)
     {
+        sampler = createSampler(traceConfig->queryPropTree("sampling"));
+
         IPropertyTree * resourceAttributesTree = traceConfig->queryPropTree("resourceAttributes");
         if (resourceAttributesTree)
         {
@@ -1433,6 +1486,11 @@ void CTraceManager::initTracerProviderAndGlobalInternals(const IPropertyTree * t
         enableDefaultLogExporter = traceConfig->getPropBool("@enableDefaultLogExporter", enableDefaultLogExporter);
     }
 
+    if (!sampler) 
+    {
+        sampler = std::unique_ptr<opentelemetry::sdk::trace::AlwaysOnSampler>(new opentelemetry::sdk::trace::AlwaysOnSampler);
+    }
+
     if (enableDefaultLogExporter)
     {
         //Simple option to create logging to the log file - primarily to aid developers.
@@ -1442,9 +1500,9 @@ void CTraceManager::initTracerProviderAndGlobalInternals(const IPropertyTree * t
 
     auto jtraceResource = opentelemetry::sdk::resource::Resource::Create(resourceAtts);
 
-    // Default is an always-on sampler.
     std::unique_ptr<opentelemetry::sdk::trace::TracerContext> context =
-        opentelemetry::sdk::trace::TracerContextFactory::Create(std::move(processors), jtraceResource);
+        opentelemetry::sdk::trace::TracerContextFactory::Create(std::move(processors), jtraceResource, std::move(sampler));
+
     std::shared_ptr<opentelemetry::trace::TracerProvider> provider =
         opentelemetry::sdk::trace::TracerProviderFactory::Create(std::move(context));
 
@@ -1468,9 +1526,14 @@ Expected Configuration format:
         disabled: true                  #optional - disable OTel tracing
         alwaysCreateGlobalIds : false   #optional - should global ids always be created?
         alwaysCreateTraceIds            #optional - should trace ids always be created?
-        exporters:                       #optional - Controls how trace data is exported/reported
-        -   type: OTLP                    #OS|OTLP|Prometheus|JLOG
-            endpoint: "localhost:4317"    #exporter specific key/value pairs
+        sampling:                       #optional - controls how traces are either suppressed or sampled
+          type:                         #"AlwaysOff" | "AlwaysOn" | "Ratio"
+          ratio:                        #optional - Required if Ratio sampling used, represents the sampling ratio [0.0 - 1.0]
+          parentBased:                  #optional - Sets OTel's parentbased sampling option as defined here:
+                                        #           https://opentelemetry.io/docs/languages/erlang/sampling/#parentbasedsampler
+        exporters:                      #optional - Controls how trace data is exported/reported
+        -   type: OTLP                  #OS|OTLP|Prometheus|JLOG
+            endpoint: "localhost:4317"  #exporter specific key/value pairs
             useSslCredentials: true
             sslCredentialsCACcert: "ssl-certificate"
             batch:                        #optional - Controls span processing style


Reviews:
user: login: jakesmith
body: @rpastrana - looks good, some very minor comments.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-01-29T13:12:17Z
user: login: rpastrana
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-01-29T17:04:53Z
user: login: rpastrana
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-01T01:18:08Z
user: login: rpastrana
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-04T15:26:23Z
user: login: jakesmith
body: @rpastrana - looks good. Please squash.
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-02-06T16:26:28Z
user: login: jakesmith
body: > Do not merge yet, I'm not happy w/ the hang I've noticed (even if avoided by removing call to span->IsRecording

merely commenting, to keep the status of the PR on track (not pending review at this stage).
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-14T15:32:28Z
user: login: jakesmith
body: @rpastrana - looks good. Please squash.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-02-21T10:19:18Z
user: login: jakesmith
body: @rpastrana - looks good afaics. Please squash.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-02-26T11:29:27Z
user: login: ghalliday
body: @rpastrana a few comments - mainly regarding ensuring the configuration makes sense from a user's point of view,  rather than reflecting the underlying implementation.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-03T11:43:26Z
user: login: rpastrana
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T20:02:33Z
user: login: ghalliday
body: A good set of changes, but I think the using a generic "argument" is wrong.
I don't think there are any other examples like it in the helm chart, and it isn't immediately obvious what it means, and requires the user to specify the value using the wrong type.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-04T10:20:48Z
user: login: ghalliday
body: Looks good.  Please squash, and a couple of things to tidy up.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-05T09:25:28Z

Review comments:
diff_hunk: @@ -1350,6 +1359,59 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            const char * samplerArgument = samplerTree->queryProp("@argument");
+            if (strcmp("AlwaysOff", samplerType)==0)
path: system/jlib/jtrace.cpp
user: login: jakesmith
body: trivial: could use jlib's streq
created_at: 2025-01-29T12:53:13Z
author_association: MEMBER
diff_hunk: @@ -1350,6 +1359,59 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            const char * samplerArgument = samplerTree->queryProp("@argument");
+            if (strcmp("AlwaysOff", samplerType)==0)
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (strcmp("AlwaysOn", samplerType)==0)
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (strcmp("Ratio", samplerType)==0)
+            {
+                size_t pos;
+                double ratio = std::stod(samplerArgument, &pos);
path: system/jlib/jtrace.cpp
user: login: jakesmith
body: > const char * samplerArgument = samplerTree->queryProp("@argument");

could move to here, as only use of argument, and do:
```
    double ratio = samplerTree->getPropReal("@argument");
```
created_at: 2025-01-29T13:01:26Z
author_association: MEMBER
diff_hunk: @@ -1350,6 +1359,59 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            const char * samplerArgument = samplerTree->queryProp("@argument");
+            if (strcmp("AlwaysOff", samplerType)==0)
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (strcmp("AlwaysOn", samplerType)==0)
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (strcmp("Ratio", samplerType)==0)
+            {
+                size_t pos;
+                double ratio = std::stod(samplerArgument, &pos);
+                if (ratio < 0 || ratio > 1)
+                {
+                    OERRLOG("JTrace invalid ratio sampler configuration. Ratio must be LE 1.0 or GE 0.0");
+                }
+                else
+                {
+                    sampler.reset(new opentelemetry::sdk::trace::TraceIdRatioBasedSampler(ratio));
+                }
+            }
+            else
+            {
+                WARNLOG("JTrace initialization: Invalid sampler type configured: '%s'", samplerType);
+            }
+
+            if (sampler && samplerTree->getPropBool("@parentBased", true))
+            {
+                return std::unique_ptr<opentelemetry::sdk::trace::ParentBasedSampler>(new opentelemetry::sdk::trace::ParentBasedSampler( std::move(sampler)));
path: system/jlib/jtrace.cpp
user: login: jakesmith
body: trivial/formatting: extra unintended space before std::move in 'ParentBasedSampler( std::move(sampler))'
created_at: 2025-01-29T13:03:02Z
author_association: MEMBER
diff_hunk: @@ -1398,6 +1464,12 @@ void CTraceManager::initTracerProviderAndGlobalInternals(const IPropertyTree * t
         enableDefaultLogExporter = traceConfig->getPropBool("enableDefaultLogExporter", enableDefaultLogExporter);
     }
 
+    if (!sampler) 
+    {
+        sampler = std::unique_ptr<opentelemetry::sdk::trace::AlwaysOnSampler>
+                                (new opentelemetry::sdk::trace::AlwaysOnSampler);
path: system/jlib/jtrace.cpp
user: login: jakesmith
body: to be consistent, as per line 1378:
```
sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
```

created_at: 2025-01-29T13:05:00Z
author_association: MEMBER
diff_hunk: @@ -1350,6 +1359,59 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            const char * samplerArgument = samplerTree->queryProp("@argument");
+            if (strcmp("AlwaysOff", samplerType)==0)
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (strcmp("AlwaysOn", samplerType)==0)
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (strcmp("Ratio", samplerType)==0)
+            {
+                size_t pos;
+                double ratio = std::stod(samplerArgument, &pos);
path: system/jlib/jtrace.cpp
user: login: rpastrana
body: Agreed, it's meant to be a generic argument for future support, but in reality the ratio is likely to be the only  use, will use your suggestion
created_at: 2025-01-29T17:04:53Z
author_association: MEMBER
diff_hunk: @@ -768,9 +768,6 @@ class CSpan : public CInterfaceOf<ISpan>
         if (span == nullptr)
             return false;
 
-        if (!span->IsRecording()) //if not sampled, we shouldn't consider this valid?
path: system/jlib/jtrace.cpp
user: login: rpastrana
body: maybe outside the scope of this issue, but found this causes a hang under certain parentbased configurations -- further analysis is needed, but removing this for now
created_at: 2025-02-01T01:18:08Z
author_association: MEMBER
diff_hunk: @@ -768,9 +768,6 @@ class CSpan : public CInterfaceOf<ISpan>
         if (span == nullptr)
             return false;
 
-        if (!span->IsRecording()) //if not sampled, we shouldn't consider this valid?
path: system/jlib/jtrace.cpp
user: login: rpastrana
body: @jakesmith this amended commit includes the code review changes you requested and approved + this change 
created_at: 2025-02-04T15:26:22Z
author_association: MEMBER
diff_hunk: @@ -1382,6 +1388,56 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            if (streq(samplerType, "AlwaysOff"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (streq(samplerType, "AlwaysOn"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (streq(samplerType,"Ratio"))
+            {
+                double ratio = samplerTree->getPropReal("@argument");
+                if (ratio < 0 || ratio > 1)
+                {
+                    OERRLOG("JTrace invalid ratio sampler configuration. Ratio must be LE 1.0 or GE 0.0");
+                }
+                else
+                {
+                    sampler.reset(new opentelemetry::sdk::trace::TraceIdRatioBasedSampler(ratio));
+                }
+            }
+            else
+            {
+                WARNLOG("JTrace initialization: Invalid sampler type configured: '%s'", samplerType);
+            }
+
+            if (sampler && samplerTree->getPropBool("@parentBased", true))
+            {
+                return std::unique_ptr<opentelemetry::sdk::trace::ParentBasedSampler>(new opentelemetry::sdk::trace::ParentBasedSampler(std::move(sampler)));
+            }
+        }
+    }
+
+    if (!sampler) 
+    {
+        sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+        WARNLOG("JTrace sampler set to 'Always ON' by default!");
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: I am not sure we want this tracing.
This if could be deleted and covered by the default case in the caller.
created_at: 2025-03-03T11:28:37Z
author_association: MEMBER
diff_hunk: @@ -1382,6 +1388,56 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            if (streq(samplerType, "AlwaysOff"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (streq(samplerType, "AlwaysOn"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (streq(samplerType,"Ratio"))
+            {
+                double ratio = samplerTree->getPropReal("@argument");
+                if (ratio < 0 || ratio > 1)
+                {
+                    OERRLOG("JTrace invalid ratio sampler configuration. Ratio must be LE 1.0 or GE 0.0");
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: picky: "Ratio must be LE 1.0 or GE 0.0"
should be "and", and symbols probably clearer
"Ratio must >= 0.0 and <= 1.0" or "between 0.0 and 1.0"
created_at: 2025-03-03T11:30:02Z
author_association: MEMBER
diff_hunk: @@ -1382,6 +1388,56 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            if (streq(samplerType, "AlwaysOff"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (streq(samplerType, "AlwaysOn"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (streq(samplerType,"Ratio"))
+            {
+                double ratio = samplerTree->getPropReal("@argument");
+                if (ratio < 0 || ratio > 1)
+                {
+                    OERRLOG("JTrace invalid ratio sampler configuration. Ratio must be LE 1.0 or GE 0.0");
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: really picky: For an if() statement, it is better to have the normal/expected case first.  It is easier to read.
created_at: 2025-03-03T11:31:25Z
author_association: MEMBER
diff_hunk: @@ -1382,6 +1388,56 @@ std::unique_ptr<opentelemetry::sdk::trace::SpanProcessor> CTraceManager::createP
     return opentelemetry::sdk::trace::SimpleSpanProcessorFactory::Create(std::move(exporter));
 }
 
+static std::unique_ptr<opentelemetry::sdk::trace::Sampler> createSampler(IPropertyTree * samplerTree)
+{
+    std::unique_ptr<opentelemetry::sdk::trace::Sampler> sampler;
+
+    if (samplerTree)
+    {
+        const char * samplerType = samplerTree->queryProp("@type");
+        if (!isEmptyString(samplerType))
+        {
+            if (streq(samplerType, "AlwaysOff"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOffSampler());
+            }
+            else if (streq(samplerType, "AlwaysOn"))
+            {
+                sampler.reset(new opentelemetry::sdk::trace::AlwaysOnSampler());
+            }
+            else if (streq(samplerType,"Ratio"))
+            {
+                double ratio = samplerTree->getPropReal("@argument");
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: add a default value of -1, so that failing to specify a ratio is caught as a problem - rather than silently removing all items.
created_at: 2025-03-03T11:32:39Z
author_association: MEMBER
diff_hunk: @@ -1468,6 +1532,10 @@ Expected Configuration format:
         disabled: true                  #optional - disable OTel tracing
         alwaysCreateGlobalIds : false   #optional - should global ids always be created?
         alwaysCreateTraceIds            #optional - should trace ids always be created?
+        sampler:                        #optional - controls how traces are either suppressed or sampled
+          type:                         #"AlwaysOff" | "AlwaysOn" | "Ratio"
+          argument:                     #optional sampler type configuration value
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: From a user's point of view argument is very generic.  ratio would be better.

You could argue this interface has been defined based on the implementation, rather than what would make sense from the user's viewpoint.  I am not sure if you want to go this way, but you could remove the type field, and just rely on a ratio (or percentage) field.  If it was 0 you would sample none, if 100 you would include all, otherwise you would create a ratio sampler.
created_at: 2025-03-03T11:36:15Z
author_association: MEMBER
diff_hunk: @@ -1468,6 +1532,10 @@ Expected Configuration format:
         disabled: true                  #optional - disable OTel tracing
         alwaysCreateGlobalIds : false   #optional - should global ids always be created?
         alwaysCreateTraceIds            #optional - should trace ids always be created?
+        sampler:                        #optional - controls how traces are either suppressed or sampled
+          type:                         #"AlwaysOff" | "AlwaysOn" | "Ratio"
+          argument:                     #optional sampler type configuration value
+          parentBased:                  #optional sets the sampling decision based on the Span’s parent, or absence of parent, to know which secondary sampler to use.
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: What does this mean? does it mean exclude if the parent is not included in a sample?  What does it mean from the user's perspective?

created_at: 2025-03-03T11:38:09Z
author_association: MEMBER
diff_hunk: @@ -1171,6 +1171,25 @@
           "type": "boolean",
           "description": "If true, sets OTEL library logging to debug level (otherwise set to warning)"
         },
+        "sampler": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "type": "string",
+              "enum": ["AlwaysOff", "AlwaysOn", "Ratio"],
+              "description": "Name of the Head Sampling type AlwaysOff|AlwaysOn|Ratio"
+            },
+            "argument" : {
+              "type": "string",
path: helm/hpcc/values.schema.json
user: login: ghalliday
body: This would naturally be a number - see comments on "argument" v "ratio" below.

created_at: 2025-03-03T11:40:43Z
author_association: MEMBER
diff_hunk: @@ -1171,6 +1171,25 @@
           "type": "boolean",
           "description": "If true, sets OTEL library logging to debug level (otherwise set to warning)"
         },
+        "sampler": {
path: helm/hpcc/values.schema.json
user: login: ghalliday
body: question: Would "sampling" be more intuitive to the user?
created_at: 2025-03-03T11:42:31Z
author_association: MEMBER
diff_hunk: @@ -1171,6 +1171,25 @@
           "type": "boolean",
           "description": "If true, sets OTEL library logging to debug level (otherwise set to warning)"
         },
+        "sampler": {
path: helm/hpcc/values.schema.json
user: login: rpastrana
body: I think the usability improvement would be negligible.

created_at: 2025-03-03T20:02:33Z
author_association: MEMBER
diff_hunk: @@ -1171,6 +1171,25 @@
           "type": "boolean",
           "description": "If true, sets OTEL library logging to debug level (otherwise set to warning)"
         },
+        "sampling": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "type": "string",
+              "enum": ["AlwaysOff", "AlwaysOn", "Ratio"],
+              "description": "Name of the Head Sampling type AlwaysOff|AlwaysOn|Ratio"
+            },
+              "ratio" : {
path: helm/hpcc/values.schema.json
user: login: ghalliday
body: trivial: indented by 2 extra
created_at: 2025-03-05T09:22:24Z
author_association: MEMBER
diff_hunk: @@ -1468,9 +1526,16 @@ Expected Configuration format:
         disabled: true                  #optional - disable OTel tracing
         alwaysCreateGlobalIds : false   #optional - should global ids always be created?
         alwaysCreateTraceIds            #optional - should trace ids always be created?
-        exporters:                       #optional - Controls how trace data is exported/reported
-        -   type: OTLP                    #OS|OTLP|Prometheus|JLOG
-            endpoint: "localhost:4317"    #exporter specific key/value pairs
+        sampling:                       #optional - controls how traces are either suppressed or sampled
+          type:                         #"AlwaysOff" | "AlwaysOn" | "Ratio"
+          argument:                     #optional - Generic argument dependent on sampling type used.
path: system/jlib/jtrace.cpp
user: login: ghalliday
body: needs updating to refer to ratio instead.

created_at: 2025-03-05T09:24:03Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-01-24T20:16:45Z
body: Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-33298

Jirabot Action Result:
Assigning user: rodrigo.pastrana@lexisnexisrisk.com
Workflow Transition To: Merge Pending
Updated PR

author_association: NONE
user: login: rpastrana
created_at: 2025-01-28T22:12:46Z
body: @jakesmith made some wholesale changes in order to support the parentBased sampler annotation just in case you were in the middle of reviewing 
author_association: MEMBER
user: login: rpastrana
created_at: 2025-02-07T18:44:22Z
body: Do not merge yet, I'm not happy w/ the hang I've noticed (even if avoided by removing call to span->IsRecording
author_association: MEMBER
user: login: rpastrana
created_at: 2025-02-20T15:51:45Z
body: @jakesmith I've reviewed, and this is safe to merge. 

author_association: MEMBER
user: login: rpastrana
created_at: 2025-02-21T14:03:16Z
body: @ghalliday ready to be merged

Just noticed the conflicts (not sure how that happened), will resolve asap

Conflicts were resolved
author_association: MEMBER
user: login: rpastrana
created_at: 2025-02-25T18:07:10Z
body: @jakesmith in the process of rebasing due to conflicts, I noticed the sample yaml file was incorrect. Latest commit addresses that minor issue. Please review
author_association: MEMBER
user: login: rpastrana
created_at: 2025-02-26T15:44:12Z
body: Squashed and ready to merge
author_association: MEMBER
user: login: rpastrana
created_at: 2025-03-05T20:40:12Z
body: squashed
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-03-06T16:48:25Z
body: Jirabot Action Result:
Added fix version: 9.8.66
Added fix version: 9.10.12
Workflow Transition: 'Resolve issue'

author_association: NONE