Repository: hpcc-systems/HPCC-Platform
Pull request #19732: HPCC-33809 RoxiePipe trace context propagation
Author: login: rpastrana
Created: 2025-04-11T15:54:44Z | Merged: 2025-04-17T15:04:40Z

- Adds new trace parent option (-tp)
- Injects traceparent to outgoing roxie bound request
- Adds new roxiepipetraced ecl testing case

<!-- Thank you for submitting a pull request to the HPCC project

 PLEASE READ the following before proceeding.

 This project only accepts pull requests related to open JIRA issues.
 If suggesting a new feature or change, please discuss it in a JIRA issue first.
 If fixing a bug, there should be an issue describing it with steps to reproduce.
 The title line of the pull request (and of each commit within it) should refer to the
 associated issue using the format:

 HPCC-nnnnn Short description of issue

 This will allow the Jira ticket to be automatically updated to refer to this pull request,
 and will ensure that the automatically-generated changelog is properly formatted.
 Where a pull request contains a single commit the pull request title will be set automatically,
 assuming that the commit has followed the proper guidelines.

 Please go over all the following points, and put an `x` in all the boxes that apply. You may find
 it easier to press the 'Create' button first then click on the checkboxes to edit the comment.
-->

## Type of change:
- [ ] This change is a bug fix (non-breaking change which fixes an issue).
- [ ] This change is a new feature (non-breaking change which adds functionality).
- [x] This change improves the code (refactor or other change that does not change the functionality)
- [ ] This change fixes warnings (the fix does not alter the functionality or the generated code)
- [ ] This change is a breaking change (fix or feature that will cause existing behavior to change).
- [ ] This change alters the query API (existing queries will have to be recompiled)

## Checklist:
- [x] My code follows the code style of this project.
  - [ ] My code does not create any new warnings from compiler, build system, or lint.
- [x] The commit message is properly formatted and free of typos.
  - [ ] The commit message title makes sense in a changelog, by itself.
  - [ ] The commit is signed.
- [ ] My change requires a change to the documentation.
  - [ ] I have updated the documentation accordingly, or...
  - [ ] I have created a JIRA ticket to update the documentation.
  - [ ] Any new interfaces or exported functions are appropriately commented.
- [x] I have read the CONTRIBUTORS document.
- [ ] The change has been fully tested:
  - [x] I have added tests to cover my changes.
  - [ ] All new and existing tests passed.
  - [ ] I have checked that this change does not introduce memory leaks.
  - [ ] I have used Valgrind or similar tools to check for potential issues.
- [ ] I have given due consideration to all of the following potential concerns:
  - [ ] Scalability
  - [ ] Performance
  - [ ] Security
  - [ ] Thread-safety
  - [ ] Cloud-compatibility
  - [ ] Premature optimization
  - [ ] Existing deployed queries will not be broken
  - [ ] This change fixes the problem, not just the symptom
  - [ ] The target branch of this pull request is appropriate for such a change.
- [ ] There are no similar instances of the same problem that should be addressed
  - [ ] I have addressed them here
  - [ ] I have raised JIRA issues to address them separately
- [ ] This is a user interface / front-end modification
  - [ ] I have tested my changes in multiple modern browsers
  - [ ] The component(s) render as expected

## Smoketest:
- [ ] Send notifications about my Pull Request position in Smoketest queue.
- [ ] Test my draft Pull Request.

## Testing:
<!-- Please describe how this change has been tested.-->

<!-- Thank you for taking the time to submit this pull request and to answer all of the above-->



Diff:
diff --git a/roxie/roxiepipe/roxiepipe.cpp b/roxie/roxiepipe/roxiepipe.cpp
index 7dd068d552d..52ef9ea949e 100644
--- a/roxie/roxiepipe/roxiepipe.cpp
+++ b/roxie/roxiepipe/roxiepipe.cpp
@@ -79,10 +79,11 @@ class RoxieThread : public Thread
     Owned<ISmartSocket> roxieSock;
     StringBuffer query;
     StringBuffer resultName;
+    StringBuffer traceParent;
 
     QueueOf<MemoryBuffer, false> outputQ;
 
-    char *inputBuffer = nullptr;;
+    char *inputBuffer = nullptr;
     unsigned inputLen = 0;
     unsigned curInput = 0;
 
@@ -93,7 +94,7 @@ class RoxieThread : public Thread
     unsigned recordsRead = 0;
     unsigned recordsWritten = 0;
 
-    RoxieThread(const char *_query, const char *_resultName) : query(_query), resultName(_resultName)
+    RoxieThread(const char *_query, const char *_resultName, const char * _traceParent) : query(_query), resultName(_resultName), traceParent(_traceParent)
     {
         recordsRead = recordsWritten = 0;
         bytesPerQuery = recordsPerQuery * in_width;
@@ -104,6 +105,22 @@ class RoxieThread : public Thread
     void sendQuery()
     {
         MemoryBuffer sendBuffer;
+        if (!traceParent.isEmpty())
+        {
+            try
+            {
+                Owned<IPTree> queryTree = createPTreeFromXMLString(query, ipt_none, ptr_none);
+                ensurePTree(queryTree, "_trace");
+                queryTree->setProp("_trace/traceparent", traceParent.str());
+                toXML(queryTree, query.clear());
+            }
+            catch (IException * e)
+            {
+                StringBuffer s;
+                printf("Warning: could not inject trace parent: '%s'", e->errorMessage(s).str());
+                e->Release();
+            }
+        }
         unsigned queryLen = query.length();
         unsigned revQueryLen = queryLen;
         _WINREV(revQueryLen);
@@ -476,6 +493,8 @@ void usage()
     printf("  -to   N      Read timeout\n");
     printf("  -wu   wuid   Workunit id\n");
     printf("  -l    name   Logfile name\n");
+    printf("  -tp   trace  Provide trace context (traceparent)\n");
+    printf("               Sample format: 00-beca49ca8f3138a2842e5cf21402bfff-4b960b3e4647da3f-01\n");
     printf("  -ssl         Use SSL connections\n");
 }
 
@@ -487,6 +506,7 @@ int main(int argc, char *argv[])
     StringBuffer query;
     StringBuffer resultName;
     StringBuffer logFile;
+    StringBuffer traceParent;
     unsigned numThreads = 2;
     bool retryMode = true;
 
@@ -607,6 +627,12 @@ int main(int argc, char *argv[])
             if (i <argc)
                 logFile.clear().append(argv[i]);
         }
+        else if (stricmp(argv[i], "-tp") == 0)   //trace parent provided
+        {
+            i++;
+            if (i < argc)
+                traceParent.clear().append(argv[i]);
+        }
 #if 0
         else if (stricmp(argv[i], "-d") == 0)
         {
@@ -692,7 +718,7 @@ int main(int argc, char *argv[])
             {
                 for (i=0; i<(int)numThreads; i++)
                 {
-                    rt[i] = new RoxieThread(query.str(), resultName.str());
+                    rt[i] = new RoxieThread(query.str(), resultName.str(), traceParent.str());
                     PROGLOG("Starting thread %d", i);
                     rt[i]->start(true);
                 }
diff --git a/testing/regress/ecl/roxiepipetraced.ecl b/testing/regress/ecl/roxiepipetraced.ecl
new file mode 100644
index 00000000000..1ec2d71b4f3
--- /dev/null
+++ b/testing/regress/ecl/roxiepipetraced.ecl
@@ -0,0 +1,86 @@
+/*##############################################################################
+
+    HPCC SYSTEMS software Copyright (C) 2025 HPCC Systems®.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+############################################################################## */
+
+//nokey
+
+import lib_logging; // only needed if lib_logging.Logging.getTraceSpanHeader used below
+string TargetIP := '.' : stored('TargetIP');
+
+NameRec := RECORD
+  string10 First;
+  string15 Last;
+END;
+
+AddressRec := RECORD
+  string10 City;
+  string2 State;
+  integer4 ZipCode;
+END;
+
+PersonRec := RECORD
+  NameRec Name;
+  AddressRec Address;
+END;
+
+pipe_send := DATASET([{{'Joe', 'Doe'}, {'Fresno', 'CA', 11111}},
+{{'Jason', 'Jones'}, {'Tempe', 'AZ', 22222}},
+{{'Becky', 'Lopez'}, {'New York', 'NY', 33333}},
+{{'Roderic', 'Lykke'}, {'Lubbock', 'TX', 44444}},
+{{'Rina', 'Yonkers'}, {'Denver', 'CO', 55555}},
+{{'Laverna', 'Campa'}, {'Detroit', 'MI', 66666}},
+{{'Shantell', 'Mattera'}, {'Toledo', 'OH', 77777}},
+{{'John', 'Smith'}, {'Boise', 'ID', 88888}},
+{{'Jane','Smith'}, {'Smallville', 'KY', 99999}}], PersonRec);
+
+/*  NOTES:
+	roxiepipe = the executable we're calling
+	iw = Input Width
+	vip = Virtual IP
+	t = # threads
+	ow = output width
+	b = # records in a batch
+	mr = max retries
+	h = ??? -- the URL to the Roxie batch service
+	r = name of the results dataset (case sensitive!)
+  tp = traceparent (trace context) propagatated to roxie
+       can be hardcoded or current trace (if available) can be fetched dynamically
+       via lib_logging.Logging.getTraceSpanHeader()
+	q = query
+*/
+
+
+string CurrentTraceContext := lib_logging.Logging.getTraceSpanHeader(); //Fetches current active trace (if available on target engine).
+// This is the remote call to the 'roxie_echo' service.
+pipe_recv := PIPE(pipe_send,
+    'roxiepipe' +
+    ' -iw ' + SIZEOF(PersonRec) +
+    ' -t 1' +
+    ' -ow ' + SIZEOF(PersonRec) +
+    ' -b 3' +
+    ' -mr 2' +
+    ' -h ' + TargetIP + ':9876 ' +
+    ' -r Peeps' +
+    ' -tp ' +                   //Propagates trace context to target roxie.
+      CurrentTraceContext  +    //Dynamically fetched traceContext from target engine
+                                //Otherwise, a hard-coded trace contexted can be constructed
+                                //in the following format: 00-0c483675c6c887d9fb49885c0f2ba116-379563f3fcd56bc9-01
+    ' -q "<roxie_echo format=\'raw\'><peeps id=\'id\' format=\'raw\'></peeps></roxie_echo>"'
+, PersonRec);
+
+OUTPUT(CurrentTraceContext);
+OUTPUT(pipe_recv);
+


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: 

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

<details>
<summary>Files not reviewed (1)</summary>

* **testing/regress/ecl/roxiepipetraced.ecl**: Language not supported
</details>

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**roxie/roxiepipe/roxiepipe.cpp:108**
* Since 'traceParent' is a StringBuffer object and not a pointer, checking against nullptr is unnecessary. Simplify the condition to just check if the string is empty.
```
if (traceParent != nullptr && !traceParent.isEmpty())
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T15:55:05Z
user: login: ghalliday
body: @rpastrana looks good.  one trivial comment, and one request for clarification.

state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T11:02:36Z
user: login: rpastrana
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T14:53:37Z
user: login: ghalliday
body: Minor- updates needed so regression test passes.
Also, please update the conclusion in the jira with what was implemented, and create a documentation jira to ensure it gets added to documentation/we circulate information about how to use it.
 
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-15T10:26:43Z
user: login: ghalliday
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-17T15:04:22Z

Review comments:
diff_hunk: @@ -104,6 +105,22 @@ class RoxieThread : public Thread
     void sendQuery()
     {
         MemoryBuffer sendBuffer;
+        if (traceParent != nullptr && !traceParent.isEmpty())
path: roxie/roxiepipe/roxiepipe.cpp
user: login: ghalliday
body: As copilot points out - not need to check for nullptr.
created_at: 2025-04-14T11:00:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,84 @@
+/*##############################################################################
+
+    HPCC SYSTEMS software Copyright (C) 2025 HPCC Systems®.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+############################################################################## */
+
+//nothor
+//nohthor
+import lib_logging; // only needed if lib_logging.Logging.getTraceSpanHeader used below
+string TargetIP := '.' : stored('TargetIP');
+
+NameRec := RECORD
+  string10 First;
+  string15 Last;
+END;
+
+AddressRec := RECORD
+  string10 City;
+  string2 State;
+  integer4 ZipCode;
+END;
+
+PersonRec := RECORD
+  NameRec Name;
+  AddressRec Address;
+END;
+
+pipe_send := DATASET([{{'Joe', 'Doe'}, {'Fresno', 'CA', 11111}},
+{{'Jason', 'Jones'}, {'Tempe', 'AZ', 22222}},
+{{'Becky', 'Lopez'}, {'New York', 'NY', 33333}},
+{{'Roderic', 'Lykke'}, {'Lubbock', 'TX', 44444}},
+{{'Rina', 'Yonkers'}, {'Denver', 'CO', 55555}},
+{{'Laverna', 'Campa'}, {'Detroit', 'MI', 66666}},
+{{'Shantell', 'Mattera'}, {'Toledo', 'OH', 77777}},
+{{'John', 'Smith'}, {'Boise', 'ID', 88888}},
+{{'Jane','Smith'}, {'Smallville', 'KY', 99999}}], PersonRec);
+
+/*  NOTES:
+	roxiepipe = the executable we're calling
+	iw = Input Width
+	vip = Virtual IP
+	t = # threads
+	ow = output width
+	b = # records in a batch
+	mr = max retries
+	h = ??? -- the URL to the Roxie batch service
+	r = name of the results dataset (case sensitive!)
+  tp = traceparent (trace context) propagatated to roxie
+       can be hardcoded or current trace (if available) can be fetched dynamically
+       via lib_logging.Logging.getTraceSpanHeader()
+	q = query
+*/
+
+
+// This is the remote call to the 'roxie_echo' service.
+pipe_recv := PIPE(pipe_send,
+    'roxiepipe' +
+    ' -iw ' + SIZEOF(PersonRec) +
+    ' -t 1' +
+    ' -ow ' + SIZEOF(PersonRec) +
+    ' -b 3' +
+    ' -mr 2' +
+    ' -h ' + TargetIP + ':9876 ' +
+    ' -r Peeps' +
+    ' -tp ' + lib_logging.Logging.getTraceSpanHeader() +  //Fetches current active trace/span context
+                                                          //Not currently supported on thor - run on hthor
path: testing/regress/ecl/roxiepipetraced.ecl
user: login: ghalliday
body: Can you clarify this comment?  It should be supported, and if it isn't please open a jira to ensure that it is.

created_at: 2025-04-14T11:01:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,84 @@
+/*##############################################################################
+
+    HPCC SYSTEMS software Copyright (C) 2025 HPCC Systems®.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+############################################################################## */
+
+//nothor
+//nohthor
+import lib_logging; // only needed if lib_logging.Logging.getTraceSpanHeader used below
+string TargetIP := '.' : stored('TargetIP');
+
+NameRec := RECORD
+  string10 First;
+  string15 Last;
+END;
+
+AddressRec := RECORD
+  string10 City;
+  string2 State;
+  integer4 ZipCode;
+END;
+
+PersonRec := RECORD
+  NameRec Name;
+  AddressRec Address;
+END;
+
+pipe_send := DATASET([{{'Joe', 'Doe'}, {'Fresno', 'CA', 11111}},
+{{'Jason', 'Jones'}, {'Tempe', 'AZ', 22222}},
+{{'Becky', 'Lopez'}, {'New York', 'NY', 33333}},
+{{'Roderic', 'Lykke'}, {'Lubbock', 'TX', 44444}},
+{{'Rina', 'Yonkers'}, {'Denver', 'CO', 55555}},
+{{'Laverna', 'Campa'}, {'Detroit', 'MI', 66666}},
+{{'Shantell', 'Mattera'}, {'Toledo', 'OH', 77777}},
+{{'John', 'Smith'}, {'Boise', 'ID', 88888}},
+{{'Jane','Smith'}, {'Smallville', 'KY', 99999}}], PersonRec);
+
+/*  NOTES:
+	roxiepipe = the executable we're calling
+	iw = Input Width
+	vip = Virtual IP
+	t = # threads
+	ow = output width
+	b = # records in a batch
+	mr = max retries
+	h = ??? -- the URL to the Roxie batch service
+	r = name of the results dataset (case sensitive!)
+  tp = traceparent (trace context) propagatated to roxie
+       can be hardcoded or current trace (if available) can be fetched dynamically
+       via lib_logging.Logging.getTraceSpanHeader()
+	q = query
+*/
+
+
+// This is the remote call to the 'roxie_echo' service.
+pipe_recv := PIPE(pipe_send,
+    'roxiepipe' +
+    ' -iw ' + SIZEOF(PersonRec) +
+    ' -t 1' +
+    ' -ow ' + SIZEOF(PersonRec) +
+    ' -b 3' +
+    ' -mr 2' +
+    ' -h ' + TargetIP + ':9876 ' +
+    ' -r Peeps' +
+    ' -tp ' + lib_logging.Logging.getTraceSpanHeader() +  //Fetches current active trace/span context
+                                                          //Not currently supported on thor - run on hthor
path: testing/regress/ecl/roxiepipetraced.ecl
user: login: rpastrana
body: Comment updated inline
created_at: 2025-04-14T14:53:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,86 @@
+/*##############################################################################
path: testing/regress/ecl/roxiepipetraced.ecl
user: login: ghalliday
body: This needs a corresponding xml file - see the failure from the regression suite.  
Also/alternatively, because the trace context will be different each time you may need to either remove that output or mark this test as not having a key file.

created_at: 2025-04-15T10:24:41Z
author_association: MEMBER

PR comments:
user: login: rpastrana
created_at: 2025-04-11T16:20:12Z
body: Resulting span on Roxie side inherits parent from provided trace/span context:
![image](https://github.com/user-attachments/assets/f1aec368-9824-4af8-892a-aea7db0675d6)


The "run_workunit" span tracks hthor portion of the job, not sure why there's multiple roxie_echo, but they are treated as children of the root span as expected. The immediate question raised by the trace, are the wide gaps between the start of the run_workunit and the roxie_echo, the end of the graph1 span under each roxie_echo and the roxie_echo end, and the end of the last roxie_echo and the end of the run_workunit 

```
ecl run hthor /home/ubuntu/GIT/HPCC-Platform/testing/regress/ecl/roxiepipetraced.ecl
```


<Result>
<Dataset name='Result 1'>
 <Row><name><first>Joe       </first><last>Doe            </last></name><address><city>Fresno    </city><state>CA</state><zipcode>11111</zipcode></address></Row>
...
 <Row><name><first>Jane      </first><last>Smith          </last></name><address><city>Smallville</city><state>KY</state><zipcode>99999</zipcode></address></Row>
</Dataset>
</Result>
```


author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-04-11T16:20:27Z
body: Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-33809

Jirabot Action Result:
Workflow Transition To: Merge Pending
Updated PR

author_association: NONE
user: login: rpastrana
created_at: 2025-04-15T14:22:45Z
body: > Minor- updates needed so regression test passes. Also, please update the conclusion in the jira with what was implemented, and create a documentation jira to ensure it gets added to documentation/we circulate information about how to use it.

Will update the regression. I wish there was a way to declare an expected format in regex-like form
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-04-17T15:05:26Z
body: Jirabot Action Result:
Added fix version: 9.10.22
Workflow Transition: 'Resolve issue'

author_association: NONE
user: login: rpastrana
created_at: 2025-04-17T15:06:01Z
body: > Resulting span on Roxie side inherits parent from provided trace/span context: ![image](https://private-user-images.githubusercontent.com/915949/432845649-f1aec368-9824-4af8-892a-aea7db0675d6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDQ4OTczNDUsIm5iZiI6MTc0NDg5NzA0NSwicGF0aCI6Ii85MTU5NDkvNDMyODQ1NjQ5LWYxYWVjMzY4LTk4MjQtNGFmOC04OTJhLWFlYTdkYjA2NzVkNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwNDE3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDQxN1QxMzM3MjVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zOTM4YjI3MTY5MDExM2U5OGU0ZTQ5YjA5N2MzYjJiMzIxNjY3ZjBhNDdlZmFiOGRmZTE1NzU5MjJmMzljNzYxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.On2RD7SK4cfmNdv7NRZD62pynbF6h7srR2kX_gxKsLc)
> 
> The "run_workunit" span tracks hthor portion of the job, not sure why there's multiple roxie_echo, but they are treated as children of the root span as expected. The immediate question raised by the trace, are the wide gaps between the start of the run_workunit and the roxie_echo, the end of the graph1 span under each roxie_echo and the roxie_echo end, and the end of the last roxie_echo and the end of the run_workunit
> 
> ```
> ecl run hthor /home/ubuntu/GIT/HPCC-Platform/testing/regress/ecl/roxiepipetraced.ecl
> ```
> 
> Joe Doe Fresno CA11111 ... Jane Smith SmallvilleKY99999 ```



> Resulting span on Roxie side inherits parent from provided trace/span context: ![image](https://private-user-images.githubusercontent.com/915949/432845649-f1aec368-9824-4af8-892a-aea7db0675d6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDQ4OTczNDUsIm5iZiI6MTc0NDg5NzA0NSwicGF0aCI6Ii85MTU5NDkvNDMyODQ1NjQ5LWYxYWVjMzY4LTk4MjQtNGFmOC04OTJhLWFlYTdkYjA2NzVkNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwNDE3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDQxN1QxMzM3MjVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zOTM4YjI3MTY5MDExM2U5OGU0ZTQ5YjA5N2MzYjJiMzIxNjY3ZjBhNDdlZmFiOGRmZTE1NzU5MjJmMzljNzYxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.On2RD7SK4cfmNdv7NRZD62pynbF6h7srR2kX_gxKsLc)
> 
> The "run_workunit" span tracks hthor portion of the job, not sure why there's multiple roxie_echo, but they are treated as children of the root span as expected. The immediate question raised by the trace, are the wide gaps between the start of the run_workunit and the roxie_echo, the end of the graph1 span under each roxie_echo and the roxie_echo end, and the end of the last roxie_echo and the end of the run_workunit

Following up on this conversation we had regarding this chart, the 3 roxie_echo sub-spans do seem to correlate to the number of "batches" expected, based on the -b option of "3" for the sample dataset of 9 records. My concern is the number of subspans for larger datasets - can we ensure appropriate batch sizes?



author_association: MEMBER