Repository: iterative/cml
Pull request #581 (2021-06-08 15:27:27)

Add `cml-send-comment --update`
Running `cml-send-comment` with the newly introduced `--update` flag will update the last CML comment instead of creating a new one. Comment authorship is determined by checking an invisible _watermark_ [sic] added by CML to every machine-made comment. Watermarks are just void images appended with Markdown at the end of comments.

### Supported platforms

- [X] GitHub
- [ ] GitLab<sup>1</sup>
- [X] Bitbucket<sup>2</sup>

***

<sup>1</sup> Blocked by [gitlab-org/gitlab#333000](https://gitlab.com/gitlab-org/gitlab/-/issues/333000); there is no known workaround.
<sup>2</sup> Requires user interaction for the first time, as per https://github.com/iterative/cml/pull/581#issuecomment-861898718.

casperdcl: Given https://github.com/iterative/cml/wiki/Backend-Supported-Features we could still merge anyway
0x2b3bfa0: > [_Pull Request Commit Links app must be installed first before using this API; installation automatically occurs when 'Go to pull request' is clicked from the web interface for a commit's details._](https://developer.atlassian.com/bitbucket/api/2/reference/resource/repositories/%7Bworkspace%7D/%7Brepo_slug%7D/commit/%7Bcommit%7D/pullrequests) (Clinical definition of insanity)
casperdcl: > > [_Pull Request Commit Links app must be installed first before using this API; installation automatically occurs when 'Go to pull request' is clicked from the web interface for a commit's details._](https://developer.atlassian.com/bitbucket/api/2/reference/resource/repositories/%7Bworkspace%7D/%7Brepo_slug%7D/commit/%7Bcommit%7D/pullrequests) (Clinical definition of insanity)

can we

- auto-detect this and emit a warning
- document this (probably in https://github.com/iterative/cml/wiki/cml-send-comment)

0x2b3bfa0: Done! The documentation part is a bit terse, but probably good enough for now. Feel free to expand it if you want to.

Patch:
diff --git a/Dockerfile b/Dockerfile
index d97d5c7..2e52f3b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,8 +1,7 @@
 ARG BASE_IMAGE=ubuntu:20.04
 FROM ${BASE_IMAGE}
 
-# TODO: consider using iterative.ai or something else
-LABEL maintainer="dvcorg <support@cml.dev>"
+LABEL maintainer="CML <support@cml.dev>"
 
 # CONFIGURE NON-INTERACTIVE APT
 ENV DEBIAN_FRONTEND=noninteractive
diff --git a/README.md b/README.md
index 85dbe08..eff3bf6 100644
--- a/README.md
+++ b/README.md
@@ -109,12 +109,14 @@ into a CML report.
 Below is a table of CML functions for writing markdown reports and delivering
 those reports to your CI system (GitHub Actions or GitLab CI).
 
-| Function                | Description                                                    | Inputs                                                    |
-| ----------------------- | -------------------------------------------------------------- | --------------------------------------------------------- |
-| `cml-send-comment`      | Return CML report as a comment in your GitHub/GitLab workflow. | `<path to report> --head-sha <sha>`                       |
-| `cml-send-github-check` | Return CML report as a check in GitHub                         | `<path to report> --head-sha <sha>`                       |
-| `cml-publish`           | Publish an image for writing to CML report.                    | `<path to image> --title <image title> --md`              |
-| `cml-tensorboard-dev`   | Return a link to a Tensorboard.dev page                        | `--logdir <path to logs> --title <experiment title> --md` |
+| Function                | Description                                                    | Inputs                                                      |
+| ----------------------- | -------------------------------------------------------------- | ----------------------------------------------------------- |
+| `cml-runner`            | Starts a runner locally or in cloud providers                  | See [Arguments](https://github.com/iterative/cml#arguments) |
+| `cml-publish`           | Publish an image for writing to CML report.                    | `<path to image> --title <image title> --md`                |
+| `cml-send-comment`      | Return CML report as a comment in your GitHub/GitLab workflow. | `<path to report> --head-sha <sha>`                         |
+| `cml-send-github-check` | Return CML report as a check in GitHub                         | `<path to report> --head-sha <sha>`                         |
+| `cml-pr`                | Create a pull request.                                         | TODO                                                        |
+| `cml-tensorboard-dev`   | Return a link to a Tensorboard.dev page                        | `--logdir <path to logs> --title <experiment title> --md`   |
 
 ### Customizing your CML report
 
diff --git a/bin/cml-runner.js b/bin/cml-runner.js
index 55bc282..c4abaa5 100755
--- a/bin/cml-runner.js
+++ b/bin/cml-runner.js
@@ -114,8 +114,6 @@ const shutdown = async (opts) => {
 };
 
 const runCloud = async (opts) => {
-  const { cloudSshPrivateVisible } = opts;
-
   const runTerraform = async (opts) => {
     console.log('Terraform apply...');
 
@@ -184,21 +182,31 @@ const runCloud = async (opts) => {
   console.log('Deploying cloud runner plan...');
   const tfstate = await runTerraform(opts);
   const { resources } = tfstate;
-  for (let i = 0; i < resources.length; i++) {
-    const resource = resources[i];
-
+  for (const resource of resources) {
     if (resource.type.startsWith('iterative_')) {
-      const { instances } = resource;
-
-      for (let j = 0; j < instances.length; j++) {
-        const instance = instances[j];
-
-        if (!cloudSshPrivateVisible) {
-          instance.attributes.ssh_private = '[MASKED]';
-        }
-
-        instance.attributes.token = '[MASKED]';
-        console.log(JSON.stringify(instance));
+      for (const { attributes } of resource.instances) {
+        const nonSensitiveValues = {
+          awsSecurityGroup: attributes.aws_security_group,
+          cloud: attributes.cloud,
+          driver: attributes.driver,
+          id: attributes.id,
+          idleTimeout: attributes.idle_timeout,
+          image: attributes.image,
+          instanceGpu: attributes.instance_gpu,
+          instanceHddSize: attributes.instance_hdd_size,
+          instanceIp: attributes.instance_ip,
+          instanceLaunchTime: attributes.instance_launch_time,
+          instanceType: attributes.instance_type,
+          labels: attributes.labels,
+          name: attributes.name,
+          region: attributes.region,
+          repo: attributes.repo,
+          single: attributes.single,
+          spot: attributes.spot,
+          spotPrice: attributes.spot_price,
+          timeouts: attributes.timeouts
+        };
+        console.log(JSON.stringify(nonSensitiveValues));
       }
     }
   }
@@ -379,11 +387,7 @@ const opts = yargs
     'cloud-ssh-private',
     'Custom private RSA SSH key. If not provided an automatically generated throwaway key will be used'
   )
-  .boolean('cloud-ssh-private-visible')
-  .describe(
-    'cloud-ssh-private-visible',
-    'Show the private SSH key in the output with the rest of the instance properties (not recommended)'
-  )
+  .coerce('cloud-ssh-private', (val) => val.replace(/\n/g, '\\n'))
   .boolean('cloud-spot')
   .describe('cloud-spot', 'Request a spot instance')
   .default('cloud-spot-price', '-1')
diff --git a/bin/cml-send-comment.js b/bin/cml-send-comment.js
index 6ee5d10..266e463 100755
--- a/bin/cml-send-comment.js
+++ b/bin/cml-send-comment.js
@@ -1,5 +1,6 @@
 #!/usr/bin/env node
 
+const print = console.log;
 console.log = console.error;
 
 const fs = require('fs').promises;
@@ -11,7 +12,7 @@ const run = async (opts) => {
   const path = opts._[0];
   const report = await fs.readFile(path, 'utf-8');
   const cml = new CML(opts);
-  await cml.commentCreate({ ...opts, report });
+  print(await cml.commentCreate({ ...opts, report }));
 };
 
 const opts = yargs
@@ -23,6 +24,11 @@ const opts = yargs
     'Commit SHA linked to this comment. Defaults to HEAD.'
   )
   .alias('commit-sha', 'head-sha')
+  .boolean('update')
+  .describe(
+    'update',
+    'Update the last CML comment (if any) instead of creating a new one'
+  )
   .boolean('rm-watermark')
   .describe(
     'rm-watermark',
@@ -36,10 +42,10 @@ const opts = yargs
   .default('token')
   .describe(
     'token',
-    'Personal access token to be used. If not specified in extracted from ENV REPO_TOKEN.'
+    'Personal access token to be used. If not specified is extracted from ENV REPO_TOKEN.'
   )
   .default('driver')
-  .choices('driver', ['github', 'gitlab'])
+  .choices('driver', ['github', 'gitlab', 'bitbucket'])
   .describe('driver', 'If not specify it infers it from the ENV.')
   .help('h')
   .demand(1).argv;
diff --git a/package-lock.json b/package-lock.json
index 1037c56..9443b0a 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,6 +1,6 @@
 {
   "name": "@dvcorg/cml",
-  "version": "0.4.5",
+  "version": "0.4.6",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {
diff --git a/package.json b/package.json
index e2a0bb8..ee6886d 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@dvcorg/cml",
-  "version": "0.4.5",
+  "version": "0.4.6",
   "author": {
     "name": "DVC",
     "url": "http://cml.dev"
diff --git a/src/cml.js b/src/cml.js
index b7df63a..c6343e6 100644
--- a/src/cml.js
+++ b/src/cml.js
@@ -96,8 +96,11 @@ class CML {
     const {
       report: userReport,
       commitSha = await this.headSha(),
-      rmWatermark
+      rmWatermark,
+      update
     } = opts;
+    if (rmWatermark && update)
+      throw new Error('watermarks are mandatory for updateable comments');
     const watermark = rmWatermark
       ? ''
       : ' \n\n  ![CML watermark](https://raw.githubusercontent.com/iterative/cml/master/assets/watermark.svg)';
@@ -106,7 +109,8 @@ class CML {
     return await getDriver(this).commentCreate({
       ...opts,
       report,
-      commitSha
+      commitSha,
+      watermark
     });
   }
 
diff --git a/src/drivers/bitbucket_cloud.js b/src/drivers/bitbucket_cloud.js
index 1f6c097..d4ee176 100644
--- a/src/drivers/bitbucket_cloud.js
+++ b/src/drivers/bitbucket_cloud.js
@@ -22,43 +22,67 @@ class BitBucketCloud {
 
   async commentCreate(opts = {}) {
     const { projectPath } = this;
-    const { commitSha, report } = opts;
-
-    // Make a comment in the commit
-    const commitEndpoint = `/repositories/${projectPath}/commit/${commitSha}/comments/`;
-    const commitBody = JSON.stringify({ content: { raw: report } });
-    const commitOutput = await this.request({
-      endpoint: commitEndpoint,
-      method: 'POST',
-      body: commitBody
-    });
+    const { commitSha, report, update, watermark } = opts;
 
     // Check for a corresponding PR. If it exists, also put the comment there.
-    const getPrEndpt = `/repositories/${projectPath}/commit/${commitSha}/pullrequests`;
-    const { values: prs } = await this.request({ endpoint: getPrEndpt });
+    let prs;
+    try {
+      const getPrEndpoint = `/repositories/${projectPath}/commit/${commitSha}/pullrequests`;
+      prs = await this.paginatedRequest({ endpoint: getPrEndpoint });
+    } catch (err) {
+      if (err.message === 'Not Found Resource not found')
+        err.message =
+          "Click 'Go to pull request' on any commit details page to enable this API";
+      throw err;
+    }
 
     if (prs && prs.length) {
       for (const pr of prs) {
-        try {
-          // Append a watermark to the report with a link to the commit
-          const commitLink = commitSha.substr(0, 7);
-          const longReport = `${commitLink}   \n${report}`;
-          const prBody = JSON.stringify({ content: { raw: longReport } });
-
-          // Write a comment on the PR
-          const prEndpoint = `/repositories/${projectPath}/pullrequests/${pr.id}/comments`;
-          await this.request({
-            endpoint: prEndpoint,
-            method: 'POST',
-            body: prBody
-          });
-        } catch (err) {
-          console.debug(err.message);
-        }
+        // Append a watermark to the report with a link to the commit
+        const commitLink = commitSha.substr(0, 7);
+        const longReport = `${commitLink}\n\n${report}`;
+        const prBody = JSON.stringify({ content: { raw: longReport } });
+
+        // Write a comment on the PR
+        const prEndpoint = `/repositories/${projectPath}/pullrequests/${pr.id}/comments/`;
+        const existingPr = (
+          await this.paginatedRequest({ endpoint: prEndpoint, method: 'GET' })
+        )
+          .filter((comment) => {
+            const { content: { raw = '' } = {} } = comment;
+            return raw.endsWith(watermark);
+          })
+          .sort((first, second) => first.id < second.id)
+          .pop();
+        await this.request({
+          endpoint: prEndpoint + (update && existingPr ? existingPr.id : ''),
+          method: update && existingPr ? 'PUT' : 'POST',
+          body: prBody
+        });
       }
     }
 
-    return commitOutput;
+    const commitEndpoint = `/repositories/${projectPath}/commit/${commitSha}/comments/`;
+
+    const existingCommmit = (
+      await this.paginatedRequest({ endpoint: commitEndpoint, method: 'GET' })
+    )
+      .filter((comment) => {
+        const { content: { raw = '' } = {} } = comment;
+        return raw.endsWith(watermark);
+      })
+      .sort((first, second) => first.id < second.id)
+      .pop();
+
+    return (
+      await this.request({
+        endpoint:
+          commitEndpoint +
+          (update && existingCommmit ? existingCommmit.id : ''),
+        method: update && existingCommmit ? 'PUT' : 'POST',
+        body: JSON.stringify({ content: { raw: report } })
+      })
+    ).links.html.href;
   }
 
   async checkCreate() {
@@ -150,15 +174,18 @@ class BitBucketCloud {
 
   async request(opts = {}) {
     const { token, api } = this;
-    const { endpoint, method = 'GET', body } = opts;
-
-    if (!endpoint) throw new Error('BitBucket Cloud API endpoint not found');
+    const { url, endpoint, method = 'GET', body } = opts;
+    if (!(url || endpoint))
+      throw new Error('BitBucket Cloud API endpoint not found');
     const headers = {
       'Content-Type': 'application/json',
       Authorization: 'Basic ' + `${token}`
     };
-    const url = `${api}${endpoint}`;
-    const response = await fetch(url, { method, headers, body });
+    const response = await fetch(url || `${api}${endpoint}`, {
+      method,
+      headers,
+      body
+    });
 
     if (response.status > 300) {
       const {
@@ -170,6 +197,22 @@ class BitBucketCloud {
     return await response.json();
   }
 
+  async paginatedRequest(opts = {}) {
+    const { method = 'GET', body } = opts;
+    const { next, values } = await this.request(opts);
+
+    if (next) {
+      const nextValues = await this.paginatedRequest({
+        url: next,
+        method,
+        body
+      });
+      values.push(...nextValues);
+    }
+
+    return values;
+  }
+
   get sha() {
     return BITBUCKET_COMMIT;
   }
diff --git a/src/drivers/github.js b/src/drivers/github.js
index 1a53da5..16b8fed 100644
--- a/src/drivers/github.js
+++ b/src/drivers/github.js
@@ -70,18 +70,40 @@ class Github {
   }
 
   async commentCreate(opts = {}) {
-    const { report: body, commitSha } = opts;
-
-    const { url: commitUrl } = await octokit(
-      this.token,
-      this.repo
-    ).repos.createCommitComment({
-      ...ownerRepo({ uri: this.repo }),
-      body,
-      commit_sha: commitSha
-    });
-
-    return commitUrl;
+    const { report: body, commitSha, update, watermark } = opts;
+
+    const { paginate, repos } = octokit(this.token, this.repo);
+
+    const existing = Object.values(
+      await paginate(repos.listCommentsForCommit, {
+        ...ownerRepo({ uri: this.repo }),
+        commit_sha: commitSha
+      })
+    )
+      .filter((comment) => {
+        const { body = '' } = comment;
+        return body.endsWith(watermark);
+      })
+      .sort((first, second) => first.id < second.id)
+      .pop();
+
+    if (update && existing) {
+      return (
+        await repos.updateCommitComment({
+          ...ownerRepo({ uri: this.repo }),
+          comment_id: existing.id,
+          body
+        })
+      ).data.html_url;
+    } else {
+      return (
+        await repos.createCommitComment({
+          ...ownerRepo({ uri: this.repo }),
+          commit_sha: commitSha,
+          body
+        })
+      ).data.html_url;
+    }
   }
 
   async checkCreate(opts = {}) {
diff --git a/src/drivers/gitlab.js b/src/drivers/gitlab.js
index 85a9dd6..0ecdb42 100644
--- a/src/drivers/gitlab.js
+++ b/src/drivers/gitlab.js
@@ -71,7 +71,9 @@ class Gitlab {
   }
 
   async commentCreate(opts = {}) {
-    const { commitSha, report } = opts;
+    const { commitSha, report, update } = opts;
+
+    if (update) throw new Error('GitLab does not support comment updates!');
 
     const projectPath = await this.projectPath();
     const endpoint = `/projects/${projectPath}/repository/commits/${commitSha}/comments`;


Test patch:
diff --git a/bin/cml-send-comment.test.js b/bin/cml-send-comment.test.js
index 1eb3902..ad914ca 100644
--- a/bin/cml-send-comment.test.js
+++ b/bin/cml-send-comment.test.js
@@ -21,15 +21,17 @@ describe('Comment integration tests', () => {
       Options:
         --version                 Show version number                        [boolean]
         --commit-sha, --head-sha  Commit SHA linked to this comment. Defaults to HEAD.
+        --update                  Update the last CML comment (if any) instead of
+                                  creating a new one                         [boolean]
         --rm-watermark            Avoid watermark. CML needs a watermark to be able to
                                   distinguish CML reports from other comments in order
                                   to provide extra functionality.            [boolean]
         --repo                    Specifies the repo to be used. If not specified is
                                   extracted from the CI ENV.
         --token                   Personal access token to be used. If not specified
-                                  in extracted from ENV REPO_TOKEN.
+                                  is extracted from ENV REPO_TOKEN.
         --driver                  If not specify it infers it from the ENV.
-                                                         [choices: \\"github\\", \\"gitlab\\"]
+                                            [choices: \\"github\\", \\"gitlab\\", \\"bitbucket\\"]
         -h                        Show help                                  [boolean]"
     `);
   });
diff --git a/src/cml.test.js b/src/cml.test.js
index b307b49..e48f2ff 100644
--- a/src/cml.test.js
+++ b/src/cml.test.js
@@ -1,5 +1,6 @@
 const CML = require('../src/cml').default;
 
+jest.setTimeout(40000);
 describe('Github tests', () => {
   const OLD_ENV = process.env;
 
diff --git a/src/drivers/bitbucket_cloud.test.js b/src/drivers/bitbucket_cloud.test.js
index f619ea8..bcd9dfa 100644
--- a/src/drivers/bitbucket_cloud.test.js
+++ b/src/drivers/bitbucket_cloud.test.js
@@ -1,4 +1,4 @@
-jest.setTimeout(20000);
+jest.setTimeout(120000);
 const BitBucketCloud = require('./bitbucket_cloud');
 const {
   TEST_BBCLOUD_TOKEN: TOKEN,
diff --git a/src/drivers/github.test.js b/src/drivers/github.test.js
index 6a9cf68..772648c 100644
--- a/src/drivers/github.test.js
+++ b/src/drivers/github.test.js
@@ -1,4 +1,4 @@
-jest.setTimeout(20000);
+jest.setTimeout(40000);
 
 const GithubClient = require('./github');