Repository: amzn/ion-js
Pull request #657 (2020-12-02 18:19:47)

Adding `postion()` API to Reader (#654)
*Issue #, if available:*
#654

*Description of changes:*
(Copied from issue description)
Hi, 

**Description**
As a user of ion-js, I would love to be able to get position information from ionReader, so that when using ionReader to read a ion file, I can know which position I'm currently at.

**Application scenario**
Our team uses ion to define a format, e.g. paragraph `P` uses certain format style `S`, and the definition of style `S` is out side of the paragraph itself. see blow:
```
style::{
  name:S,
  font:Arial,
  .....
}
....
....
paragraph::{
  name:A,
  style:S,
  .....
}
```
I'm working on a vs code extension: ion-style-peek, so that in the above ion file, when users are viewing paragraph `A` and call `go to definition` on `S`, the editor will jump to style `S`.  Similar implementation can be found: https://github.com/pranaygp/vscode-css-peek

To implement the above, we need to get the position when parsing style in the ion, hence the requesting issue. 

We can get position information from the below two paths:
```
Reader -> BinaryReader -> ParserBinaryRaw -> StringSpan
Reader -> TextReader -> ParserTextRaw -> BinarySpan
```

Thanks!

Ethan



By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


zslayton: Ok, looking good! 

One other thing that I should've mentioned before (sorry! 😞): we need a couple of unit tests to prove that this works like we think it does.

Could you add some tests to [`test/BinaryReader.ts`](https://github.com/amzn/ion-js/blob/master/test/IonBinaryReader.ts#L28) and [`test/IonTextReader.ts`](https://github.com/amzn/ion-js/blob/master/test/IonTextReader.ts#L233) demonstrating the expected behavior? Some ideas:

* Calling `position()` on new text readers returns `0`
* Calling `position()` on new binary readers returns `0` (or `4` -- they might eagerly consume the Ion Version Marker?)
* Both formats: calling `position()` after calling `next()` returns the expected value for each of a few values in a stream
* A text stream with some multi-byte and multi-code-unit characters returns the expected position in UTF-16 code units.
bingkh: Tests added!
No worry, I should've added the tests from the very beginning, tests are also a good demonstration of the reader APIs.
Please let me know if there needs to be any update!
bingkh: Picked all suggestions! 👍🏼

Patch:
diff --git a/src/IonBinaryReader.ts b/src/IonBinaryReader.ts
index 02a9655..ba8708c 100644
--- a/src/IonBinaryReader.ts
+++ b/src/IonBinaryReader.ts
@@ -103,6 +103,10 @@ export class BinaryReader implements Reader {
     this._raw_type = BOC;
   }
 
+  position(): number {
+    return this._parser.source().position();
+  }
+
   next(): IonType | null {
     this._annotations = null;
 
diff --git a/src/IonParserBinaryRaw.ts b/src/IonParserBinaryRaw.ts
index 188e3f9..398a7ff 100644
--- a/src/IonParserBinaryRaw.ts
+++ b/src/IonParserBinaryRaw.ts
@@ -317,6 +317,10 @@ export class ParserBinaryRaw {
     return Decimal._fromBigIntCoefficient(isNegative, coefficient, exponent);
   }
 
+  source(): BinarySpan {
+    return this._in;
+  }
+
   next(): any {
     if (this._curr === undefined && this._len > 0) {
       this._in.skip(this._len);
diff --git a/src/IonParserTextRaw.ts b/src/IonParserTextRaw.ts
index aefc45d..9591c97 100644
--- a/src/IonParserTextRaw.ts
+++ b/src/IonParserTextRaw.ts
@@ -386,6 +386,10 @@ export class ParserTextRaw {
     return this._fieldnameType;
   }
 
+  source(): StringSpan {
+    return this._in;
+  }
+
   annotations(): SymbolToken[] {
     return this._ann;
   }
diff --git a/src/IonReader.ts b/src/IonReader.ts
index 3a386c7..2ac8b1a 100644
--- a/src/IonReader.ts
+++ b/src/IonReader.ts
@@ -41,6 +41,28 @@ export type ReaderScalarValue =
  * invoke [[next]] to position a newly created reader on the first value.
  */
 export interface Reader {
+  /**
+   * Returns the Reader's offset from the beginning of its input.
+   *
+   * For binary Readers, the return value is the number of bytes that have
+   * been processed.
+   *
+   * For text Readers, the return value is the number of UTF-16 code units
+   * that have been processed, regardless of the input's original encoding.
+   * For more on JavaScript's in-memory representation of text, see:
+   * https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length#Description
+   *
+   * Note that a Reader cannot safely skip to a given position in input without
+   * processing the stream leading up to that position. This is because there are
+   * mid-stream system level values that must be processed to guarantee that the
+   * Reader is in a valid state. It is safe, however, to start at the beginning of a data
+   * source and call next() until you reach the desired position, as the reader
+   * will still have the opportunity to process system-level values along the way.)
+   *
+   * @returns the [[number]] of bytes or UTF-16 code units that the reader has processed.
+   */
+  position(): number;
+
   /**
    * Advances the reader to the next value in the stream at the current depth.
    *
diff --git a/src/IonTextReader.ts b/src/IonTextReader.ts
index ffe0c44..88e0a37 100644
--- a/src/IonTextReader.ts
+++ b/src/IonTextReader.ts
@@ -140,6 +140,10 @@ export class TextReader implements Reader {
     return false;
   }
 
+  position() {
+    return this._parser.source().position();
+  }
+
   next() {
     this._raw = undefined;
     if (this._raw_type === EOF) {


Test patch:
diff --git a/test/IonBinaryReader.ts b/test/IonBinaryReader.ts
index dc76283..709a804 100644
--- a/test/IonBinaryReader.ts
+++ b/test/IonBinaryReader.ts
@@ -26,4 +26,51 @@ describe('Binary Reader', () => {
         reader.next();
         assert.deepEqual(reader.timestampValue(), timestamp);
     });
+    it('test position', () => {
+        // In the comments below, the vertical bar '|' indicates the current position of the cursor at each step.
+        const ionBinary: Uint8Array = ion.dumpBinary(null, 7, -17, "Hello", [1, 2, 3]);
+        // [|0xE0, 0x1, 0x0, 0xEA, |0xF, |0x21, 0x7, |0x31, 0x11, |0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, |0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3 |]
+        //   Version                null  7           -17          "hello"                              [1,2,3]
+        const binaryReader = ion.makeReader(ionBinary);
+        // init pos 0
+        // [|0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 0);
+        binaryReader.next();
+        // at null
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF,| 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 5);
+        binaryReader.next();
+        // at 7
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7,| 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 6);
+        binaryReader.next();
+        // at 17
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11,| 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 8);
+        binaryReader.next();
+        // at hello
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48,| 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 10);
+        binaryReader.next();
+        // at [1,2,3] 
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21,| 0x1, 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 16);
+        binaryReader.stepIn();
+        binaryReader.next();
+        // at 1
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1,| 0x21, 0x2, 0x21, 0x3]
+        assert.equal(binaryReader.position(), 17);
+        binaryReader.next();
+        // at 2
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2,| 0x21, 0x3]
+        assert.equal(binaryReader.position(), 19);
+        binaryReader.next();
+        // at 3
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3|]
+        assert.equal(binaryReader.position(), 21);
+        binaryReader.stepOut();
+        // out of stream
+        // [0xE0, 0x1, 0x0, 0xEA, 0xF, 0x21, 0x7, 0x31, 0x11, 0x85, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0xB6, 0x21, 0x1, 0x21, 0x2, 0x21, 0x3]|
+        assert.equal(binaryReader.position(), 22);
+    });
 });
diff --git a/test/IonTextReader.ts b/test/IonTextReader.ts
index 5492db2..fa96923 100644
--- a/test/IonTextReader.ts
+++ b/test/IonTextReader.ts
@@ -249,4 +249,70 @@ class IonTextReaderTests {
             IonTextReaderTests.ivmTest(textReader, unsupportedIVM[i], false, 0, ["taco"]);
         }
     }
+
+    @test "get position from reader" () {
+        const ionToRead = "ann1::{ key: valX } ann2::{ keyX: ['v1', 'v2'], keyY: ['v3', 'v4'] }";
+        const ionTextReader = ion.makeReader(ionToRead);
+        
+        // In the comments below, the vertical bar '|' indicates the current position of the cursor at each step.
+        const pos1 = ionTextReader.position();
+        // |ann1::{ key: valX } ann2::{ keyX: ['v1', 'v2'], keyY: ['v3', 'v4'] }
+        assert.equal(pos1, 0);
+        
+        ionTextReader.next();
+        // ann1::{| key: valX } ann2::{ keyX: ['v1', 'v2'], keyY: ['v3', 'v4'] }
+        const pos2 = ionTextReader.position();
+        assert.equal(pos2, 7);
+
+        ionTextReader.stepIn();
+        ionTextReader.next();
+        // ann1::{ key: valX |} ann2::{ keyX: ['v1', 'v2'], keyY: ['v3', 'v4'] }
+        const pos7 = ionTextReader.position();
+        assert.equal(pos7, 18);
+        
+        ionTextReader.stepOut();
+        ionTextReader.next();
+        // ann1::{ key: valX } ann2::{| keyX: ['v1', 'v2'], keyY: ['v3', 'v4'] }
+        const pos3 = ionTextReader.position();
+        assert.equal(pos3, 27);
+        
+        ionTextReader.stepIn();
+        ionTextReader.next();
+        // ann1::{ key: valX } ann2::{ keyX: [|'v1', 'v2'], keyY: ['v3', 'v4'] }
+        const pos4 = ionTextReader.position();
+        assert.equal(pos4, 35);
+        
+        ionTextReader.stepIn();
+        ionTextReader.next();
+        // ann1::{ key: valX } ann2::{ keyX: ['v1'|, 'v2'], keyY: ['v3', 'v4'] }
+        const pos5 = ionTextReader.position();
+        assert.equal(pos5, 39);
+        
+        ionTextReader.stepOut();
+        // ann1::{ key: valX } ann2::{ keyX: ['v1', 'v2']|, keyY: ['v3', 'v4'] }
+        const pos6 = ionTextReader.position();
+        assert.equal(pos6, 46);
+
+        ionTextReader.stepOut();
+        // ann1::{ key: valX } ann2::{ keyX: ['v1', 'v2'], keyY: ['v3', 'v4'] }|
+        const pos8 = ionTextReader.position();
+        assert.equal(pos8, 68);
+    }
+
+    @test "get position from multi-byte" () {
+
+        // 's': \u0073
+        // '好': \u597d
+        // 'の': \u306e
+        // 'ç': \u00e7
+        // '👩': \ud83d \udc69
+        // '👩🏽': \ud83d \udc69 \ud83c \udffd
+        const reader: ion.Reader = ion.makeReader("\"s\" \"好\" \"の\" \"ç\" \"👩\" \"👩🏽\"");
+
+        const pos = [3, 7, 11, 15, 20, 27];
+        let i = 0;
+        while (reader.next()) {
+            assert.equal(reader.position(), pos[i++]);
+        }
+    }
 }