Repository: amzn/ion-java
Pull request #192 (2018-12-14 23:57:58)

Clob nonascii error 99
*Issue #, if available:*
 #99 

*Description of changes:*
Throws error when reading clobs with non ascii characters

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.


codecov[bot]: # [Codecov](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=h1) Report
> Merging [#192](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=desc) into [master](https://codecov.io/gh/amzn/ion-java/commit/fcd528e308f9ea585ae627a64d5eda57b416ecd3?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `20%`.

[![Impacted file tree graph](https://codecov.io/gh/amzn/ion-java/pull/192/graphs/tree.svg?width=650&token=M93Y3e3kKn&height=150&src=pr)](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=tree)

```diff
@@             Coverage Diff              @@
##             master     #192      +/-   ##
============================================
- Coverage     71.67%   71.66%   -0.02%     
- Complexity     4290     4292       +2     
============================================
  Files           123      123              
  Lines         16002    16006       +4     
  Branches       2626     2628       +2     
============================================
+ Hits          11470    11471       +1     
- Misses         3423     3424       +1     
- Partials       1109     1111       +2
```


| [Impacted Files](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...software/amazon/ion/impl/IonReaderTextSystemX.java](https://codecov.io/gh/amzn/ion-java/pull/192/diff?src=pr&el=tree#diff-c3JjL3NvZnR3YXJlL2FtYXpvbi9pb24vaW1wbC9Jb25SZWFkZXJUZXh0U3lzdGVtWC5qYXZh) | `58.55% <0%> (ø)` | `107 <0> (ø)` | :arrow_down: |
| [...tware/amazon/ion/impl/IonReaderTextRawTokensX.java](https://codecov.io/gh/amzn/ion-java/pull/192/diff?src=pr&el=tree#diff-c3JjL3NvZnR3YXJlL2FtYXpvbi9pb24vaW1wbC9Jb25SZWFkZXJUZXh0UmF3VG9rZW5zWC5qYXZh) | `74.54% <25%> (-0.18%)` | `358 <0> (+2)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=footer). Last update [fcd528e...8f646f1](https://codecov.io/gh/amzn/ion-java/pull/192?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

raganhan: Merging manually to make it easier to update ion-tests submodule 
raganhan: Can't merge manually anymore, reopening 

Patch:
diff --git a/src/software/amazon/ion/impl/IonReaderTextRawTokensX.java b/src/software/amazon/ion/impl/IonReaderTextRawTokensX.java
index 33757bc3..aba793f1 100644
--- a/src/software/amazon/ion/impl/IonReaderTextRawTokensX.java
+++ b/src/software/amazon/ion/impl/IonReaderTextRawTokensX.java
@@ -2049,6 +2049,9 @@ final class IonReaderTextRawTokensX
     protected int read_double_quoted_char(boolean is_clob) throws IOException
     {
         int c = read_char();
+        if(is_clob && c > 127) {
+            throw new IonReaderTextTokenException("non ASCII character in clob: " + c);
+        }
 
         switch (c) {
         case '"':
@@ -2174,6 +2177,10 @@ final class IonReaderTextRawTokensX
     protected int read_triple_quoted_char(boolean is_clob) throws IOException
     {
         int c = read_string_char(ProhibitedCharacters.LONG_CHAR);
+        if(is_clob && c > 127) {
+            throw new IonReaderTextTokenException("non ASCII character in clob: " + c);
+        }
+
         switch (c) {
         case '\'':
             if (is_2_single_quotes_helper()) {
diff --git a/src/software/amazon/ion/impl/IonReaderTextSystemX.java b/src/software/amazon/ion/impl/IonReaderTextSystemX.java
index 55b5a849..07c56683 100644
--- a/src/software/amazon/ion/impl/IonReaderTextSystemX.java
+++ b/src/software/amazon/ion/impl/IonReaderTextSystemX.java
@@ -849,6 +849,7 @@ class IonReaderTextSystemX
         int c = -1;
 
         switch (_lob_token) {
+        // BLOB
         case IonTokenConstsX.TOKEN_OPEN_DOUBLE_BRACE:
             while (len-- > 0) {
                 c = _scanner.read_base64_byte();
@@ -856,6 +857,7 @@ class IonReaderTextSystemX
                 buffer[offset++] = (byte)c;
             }
             break;
+        // CLOB
         case IonTokenConstsX.TOKEN_STRING_DOUBLE_QUOTE:
             while (len-- > 0) {
                 c = _scanner.read_double_quoted_char(true);
@@ -868,10 +870,11 @@ class IonReaderTextSystemX
                     }
                     break;
                 }
-                assert(c >= 0 && c <= UNSIGNED_BYTE_MAX_VALUE);
+                assert(c <= UNSIGNED_BYTE_MAX_VALUE);
                 buffer[offset++] = (byte)c;
             }
             break;
+        // CLOB 
         case IonTokenConstsX.TOKEN_STRING_TRIPLE_QUOTE:
             while (len-- > 0) {
                 c = _scanner.read_triple_quoted_char(true);


Test patch:
diff --git a/ion-tests b/ion-tests
index ad489bf2..0281240a 160000
--- a/ion-tests
+++ b/ion-tests
@@ -1,1 +1,1 @@
-Subproject commit ad489bf2df23381eedb3df8ccd69931f581dea18
+Subproject commit 0281240a9039a1f609fc8e289721efff489956be
diff --git a/test/software/amazon/ion/BadIonTest.java b/test/software/amazon/ion/BadIonTest.java
index ff021201..32606684 100644
--- a/test/software/amazon/ion/BadIonTest.java
+++ b/test/software/amazon/ion/BadIonTest.java
@@ -18,10 +18,12 @@ import static software.amazon.ion.TestUtils.GLOBAL_SKIP_LIST;
 import static software.amazon.ion.TestUtils.testdataFiles;
 
 import java.io.File;
+import java.io.FileReader;
 import java.io.IOException;
+import java.nio.CharBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetEncoder;
 import org.junit.Test;
-import software.amazon.ion.IonDatagram;
-import software.amazon.ion.IonException;
 import software.amazon.ion.impl.PrivateUtils;
 import software.amazon.ion.junit.Injected.Inject;
 
@@ -75,7 +77,15 @@ public class BadIonTest
             }
             catch (IonException e)
             {
-                assert myTestFile.getPath().contains("bad/utf8") || myTestFile.getPath().contains("bad\\utf8");
+                // checks that test failed because of bad UTF-8 data
+                final CharBuffer buffer = CharBuffer.allocate(1024);
+                final CharsetEncoder utf8Encoder = Charset.forName("UTF-8").newEncoder();
+
+                final FileReader fileReader = new FileReader(myTestFile);
+                while(fileReader.read(buffer) != -1){
+                    assert utf8Encoder.canEncode(buffer);
+                }
+
                 return;
             }
 
diff --git a/test/software/amazon/ion/TestUtils.java b/test/software/amazon/ion/TestUtils.java
index f50ff042..d7ade58f 100644
--- a/test/software/amazon/ion/TestUtils.java
+++ b/test/software/amazon/ion/TestUtils.java
@@ -130,12 +130,16 @@ public class TestUtils
     public static final FilenameFilter GLOBAL_SKIP_LIST =
         new FileIsNot(
              "bad/clobWithNullCharacter.ion"                // TODO amzn/ion-java#43
-            , "bad/clobWithNonAsciiCharacter.ion"           // TODO amzn/ion-java#99
             , "bad/emptyAnnotatedInt.10n"                   // TODO amzn/ion-java#55
-            , "bad/utf8/surrogate_5.ion"                    // TODO amzn/ion-java#60
             , "bad/utf8/surrogate_1.ion"                    // TODO amzn/ion-java#105
             , "bad/utf8/surrogate_2.ion"                    // TODO amzn/ion-java#105
             , "bad/utf8/surrogate_4.ion"                    // TODO amzn/ion-java#105
+            , "bad/utf8/surrogate_5.ion"                    // TODO amzn/ion-java#60
+            , "bad/utf8/surrogate_6.ion"                    // TODO amzn/ion-java#105
+            , "bad/utf8/surrogate_7.ion"                    // TODO amzn/ion-java#105
+            , "bad/utf8/surrogate_8.ion"                    // TODO amzn/ion-java#105
+            , "bad/utf8/surrogate_9.ion"                    // TODO amzn/ion-java#105
+            , "bad/utf8/surrogate_10.ion"                   // TODO amzn/ion-java#105
             , "equivs/paddedInts.10n"                       // TODO amzn/ion-java#54
             , "good/subfieldVarUInt32bit.ion"               // TODO amzn/ion-java#62
             , "good/utf16.ion"                              // TODO amzn/ion-java#61