Repository: w3c/epubcheck (java)

Issue:
Null exception on checking data URL - EpubCheck 5.0.1 and 5.1.0
Hi, 

When I check a fixed layout epub using the EpubCheck 5.0.1 or EpubCheck 5.1.0, I get the following. No errors on EpubCheck 5.0.0 and earlier versions. OPF file is also validated and has no resource path issues: 

java.lang.NullPointerException: null input
	at io.mola.galimatias.URLParser.parse(URLParser.java:215)
	at io.mola.galimatias.URL.withPath(URL.java:397)
	at io.mola.galimatias.canonicalize.DecodeUnreservedCanonicalizer.canonicalize(DecodeUnreservedCanonicalizer.java:41)
	at org.w3c.epubcheck.util.url.URLUtils.normalize(URLUtils.java:188)
	at com.adobe.epubcheck.ocf.OCFContainer.contains(OCFContainer.java:88)
	at com.adobe.epubcheck.ocf.OCFContainer.isRemote(OCFContainer.java:134)
	at org.w3c.epubcheck.core.references.ResourceReferencesChecker.checkReference(ResourceReferencesChecker.java:120)
	at org.w3c.epubcheck.core.references.ResourceReferencesChecker.check(ResourceReferencesChecker.java:102)
	at com.adobe.epubcheck.opf.OPFChecker.checkPackage(OPFChecker.java:149)
	at com.adobe.epubcheck.opf.OPFChecker30.checkPackage(OPFChecker30.java:67)
	at com.adobe.epubcheck.opf.OPFChecker.check(OPFChecker.java:94)
	at com.adobe.epubcheck.ocf.OCFChecker.check(OCFChecker.java:173)
	at com.adobe.epubcheck.api.EpubCheck.doValidate(EpubCheck.java:218)
	at com.adobe.epubcheck.tool.EpubChecker.validateFile(EpubChecker.java:250)
	at com.adobe.epubcheck.tool.EpubChecker.processFile(EpubChecker.java:325)
	at com.adobe.epubcheck.tool.EpubChecker.run(EpubChecker.java:150)
	at com.adobe.epubcheck.tool.Checker.main(Checker.java:31)


Pull request description:
fix: NullPointerException was raised on some data URLs
A NullPointerException was raised on `data` URLs ending with a query-like string.
This was caused by a double-bug in Galimatias:
- removing the query-like string (as we do when registering references) with `URL#withQuery(null)` resulted in a hierarchical URL despite `data` URLs being non-hierarchical.
- such hybrid `data` URLs caused an NPE to be raised when canonicalized

We now add some check to not remove the query component on non-hierarchical URLs.

Also, `OCFContainer#isRemote(URL)` now returns false without further checks for `data` URLs.

Fixes #1536

Patch:
diff --git a/src/main/java/com/adobe/epubcheck/ocf/OCFContainer.java b/src/main/java/com/adobe/epubcheck/ocf/OCFContainer.java
index ebaf4d08..554d3f2e 100644
--- a/src/main/java/com/adobe/epubcheck/ocf/OCFContainer.java
+++ b/src/main/java/com/adobe/epubcheck/ocf/OCFContainer.java
@@ -131,7 +131,7 @@ public final class OCFContainer implements GenericResourceProvider
   public boolean isRemote(URL url)
   {
     Preconditions.checkArgument(url != null, "URL is null");
-    if (contains(url))
+    if (!url.isHierarchical() || contains(url))
     {
       return false;
     }
diff --git a/src/main/java/org/w3c/epubcheck/core/references/ReferenceRegistry.java b/src/main/java/org/w3c/epubcheck/core/references/ReferenceRegistry.java
index 7c7662d0..1e36ff21 100644
--- a/src/main/java/org/w3c/epubcheck/core/references/ReferenceRegistry.java
+++ b/src/main/java/org/w3c/epubcheck/core/references/ReferenceRegistry.java
@@ -49,7 +49,10 @@ public final class ReferenceRegistry
     if (url == null) return;
 
     // Remove query component of local URLs
-    if (url.query() != null && !container.isRemote(url))
+    // Note: we only do this for hierarchical URLs, to work around a bug
+    // in Galimatias that would transform a non-hierarchical URL into a
+    // hierarchical one. Queries for data URLs can safely be ignored here.
+    if (url.isHierarchical() && url.query() != null && !container.isRemote(url))
     {
       try
       {


Test patch:
diff --git a/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/content_001.xhtml b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/content_001.xhtml
new file mode 100644
index 00000000..5ad0c107
--- /dev/null
+++ b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/content_001.xhtml
@@ -0,0 +1,13 @@
+<!DOCTYPE html>
+<html xmlns:epub="http://www.idpf.org/2007/ops"
+	xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+	<head>
+		<meta charset="utf-8" />
+		<title>Minimal EPUB</title>
+	</head>
+	<body>
+		<h1>Loomings</h1>
+		<p>Call me Ishmael.</p>
+		<img src="data:image/gif;base64,XXX??aaa" alt="" />
+	</body>
+</html>
diff --git a/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/nav.xhtml b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/nav.xhtml
new file mode 100644
index 00000000..240745e6
--- /dev/null
+++ b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/nav.xhtml
@@ -0,0 +1,14 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xml:lang="en" lang="en">
+	<head>
+		<meta charset="utf-8"/>
+		<title>Minimal Nav</title>
+	</head>
+	<body>
+		<nav epub:type="toc">
+			<ol>
+				<li><a href="content_001.xhtml">content 001</a></li>
+			</ol>
+		</nav>
+	</body>
+</html>
diff --git a/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/package.opf b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/package.opf
new file mode 100644
index 00000000..0d1eec6e
--- /dev/null
+++ b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/EPUB/package.opf
@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<package xmlns="http://www.idpf.org/2007/opf" version="3.0" xml:lang="en" unique-identifier="q">
+<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
+  <dc:title id="title">Minimal EPUB 3.0</dc:title>
+  <dc:language>en</dc:language>
+  <dc:identifier id="q">NOID</dc:identifier>
+  <meta property="dcterms:modified">2017-06-14T00:00:01Z</meta>
+</metadata>
+<manifest>
+  <item id="content_001"  href="content_001.xhtml" media-type="application/xhtml+xml"/>
+  <item id="nav"  href="nav.xhtml" media-type="application/xhtml+xml" properties="nav"/>
+</manifest>
+<spine>
+  <itemref idref="content_001" />
+</spine>
+</package>
\ No newline at end of file
diff --git a/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/META-INF/container.xml b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/META-INF/container.xml
new file mode 100644
index 00000000..31878217
--- /dev/null
+++ b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/META-INF/container.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
+	<rootfiles>
+		<rootfile full-path="EPUB/package.opf" media-type="application/oebps-package+xml"/>
+	</rootfiles>
+</container>
diff --git a/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/mimetype b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/mimetype
new file mode 100644
index 00000000..57ef03f2
--- /dev/null
+++ b/src/test/resources/epub3/03-resources/files/data-url-with-unescaped-query-valid/mimetype
@@ -0,0 +1,1 @@
+application/epub+zip
\ No newline at end of file
diff --git a/src/test/resources/epub3/03-resources/resources.feature b/src/test/resources/epub3/03-resources/resources.feature
index 51a20fb0..d9cac61c 100644
--- a/src/test/resources/epub3/03-resources/resources.feature
+++ b/src/test/resources/epub3/03-resources/resources.feature
@@ -549,6 +549,12 @@
     Then error RSC-032 is reported
     And no other errors or warnings are reported
 
+  @spec @xref:sec-data-urls
+  Scenario: Verify a data URL having unesapced query-like component
+    See https://github.com/w3c/epubcheck/issues/1536
+    When checking EPUB 'data-url-with-unescaped-query-valid'
+    Then no other errors or warnings are reported
+
   ## 3.8 File URLs
 
   @spec @xref:sec-file-urls