RDFLib/rdflib — Issue #170: N3 serialization fails for certain Literals
Author: gjhiggins
Created: 2012-02-20 08:34:19+00:00
State: open

bernhard...@gmail.com, 2011-08-23T07:08:34.000Z

A Serialization/deserialization roundtrip of a certain class of Literals fails when there are both, newline characters and multiple subsequent quotation marks in the lexical form of the Literal (see below). In this case invalid N3 is emitted by the serializer, which in turn cannot be parsed correctly.

What steps will reproduce the problem?
>>> from rdflib.term import URIRef, Literal
>>> from rdflib.graph import ConjunctiveGraph
>>> g=ConjunctiveGraph()
>>> g.add((URIRef('http://foobar'), URIRef('http://fooprop'), Literal('abc\ndef"""""')))
>>> g.serialize(format='n3') # emits invalid N3
'@prefix ns1: .\n\nns1:foobar ns1:fooprop """abc\ndef\\""""\\"""" .\n\n'
>>> g2=ConjunctiveGraph()
>>> g2.parse(data=g.serialize(format='n3'), format='n3')
------------------------------------------------------------
Traceback (most recent call last):
File "", line 1, in
File "/site-python/rdflib/graph.py", line 988, in parse
location=location, file=file, data=data, **args)
File "/site-python/rdflib/graph.py", line 784, in parse
parser.parse(source, self, **args)
File "/site-python/rdflib/plugins/parsers/notation3.py", line 2257, in parse
p.loadStream(source.getByteStream())
File "/site-python/rdflib/plugins/parsers/notation3.py", line 892, in loadStream
return self.loadBuf(stream.read()) # Not ideal
File "/site-python/rdflib/plugins/parsers/notation3.py", line 898, in loadBuf
self.feed(buf)
File "/site-python/rdflib/plugins/parsers/notation3.py", line 924, in feed
i = self.directiveOrStatement(str,j)
File "/site-python/rdflib/plugins/parsers/notation3.py", line 939, in directiveOrStatement
if j>=0: return self.checkDot(str,j)
File "/site-python/rdflib/plugins/parsers/notation3.py", line 1478, in checkDot
str, j, "expected '.' or '}' or ']' at end of statement")
BadSyntax: at line 4 of <>:
Bad syntax (expected '.' or '}' or ']' at end of statement) at ^ in:
"...fix ns1: .

ns1:foobar ns1:fooprop """abc
def\""""^\"""" .

"
>>>

What is the expected output? What do you see instead?
In correct output all quotation marks in the Literal should be escaped, except the trailing three ones. This can be reliably parsed.
>>> data='@prefix ns1: .\n\nns1:foobar ns1:fooprop """abc\ndef\\"\\"\\"\\"\\"""" .\n\n'
>>> g3=ConjunctiveGraph()
>>> g3.parse(data=data, format='n3')
)>
>>> g.isomorphic(g3)
True
>>>

What version of the product are you using? On what operating system?
revision 37d3cbfff340

Please provide any additional information below.

Comment 1 by gromgull
I also came across this - it's even more broken with ntriples (which do not allow triple quoted strings)

I will sit down and work the regex-fu some day...

Comment 2 by bernhard...@gmail.com
For fixing the NT output I hacked a workaround some time ago, but this relies on valid N3 output. Its for sure not the most elegant way to do it, but I didn't find yet a case where it does not work (except the cases where invalid N3 is emitted).

Comment 3 by gjhigg...@gmail.com
Is this actually a bug or just an infelicity of expression, allowed by Python's extra-flexible string delimiters?

N3 strings are delimited by doublequote, singlequote does not need to be escaped - so the object of the statement in question might be more appropriately, if more cumbersomely, expressed and escaped as:

Literal("abc\\ndef\"\"\"\"\"")

And then the round-tripping flows smoothly ...

>>> from rdflib.term import URIRef, Literal
>>> from rdflib.graph import ConjunctiveGraph
>>> g=ConjunctiveGraph()
>>> g.add((URIRef('http://foobar'), URIRef('http://fooprop'), Literal("abc\\ndef\"\"\"\"\"")))
>>> g.serialize(format='n3') # would appear to emit valid N3 this time
'@prefix ns1: .\n\nns1:foobar ns1:fooprop "abc\\\\ndef\\"\\"\\"\\"\\"" .\n\n'
>>> g2=ConjunctiveGraph()
>>> g2.parse(data=g.serialize(format='n3'), format='n3')
)>
>>> g2.serialize(format="n3")
'@prefix ns1: .\n\nns1:foobar ns1:fooprop "abc\\\\ndef\\"\\"\\"\\"\\"" .\n\n'
>>> print(g2.serialize(format="n3"))
@prefix ns1: .

ns1:foobar ns1:fooprop "abc\\ndef\"\"\"\"\"" .

>>> g.isomorphic(g2)
True
>>>

Comment 4 by bernhard...@gmail.com
There is a mistake in your code, the string you used is not the same as in my example, as you inserted a double backslash.

>>> a='abc\ndef"""""'
>>> b="abc\\ndef\"\"\"\"\""
>>> a==b
False

The problem comes from the combination of double quotes and newlines:

>>> a='abc\ndef"""""'
>>> b="abc\ndef\"\"\"\"\""
>>> a==b
True

>>> from rdflib.term import URIRef, Literal
>>> from rdflib.graph import ConjunctiveGraph
>>> g=ConjunctiveGraph()
>>> g.add((URIRef('http://foobar'), URIRef('http://fooprop'), Literal("abc\ndef\"\"\"\"\""))) # note: \n but not \\n
>>> g2=ConjunctiveGraph()
>>> g2.parse(data=g.serialize(format='n3'), format='n3')
------------------------------------------------------------
Traceback (most recent call last):
File "", line 1, in
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/graph.py", line 988, in parse
location=location, file=file, data=data, **args)
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/graph.py", line 784, in parse
parser.parse(source, self, **args)
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/plugins/parsers/notation3.py", line 2257, in parse
p.loadStream(source.getByteStream())
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/plugins/parsers/notation3.py", line 892, in loadStream
return self.loadBuf(stream.read()) # Not ideal
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/plugins/parsers/notation3.py", line 898, in loadBuf
self.feed(buf)
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/plugins/parsers/notation3.py", line 924, in feed
i = self.directiveOrStatement(str,j)
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/plugins/parsers/notation3.py", line 939, in directiveOrStatement
if j>=0: return self.checkDot(str,j)
File "/Users/bs/Data/work/projects/gnowsis/gnowsisweb/site-python/rdflib/plugins/parsers/notation3.py", line 1478, in checkDot
str, j, "expected '.' or '}' or ']' at end of statement")
BadSyntax: at line 4 of :
Bad syntax (expected '.' or '}' or ']' at end of statement) at ^ in:
"...fix ns1: .

ns1:foobar ns1:fooprop """abc
def\""""^\"""" .

"

Comment 5 by gjhigg...@gmail.com
Sorry about that, I was focussed on the wrong area. You're quite correct, it was an issue of serialization - in Literal._quote_encode.

Changesets 7ac96f5a5a24 and 372e190cf28d fix this issue and add an issue-specific test.