RDFLib/rdflib — Issue #162: Parsing nquads crashes for literals with language annotation
Author: gjhiggins
Created: 2012-02-20 08:34:09+00:00
State: open

cosmin.b...@gmail.com, 2011-07-16T19:43:12.000Z

What steps will reproduce the problem?
1. data = " "square kilometer"@en-US "
2. g = ConjunctiveGraph()
3. g.parse(data=data, format='nquads')

What is the expected output? What do you see instead?
should parse, but I get the following error:

....
File "build/bdist.macosx-10.4-x86_64/egg/rdflib_nquads.py", line 118, in parse
rdflib.plugins.parsers.ntriples.ParseError: Invalid line: '-US .'

Please use labels and text to provide additional information.

there seems to be a parser error. Should be somewhat relatively easy to fix (i suspect, but could be wrong). Alternatively an "ignore bad lines" option could be supported.

Cheers,
Cosmin

Comment 1 by cosmin.b...@gmail.com
it seems that the annotation 'en-US' is causing the problem, perhaps the '-' is the tricky part :]. Cosmin

Comment 2 by gromgull
I am confused - rdflib does not support nquads :)

But benosteen seems to have written a parser: https://github.com/benosteen/RDFLib-NQuads-parser/blob/master/rdflib_nquads.py

merging this into rdflib would probably make sense.

Comment 3 by gromgull
I added benosteen's nquad parser in an nquads branch, and also a serializer.

http://code.google.com/p/rdflib/source/browse/?name=nquads

Now I didn't actually check if this fixes your issue - will do soon.

Comment 4 by gjhigg...@gmail.com
It doesn't fix the issue but, by my reading of the RDF runes, RDFLib is correct (pace a less-than-useful error message) in only permitting lowercase language tags because uppercase would seem to be contraindicated ...

The W3C's RDF testcases document (http://www.w3.org/TR/rdf-testcases/#langString) presents this BNF spec, to which the RDFLib ntriples parser currently conforms, and some clarifying rubric:

"""

literal	::=	langString | datatypeString
langString	::=	'"' string '"' ( '@' language )?
datatypeString	::=	'"' string '"' '^^' uriref
language	::=	[a-z]+ ('-' [a-z0-9]+ )*
encoding a language tag.

...

optionally a language tag as defined by [RFC-3066], normalized to lowercase.

Note: The case normalization of language tags is part of the description of the abstract syntax, and consequently the abstract behaviour of RDF applications. It does not constrain an RDF implementation to actually normalize the case. Crucially, the result of comparing two language tags should not be sensitive to the case of the original input.

"""
To have the quad successfully parsed by the current code, the OP should simply lowercase the second language tag:

"square kilometer"@en-us

For RDFLib to accept uppercase language subtags, this small change to the ntriples.py "litinfo" regex would be needed ...

-litinfo = r'(?:@([a-z]+(?:-[a-z0-9]+)*)|\^\^' + uriref + r')?'
+litinfo = r'(?:@([a-z]+(?:-[A-Za-z0-9]+)*)|\^\^' + uriref + r')?'

I'm unsure what the ramifications of this latter change would be for any existing RDFLib-provided support for language tag comparisons.