RDFLib/rdflib — Issue #94: querying a tiny Sleepycat db takes 30 seconds
Author: gjhiggins
Created: 2012-02-20 08:32:34+00:00
State: open

danbrick...@gmail.com, 2009-12-29T18:30:36.000Z

I'm running this script to query a pre-populated RDF store. The store only
has two small RDFa files indxed in it...

#!/usr/bin/env python

# queries an RDF quadstore

from rdflib.graph import ConjunctiveGraph

g = ConjunctiveGraph("Sleepycat")
g.open("store", create=True)

q1="""PREFIX foaf:
SELECT ?src1 ?src2 ?x
WHERE {
GRAPH ?src1 { ?gr1 foaf:member [ foaf:openid ?x ] }
GRAPH ?src2 { ?gr2 foaf:member [ foaf:openid ?x ] }
FILTER ( ?src1 != ?src2 )
}"""

for src1, src2, x in g.query(q1):
print src1, src2, x

TellyClub:wordpress danbri$ time ./querytest.py
/Users/danbri/working/rdflib/rdflib-read-only/rdflib/store/MySQL.py:10:
UserWarning: MySQLdb is not installed
warnings.warn("MySQLdb is not installed")
/Users/danbri/working/rdflib/rdflib-read-only/rdflib/store/AbstractSQLStore.py:5:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead
import sha,sys, weakref
/Users/danbri/working/rdflib/rdflib-read-only/rdflib/sparql/Query.py:1:
DeprecationWarning: the sets module is deprecated
import types, sets, sys
http://danbri.org/words/network http://inkdroid.org/journal/network
http://danbri.org/
http://inkdroid.org/journal/network http://danbri.org/words/network
http://danbri.org/

real	0m33.301s
user	0m32.178s
sys	0m0.376s

Here's the script that loads the db:

#!/usr/bin/env python

from rdflib.graph import ConjunctiveGraph

# Test basic SPARQL aggregation of the RDFa data
# see also http://identi.ca/notice/17728953 http://identi.ca/notice/17729227

g = ConjunctiveGraph("Sleepycat")
g.open("store", create=True)
g.parse("http://inkdroid.org/journal/network", format='rdfa', lax=True)
g.parse("http://danbri.org/words/network", format='rdfa', lax=True)

q1="""PREFIX foaf:
SELECT ?src1 ?src2 ?x
WHERE {
GRAPH ?src1 { ?gr1 foaf:member [ foaf:openid ?x ] }
GRAPH ?src2 { ?gr2 foaf:member [ foaf:openid ?x ] }
FILTER ( ?src1 != ?src2 )
}"""

for src1, src2, x in g.query(q1):
print src1, src2, x

Comment 1 by danbrick...@gmail.com
ps. I've just tried adding g.close() to the crawler, and rebuilding the db... same
result.

Comment 2 by ed.summers
I see the same behavior assuming the store has been populated the graph with some
triples (attached). I wonder if there is some sort of cartesian join going on w/ this
query?

ed@curry:~$ time ./foo.py
/home/ed/Projects/rdflib/rdflib/store/MySQL.py:10: UserWarning: MySQLdb is not
installed
warnings.warn("MySQLdb is not installed")
/home/ed/Projects/rdflib/rdflib/store/AbstractSQLStore.py:5: DeprecationWarning: the
sha module is deprecated; use the hashlib module instead
import sha,sys, weakref
/home/ed/Projects/rdflib/rdflib/sparql/Query.py:1: DeprecationWarning: the sets module
is deprecated
import types, sets, sys
http://danbri.org/words/network http://inkdroid.org/journal/network http://danbri.org/
http://inkdroid.org/journal/network http://danbri.org/words/network http://danbri.org/

real	0m23.067s
user	0m22.610s
sys	0m0.150s

Comment 3 by ed.summers
Some rudimentary profiling with the Python profile module shows that it is spending
most of its time calling DBCursor.next. That query generated 25,610,678 calls to
next()!

I wonder if other sparql enabled triplestores optimize that query any better...I've
attached the profile dump if you are interested in taking a look:

import pstats
s = pstats.Stats('sparql.prof')
stats.sort_stats('time')
stats.reverse_order()
stats.print_stats()

Comment 4 by ed.summers
For a point of reference I tried the same query, over the same data with arc2. arc2
doesn't return any results at all for your query (php attached) I wonder if this isn't
so much a bug so much as a whacked query...

Comment 5 by damian.s...@gmail.com
Just tried this in jena / ruby. I certainly get results, and much quicker than this.

Comment 6 by danbrick...@gmail.com
From MacTed in IRC, re whether the query is legit or not

E: query looks OK, based on quick test on
[http://uriburner.com/sparql?default-graph-uri=&should-sponge=&query=PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0A+SELECT+%3Fsrc1+%3Fsrc2+%3Fx+%0D%0A+WHERE+{%0D%0A+GRAPH+%3Fsrc1+{+%3Fgr1+foaf%3Amember+[+foaf%3Aopenid+%3Fx+]+}%0D%0A+GRAPH+%3Fsrc2+{+%3Fgr2+foaf%3Amember+[+foaf%3Aopenid+%3Fx+]+}%0D%0A+FILTER+%28+%3Fsrc1+!%3D+%3Fsrc2+%29%0D%0A+}&format=text%2Fhtml&debug=on&timeout=|URIburner]
(different data)

http://chatlogs.planetrdf.com/swig/2009-12-29.html#T20-58-23

The query seems to run (quickly) in Virtuoso if you go here:

http://uriburner.com/sparql?default-graph-uri=&should-sponge=&query=PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0A+SELECT+%3Fsrc1+%3Fsrc2+%3Fx+%0D%0A+WHERE+{%0D%0A+GRAPH+%3Fsrc1+{+%3Fgr1+foaf%3Amember+[+foaf%3Aopenid+%3Fx+]+}%0D%0A+GRAPH+%3Fsrc2+{+%3Fgr2+foaf%3Amember+[+foaf%3Aopenid+%3Fx+]+}%0D%0A+FILTER+%28+%3Fsrc1+!%3D+%3Fsrc2+%29%0D%0A+}&format=text%2Fhtml&debug=on&timeout=

though over different data...

Comment 7 by danbrick...@gmail.com
Filed this under 'problems' in F2F wiki,
http://wiki.foaf-project.org/w/F2FPlugin#Problems

Comment 8 by danbrick...@gmail.com
re ARC, http://twitter.com/bengee/status/7362577915

Q: @bengee is this query ok in arc2?
A: The query is fine and used to run in ARC, too. It'll work again in the next revision.

Comment 9 by ed.summers
Thanks for the SPARQL sanity :-)

I guess the next thing is to see if the query ran any faster w/ the old Bison-based
SPARQL implementation to see if the time that it is taking is the result of the
switchover to PyParsing.

Comment 10 by danbrick...@gmail.com
OK, so the ARC issue was fixed today:
'''
A new ARC2 revision is now available [1]:

* ARC2 (fix): splitURI was too greedy on special namespaces
like xhtml or atom (thx to Michael Panzer)
* SelectQueryHandler (fix): new block parentheses-ing broke
regex in FILTER rewriter (thx to Dan Brickley)

Cheers,
Benji

[1] http://arc.semsol.org/download/notes
'''

Comment 11 by eik...@gmail.com

Comment 12 by eik...@gmail.com

Comment 13 by eik...@gmail.com

Comment 14 by eik...@gmail.com
These issues involve bits that have been moved out of rdflib proper for now. We will re-open them
or move them to rdfextas as appropriate.