RDFLib/rdfextras — Issue #4: "querying a tiny Sleepycat db takes 30 seconds"
Author: gjhiggins
Created: 2012-02-20 07:56:56+00:00
State: open

gjhigg...@gmail.com, 2010-06-09T10:42:42.000Z

Migrated from 'wontfix' rdflib issue :

#!/usr/bin/env python

from rdflib.graph import ConjunctiveGraph

# Test basic SPARQL aggregation of the RDFa data
# see also http://identi.ca/notice/17728953 http://identi.ca/notice/17729227

g = ConjunctiveGraph("Sleepycat")
g.open("store", create=True)
g.parse("http://inkdroid.org/journal/network", format='rdfa', lax=True)
g.parse("http://danbri.org/words/network", format='rdfa', lax=True)

q1="""PREFIX foaf:
SELECT ?src1 ?src2 ?x
WHERE {
GRAPH ?src1 { ?gr1 foaf:member [ foaf:openid ?x ] }
GRAPH ?src2 { ?gr2 foaf:member [ foaf:openid ?x ] }
FILTER ( ?src1 != ?src2 )
}"""

for src1, src2, x in g.query(q1):
print src1, src2, x

++ I wonder if other sparql enabled triplestores optimize that query any better.

FWIW, I used the example as a test target for my informal activity of checking the functionality of the no-longer-distributed persistences under rdflib 3.0.0.

Given the resource http://www.w3.org/2000/10/swap/pim/contact.rdf, all the stores listed below can consistently parse, store, read back and count the resource content.

The figures shown are for the test of basic SPARQL aggregation of RDFa data listed above.

SQLite - Parsing: 0.63s, Query: 24.73s
SleepyCat - Parsing: 0.61s, Query: 40.24s
BerkeleyDB - Parsing: 44.57s, Query: 45.70s
PostgreSQL - Parsing: 0.72s, Query: 47.68s
ZODB - Parsing: 0.67s, Query: 80.38s

MySQL - IntegrityError: 1062, Duplicate entry '350685713572143753-9684815388278768738-10362892297732962585-1036' for key 'kb_08683e75be_relations_poscIndex'

Many of the stores made available via the Redland plugin can perform the simple parse/write/read test of contact.rdf and can also parse+store the two graphs in the SPARQL test but I haven't yet figured out how to get a SPARQL query to return anything other than an empty SPARQLResults graph for these stores (or the SQLObject-mediated stores, for that matter).

Comment 1 by gromgull

Comment 2 by gromgull
Not much new info.

The good part is that this is NOT sleepycat specific, changing it to a memory store takes just as long:

{{{
#!/usr/bin/env python

import time
from rdflib import ConjunctiveGraph, plugin, query

plugin.register('sparql', query.Processor,
'rdfextras.sparql.processor', 'Processor')
plugin.register('sparql', query.Result,
'rdfextras.sparql.query', 'SPARQLQueryResult')

g=ConjunctiveGraph()

g.parse("http://inkdroid.org/journal/network", format='rdfa', lax=True)
g.parse("http://danbri.org/words/network", format='rdfa', lax=True)

start=time.time()

q1="""PREFIX foaf:
SELECT ?src1 ?src2 ?x
WHERE {
GRAPH ?src1 { ?gr1 foaf:member [ foaf:openid ?x ] }
GRAPH ?src2 { ?gr2 foaf:member [ foaf:openid ?x ] }
FILTER ( ?src1 != ?src2 )
}"""

for x in g.query(q1):
print x

print "Took %.2f"%(time.time()-start)
}}}

In fact, it takes LONGER - here Sleepycat takes 50s and memory 70s....

The problem is the asking for specific graphs - if you remove that the query is fast.