antoniogarrote/rdfstore-js — Issue #12: Turtle parser fails for large files
Author: agrueneberg
Created: 2012-01-03 18:34:19+00:00
State: open

After reading that you loaded 100,545 triples into memory for the LUBM benchmark I wanted to try loading my own data into rdfstore. However, `store.load` fails for a large Turtle file with more than 3,000 triples.
The general structure of my data looks like this:

@prefix cal: .
@prefix verb: .
cal:d31615229aa3c0fd05b1c5c25c46a987 verb:adjclose "10829.68" ;
verb:close "10829.68" ;
verb:date "2010-10-01" ;
verb:high "10907.41" ;
verb:low "10759.14" ;
verb:open "10789.72" ;
verb:volume "4298910000" .

Smaller subsets of the data with 100, 300, and 1,000 triples are imported flawlessly in approximately 1 sec, 5 secs, and 17 secs, respectively. For 3,000 triples and more, both Node and Chrome stay unresponsive. I tried to track down the problem and it seems like that the process gets stuck in the `parse` method of `TurtleParser`.

Maybe an event-based parser or chunking would help. In the meantime, could loading the data as JSON be an intermediate solution?