adds a csv generator for pages to be used with github.com/jexp/batch-import for neo4j
makes gen_csv use Neo4J 2.0 labels
adds proof of concept for translating to csv and then importing. speeds are about 100 pages / second, which is so much better than Cyper. Complete csv translation should only take 12 hours on my laptop
removes obselete files from previous iterations (graph, visual)
adds a little CLI for kapok with a generate command
adds pages/sec to kapok generate
Merge branch 'master' of github.com:aaasen/kapok
fixes the bug where some nodes would fail for no apparent reason by specifying ids and removing special characters
makes kapok generate print any errors that it encounters
makes the parser skip invalid lines instead of crashing on them
moves Generate to the generate package
adds cpu and memory profiling options. looks like most of the cpu usage comes from parsing pages with regexes, and the largest memory consumers are xml encoding and id tracking
adds cpu and memory profile graphs, generated with pprof
changes the way files are chunked from a line based system to a token based one. this results in a speedup of almost 4x, so now I'm getting 270 pages/sec on my laptop
moves away from go's builting xml parser in favor of an ad hoc one. memory consumption is down by two orders of magnitude, speed is up to 1500 pages/sec
adds newlines to the invalid character regex for pages
adds unit tests for isTitle
makes a Parser struct and adds a field for the number of bytes processed so far
removes generate tests
adds a check to make sure that all links look like real article titles