add the circbuf to speed up the de-duplication process. This time the sqlite3 snapshot works well