You could keep a dictionary of the SHA1 hashes of the files.  We already have this functionality in hashdist.