Figured out how to fix the issue mentioned before
When removing found ngrams from reference_ngrams, it is done once for each time " ".join(ngram) appears in a reference
This appears to fix the problem, and given that there will only ever be a few references, does not cause any kind of
noticeable slowdown, a fraction of a second increase in computation time per reference is what I am gathering.