Patent ID: 8086484

Claim:
A method of operating a quality review internet application engine comprising: receiving, over the internet to a processor, a plurality of creative works from creators; presenting, over the internet from the processor to a reviewer, a pair of creative works to rate, wherein the pair of creative works is selected from the plurality of creative works; providing, to the reviewer from the processor, multiple choices for rating the pair of creative works based on variable relative measurements; receiving, by the processor from the reviewer, a single variable relative rating for the pair of creative works provided to indicate the degree to which the works differ; receiving, by the processor from the reviewer, a written review of a creative work rated by the reviewer; weighting, by the processor, the received single variable relative ratings based on reviewer expertise; comparing, by the processor, the weighted variable relative ratings; maintaining a plurality of control creative works of known high or low quality, wherein the control creative works are mixed among the creative works to be rated; identifying, by the processor, a fraudulent rating from the reviewer based on the received single variable relative rating being contrary to the known quality of one of the control creative works; marking all variable relative ratings received from the reviewer as fraudulent when any of the reviewer's variable relative ratings are identified as fraudulent ratings; repeating the presentation of pairs of creative works to the same or different reviewers at least until all works in the plurality of creative works have received at least one non-fraudulent variable relative rating to another creative work; sorting and ranking, by the processor, all creative works in order from best to worst based on weighted variable relative rankings; requesting, from the processor, over the internet to creators to rate individual written reviews according to how much effort written reviewers made; and increasing, by the processor, the frequency of inclusion of works from creators with higher review effort scores in presentation of pairs of creative work to be rated.