Patent ID: 8612460

Claim:
A computer-implemented method, the method comprising: identifying one or more second content entities associated with a first content entity wherein each of the second content entities has a respective association score that is based on a measure of a recent change in a search result rank of the second content entity, and wherein the measure of the recent change in the search result rank of the second content entity exceeds a rank threshold; providing the first content entity and one or more of the second content entities associated with the first content entity for presentation to a plurality of arbiters; receiving determinations from a plurality of the arbiters within a predetermined time limit period, wherein each determination indicates an association determination of a respective arbiter of the plurality of arbiters for a second content entity of the one or more second content entities and the first content entity; determining that a second content entity is to be disassociated from the first content entity based on, at least, the received determinations by determining that a number of the received determinations that each indicate an association determination to disassociate the second content entity from the first content entity is in proportion to the respective association score for the second content entity; and crediting one or more of the plurality of arbiters for which a determination to disassociate the second content entity from the first content entity was received.