Patent ID: 9081861
Filing Date: 2015-07-14
Classification: G06F

Abstract:
1. A computer-implemented method comprising: receiving a plurality of uniform resource locators (URLs), wherein the URLs identify content files, and wherein the URLs include at least one parameter, wherein a parameter consists of a subportion of the entire received URL; identifying fingerprints of the content files; determining a first entropy of values of the fingerprints conditional on values of a first parameter of the at least one parameter included by the received URLs, wherein the first entropy represents an uncertainty that remains about the values of the parameter, given that the fingerprints are known; and determining a second entropy of values of the first parameter conditional on values of the fingerprints, wherein the second entropy represents an uncertainty that remains about which fingerprint is pointed to by a URL that includes the first parameter, given that a value of the first parameter is known; determining that the first parameter is irrelevant to the identification of a unique content file by the URLs based, at least in part, on the first and second entropy values; and based on the determination that the first parameter is irrelevant to the identification of unique content file by the URLs, determining one or more canonical URLs for identifying one or more respective content files, wherein a canonical URL is equivalent, for purposes of identifying content, to different received URLs that reference the same content; and storing the one or more canonical URLs.