Source: https://www.kirkland.com/publications/article/2017/05/searching-for-web-crawlings-legal-boundaries
Timestamp: 2019-04-19 20:17:38+00:00

Document:
Webpages are a treasure-trove of useful information for financial firms and software companies that are able to capture it using web crawling (or scraping) technology. Yet, for over 20 years, courts have struggled to draw the line between the usefulness of such information and the rights of the content owners and website operators from which that content is derived. Once a niche issue, the increased use of this technology has compounded the disputes related to it.
Whether a finance firm engaged in quantitative analysis or a software company developing new search algorithms, technology-minded businesses are routinely and automatically accessing third-party websites every day using variations on web crawling to gather content and information. Generally, they start with a seed list of webpages from which they will request content, including HTML, text, image, and other files. Then, they copy the files and either extract specific data or the entirety of the files for later analysis.
For example, search engines generally identify hyperlinks and keywords from accessed webpages, add that information to their database for later analysis to improve their search algorithm, and continue to move across the Internet looking for new sources of content. Technology-savvy businesses, however, continue to develop new uses for search technology. Thus, while early efforts may have involved creating databases of factual information or gathering contact information for marketing solicitations, all manner of uses have been developed, including follow-on and copy-cat services that repeatedly access competitors' platforms as part of their functionality. In addition to potential copyright issues not discussed here, these new services may raise concerns for website operators if they disrupt the operators' services or damage their servers.
Financial professionals (particularly "quant analysts") use similar technology with algorithmic trading and quantitative analysis. Their databases generally are more targeted and only access specific files on a website that are to be used for financial analysis. In analyzing such information, their systems identify new sources of information and analyze existing resources to optimize trading by the organization and to adjust financial models, often automatically.
Depending on how such web crawling is conducted, it could implicate the rights of the content owner, as well as the website operator. Early cases analyzed web crawling technology through the lens of trespass law and similar rights.2 It, however, did not take long for content owners to bring claims under copyright law and other intellectual property disciplines against those using web crawling technology.3 Today, claims have been raised under a range of legal disciplines from breach of contract to trade secrets misappropriation to other forms of unfair competition.
As a result, while a web crawler that accesses websites without authorization or when authorization is revoked violates the CFAA, the courts might reach different results for a company that crawls webpages that permit public access but prohibit web crawling or other activities in which the company is engaged.
The ongoing litigations referenced in this article and those filed in the future may provide greater clarity on the bounds of legal web crawling. For now, businesses using these techniques should tread carefully lest they get caught in the CFAA's web.
1. Cf. Adrianne Jeffries, "How Google eats a business whole," OUTLINE, April 17, 2017, https://theoutline.com/post/1399/how-google-ate-celebritynetworth-com.
2. eBay v. Bidder's Edge, 100 F. Supp. 2d 1058 (N.D. Cal. 2000); Am. Online v. LCGM , 46 F. Supp. 2d 444 (E.D. Va. 1998).
3. Associated Press v. Meltwater , 931 F. Supp. 2d 537 (2013); Ticketmaster v. Tickets.Com, No. 99 Civ. 7654, 2003 WL 21406289 (C.D. Cal. March 7, 2003).
4. Facebook v. Power Ventures , 844 F.3d 1058 (9th Cir. 2016).
5. United States v. Nosal , 844 F.3d 1024, 1049 (9th Cir. 2016) (Reinhardt, J., dissenting).
6. EF Cultural Travel BV v. Zefer , 318 F.3d 58 (1st Cir. 2003); CouponCabin v. Savings.com, No. 2:14 Civ. 39, 2017 WL 83337 (N.D. Ind. Jan. 10, 2017), 2016 WL 3181826 (N.D. Ind. June 8, 2016); Craigslist v. 3Taps, 942 F. Supp. 2d 962 (N.D. Cal. 2013); Snap-on Bus. Solutions v. O'Neil & Assocs. , 708 F. Supp. 2d 669 (N.D. Ohio 2010).
7. Facebook v. Power Ventures , 844 F.3d 1058, 1067 (9th Cir. 2016). The private right of action under the CFAA also requires that the plaintiff "suffer damages or loss," 18 U.S.C. §1030(g), but loss has been broadly defined and may include the time that the website operators spends "analyzing, investigating, and responding to [the web crawler's] actions." Facebook, 844 F.3d at 1066.
8. Facebook, 844 F.3d at 1067; United States v. Nosal , 844 F.3d 1024, 1036 (9th Cir. 2016); CouponCabin, 2017 WL 83337, at *3.
9. Facebook, 844 F.3d at 1068.
10. Id. (discussing United States v. Nosal , 676 F.3d 854 (9th Cir. 2012)); United States v. Valle , 807 F.3d 508 (2d Cir. 2015); WEC Carolina Energy Solutions v. Miller, 687 F.3d 199 (4th Cir. 2012).
11. United States v. John , 597 F.3d 263, 272 (5th Cir. 2010); see also United States v. Teague , 646 F.3d 1119 (8th Cir. 2011); United States v. Rodriguez , 628 F.3d 1258 (11th Cir. 2010); EF Cultural Travel BV v. Explorica , 274 F.3d 577, 581-84 (1st Cir. 2001).
12. Musacchio v. United States , 136 S. Ct. 709 (2016). Two petitions for certiorari are currently pending before the court. Power Ventures v. Facebook, No. 16-1105 (U.S.); Nosal v. United States, No. 16A840 (U.S.).
13. Facebook v. Grunin , 77 F. Supp. 3d 965 (N.D. Cal. 2015).
14. Nosal, 676 F.3d at 861.
15. Facebook, 844 F.3d at 1068.
16. EF Cultural Travel BV v. Zefer , 318 F.3d 58, 62 (1st Cir. 2003).
17. Website operators use the Protocol to tell web crawler programs what files or folders should not be visited.
18. QVC v. Resultly , 99 F. Supp. 3d 525, 540 (E.D. Pa. 2015); Healthcare Advocates v. Harding, Earley, Follmer & Frailey , 497 F. Supp. 2d 627, 648 (E.D. Pa. 2007).

References: v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 §1030
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v. 
 v.