Big Data in U.S. Agriculture
Released: 2016-01-06T00:00:00

Summary:
Recent media and industry reports have employed the term big data as a key to the future of increased food production and sustainable agriculture. A recent hearing on the private elements of big data in agriculture suggests that Congress too is interested in potential opportunities and challenges big data may hold. While there appears to be great interest, the subject of big data is complex and often misunderstood, especially within the context of agriculture.
There is no commonly accepted definition of the term big data. It is often used to describe a modern trend in which the combination of technology and advanced analytics creates a new way of processing information that is more useful and timely. In other words, big data is just as much about new methods for processing data as about the data themselves. It is dynamic, and when analyzed can provide a useful tool in a decisionmaking process. Most see big data in agriculture at the end use point, where farmers use precision tools to potentially create positive results like increased yields, reduced inputs, or greater sustainability. While this is certainly the more intriguing part of the discussion, it is but one aspect and does not necessarily represent a complete picture.
Both private and public big data play a key role in the use of technology and analytics that drive a producer's evidence-based decisions. Public-level big data represent records collected, maintained, and analyzed through publicly funded sources, specifically by federal agencies (e.g., farm program participant records and weather data). Private big data represent records generated at the production level and originate with the farmer or rancher (e.g., yield, soil analysis, irrigation levels, livestock movement, and grazing rates). While discussed separately in this report, public and private big data are typically combined to create a more complete picture of an agricultural operation and therefore better decisionmaking tools.
Big data may significantly affect many aspects of the agricultural industry, although the full extent and nature of its eventual impacts remain uncertain. Many observers predict that the growth of big data will bring positive benefits through enhanced production, resource efficiency, and improved adaptation to climate change. While lauded for its potentially revolutionary applications, big data is not without issues. From a policy perspective, issues related to big data involve nearly every stage of its existence, including its collection (how it is captured), management (how it is stored and managed), and use (how it is analyzed and used). It is still unclear how big data will progress within agriculture due to technical and policy challenges, such as privacy and security, for producers and policymakers. As Congress follows the issue a number of questions may arise, including a principal one—what is the federal role?

paragraphs: T he term big data continues to be a buzzword in many industries. While frequently discussed, no commonly accepted definition of the term big data exists. In many cases, the terminology used to discuss the topic is not always consistent and can vary by industry and user. Based on the broad scope of the subject, this report will rely on a broad and general definition of big data compiled from resources within the agricultural community. The analysis also uses a terminology based on some of the more commonly used terms by government, research, and industry within the context of agriculture. 
While technology is a key and underlying component of big data use in the industry, its ongoing growth and evolution make it difficult to discuss in great detail. For the purposes of this report, technology—including hardware, software, and telemetry—and analytics are discussed in broad terms, using examples where relevant. These should not be considered exhaustive and do not imply endorsement of a named product or company. This report is also limited to the agricultural industry. Within the industry, farming and ranching operations use big data in different ways. Where possible both are discussed; however, an increasing focus is placed on the use of big data in the production of crops. This imbalance is not intentional, but rather reflects the availability of resources for this discussion. 
For the purposes of this report, the term big data will be discussed in two contexts: public and private. 
Public-level big data represent records that are collected, maintained, and analyzed through publicly funded sources, specifically by federal agencies (e.g., farm program participant records, Soil Survey, and weather data). Private big data represent records generated at the production level and originate with the farmer or rancher (e.g., yield, soil analysis, irrigation levels, livestock movement, and grazing rates).
Both private and public big data play a key role in the use of technology and analytics that drive a producer's evidence-based decisions. Both also present challenges, such as privacy and security, for producers and policymakers. 
subsections: section_title: Definition
paragraphs: As previously stated, a commonly accepted definition of the term big data does not exist. At first glance, it appears that the term is used to describe a large collection of records. This categorization, however, is generally considered to be an understatement. For something to fall into the category of big data it need not be big . Rather, the term big data is often used to describe a modern trend in which the combination of technology and advanced analytics creates a new way of processing information that is more useful and timely. In other words, big data is just as much about new methods for processing data as about the data themselves. Big data is viewed as dynamic and when analyzed can provide a useful tool in a decisionmaking process. 
In the context of agricultural production, big data generally refers to the use of technology and advanced analytics for processing data in a useful and timely way. Big data may significantly affect many aspects of the agricultural industry, although the full extent and nature of its eventual impacts remain uncertain. Many observers predict that the growth of big data will bring positive benefits through enhanced production, resource efficiency, and improved adaptation to climate change. While lauded for its potentially revolutionary applications, big data is not without issues. It is still unclear how big data will progress within agriculture due to challenges associated with both technical and policy issues. 
section_title: Background
paragraphs: The use of technology in agriculture has continued to grow since the early part of the 20 th century, when the industry shifted from the horse-drawn plow to mechanized tractors. The advent of plant genetics, chemical inputs, and, more recently, guidance systems has transformed the industry into one that is increasingly technology-intense and data-rich. The ability to generate, capture, and store data in the agricultural industry has continued to grow with the use of mobile technology and data management software. Additionally, external data sets are now readily available to the industry, allowing for a more complete picture of the world in which production agriculture occurs. 
The technological advances that make up the modern computing environment have contributed to debate about big data. While data collecting is not new, especially in the context of public data collection, only since the advent of more efficient, mobile technologies and the digitization of data have large records been able to be evaluated and analyzed in a timely and more useful way. One key hallmark of big data is that it requires the use of analytical tools to extract value from it. Without analysis, large quantities of data can be expensive, time consuming, and distracting. 
From a policy perspective, issues related to big data involve nearly every stage of its existence, including its collection (how it is captured), management (how it is stored and managed), and use (how it is analyzed and used). These three stages exist in both public and private data, and are discussed in greater detail in the private big data section below. 
Both private and public big data play a key role in the use of technology and analytics that drive a producer's evidence-based decisions. While discussed separately in this report, they are typically combined to create a more complete picture of an operation and therefore better decisionmaking tools. For example, companies that offer private big data products (discussed in the " Data Uses " section below) will combine agronomic, environmental, and operational data from multiple sources (i.e., public and private) in order to better describe current conditions and predict future results. This combination is one of the key reasons big data for agriculture is viewed as such a valuable tool. 
section_title: Congressional Action
paragraphs: On October 22, 2015, the House Agriculture Committee conducted a hearing on private big data in the agriculture industry. Panelists discussed benefits and concerns related to private big data. Data ownership and privacy were chief among concerns, but most panelists agreed that little to no government intervention was desired.
No bills have been introduced in the last two Congresses relating specifically to big data in agriculture. Several bills in the 114 th Congress could address issues that are potentially relevant to big data applications in agriculture, such as information sharing in cybersecurity, privacy, and notification of data breaches.
section_title: Public Big Data
paragraphs: Public agricultural data sets are traditionally created through the use of surveys, samples, and statistical analysis. Advances in technology and analytics through big data have expanded this traditional role but also highlight another, previously less-used source—administrative data. While some public data records created by traditional means (e.g., surveys) are statutorily required to meet the mission of an agency, administrative data records are generally byproducts of program administration. Whether the information is voluntarily or mandatorily collected also varies. 
Using big data to inform federal actions is of increasing interest to policymakers. Specifically, it is the use of administrative data that raises the possibility of analyzing existing data records in order to make more efficient and better-informed decisions about federal farm programs and activities. It could also provide additional insight into behavioral and societal aspects of U.S. agriculture that might currently be underexplored.
subsections: section_title: Key Players
paragraphs: The agricultural industry has a number of publicly generated data sources. Primarily, these sources are located at the U.S. Department of Agriculture (USDA). Other federal agencies, such as the U.S. Bureau of Labor Statistics, the National Oceanic and Atmospheric Administration (NOAA), and the National Aeronautics and Space Administration (NASA), also produce data sets important to the industry (e.g., meteorological information and satellite imagery). 
USDA is arguably the agricultural industry's largest collector, manager, and user of public big data. A number of agencies within USDA participate in one or more of these activities. Whether the agency is a collector, manager, or user of traditional data or administrative data can sometimes make identifying an agency's role in big data difficult. Depending on how big data is defined, most USDA agencies could be considered users, generators, or managers of big data in one way or another. 
subsections: section_title: Traditional Data
paragraphs: Traditional data is identified as the collection, management, and use of data obtained and analyzed through traditional means, such as surveys and sample collection. Examples of agencies and activities that generate traditional data include
National Agricultural Statistics Service (NASS) —collects, manages, and analyzes survey data through the Census of Agriculture; Economic Research Service (ERS) —collects, manages, and uses resource, production, and financial data through the Agricultural Resource Management (ARM) survey; Agricultural Research Service (ARS) —collects, manages, and uses scientific data related to agriculture through its mission of research and information access; Natural Resources Conservation Service (NRCS) —collects, manages, and uses soil, water, and geospatial data through the Soil Survey program; Agricultural Marketing Service (AMS) — collects, manages, and uses price and sales information through its market news programs; and World Agricultural Outlook Board (WAOB) —analyzes commodity and market data to develop the World Agricultural Supply and Demand Estimates (WASDE) report. 
section_title: Administrative Data
paragraphs: Some of these and other agencies have the capability of generating administrative data, generally as a by-product of program administration. While these agencies may not typically be considered big data agencies—their data is not generally made public and their data is not always aggregated or analyzed the same way as other big data—they nonetheless have similar issues as those of other big data agencies (i.e., security, privacy, technology capacity, and funding). Examples of these agencies include
Risk Management Agency (RMA) —collects, manages, and uses individual yield and loss information to administer the Federal Crop Insurance program; Farm Service Agency (FSA) —collects and manages individual producers' farm record data, federal payments, and loan information used in administering various farm programs; and NRCS —collects and manages conservation plans, geospatial data, and conservation program activities and payments. 
section_title: Benefits
paragraphs: A number of benefits to the agricultural industry are generally associated with public big data, including but not limited to
Authority —Much public big data is governed by various statutes and guidance documents that establish standards for quality. The use of common standards is believed to make public data more statistically reliable than some private sources, and public big data is therefore typically viewed as a trusted, authoritative source. Confidentiality —Public big data is governed by statutes and guidance documents that establish requirements for privacy. In many cases, agencies are able to anonymize data so as to protect individual producers' identities. While some privacy and confidentiality concerns may also present challenges for agencies, their big data sources are generally thought to be more transparent and regulated than private ones. Equal Access —Technology has not only impacted the ability to collect, store, and analyze data, but also makes it more transparent and publicly accessible. The use of the Internet and open data initiatives allows greater access to and use of public big data. Long- T erm Investment —The collection and use of data, including within the field of agriculture, has a long history in the United States. Frequently, the greatest value in traditional public data sets is derived from their ability to provide a baseline or benchmark for the industry as well as trends over time. The use of administrative data might provide new benchmarks and trends for areas previously not measured. 
section_title: Challenges
paragraphs: While many see public big data as a trusted and reliable source, it is not without its challenges, including but not limited to
Resources —The collection, management, and analysis of data are complex, cumbersome, and frequently resource intensive. Reduced federal budgets and staffing levels, combined with a technology adoption lag, have impaired the ability of some federal agricultural agencies to collect, manage, or use big data to its fullest potential. Supporters of public big data frequently request additional or sustained resource levels. Incorporation —Traditional public data collection is typically through the use of statistically valid surveys. Increasingly, the technology and analytics of big data have allowed surveys to expand and have also resulted in the creation of additional value in administrative data. This has called into question whether the traditional public data sources can or should incorporate new big data sources. Are traditional and administrative big data compatible? What could be lost or gained if the current use of proper sampling were to change, relative to participation-based data sets? Availability —While not considered 100% "open," most public data is available in large part because of technology (see "Equal Access" bullet above). These are typically limited to the more traditional big data sources; however, there is an increasing interest in making administrative big data open as well. In some cases, public data access is driven by statutory language prohibiting its release. Security —A number of high-level data breaches have raised concerns about the government's ability to protect its own big data sources. Under current law, all federal agencies have cybersecurity responsibilities relating to their own systems. While no known breach has occurred at USDA in recent years, some are concerned about the department's ability to protect information from similar attacks.
section_title: Private Big Data
paragraphs: Similar to public big data, private big data refers to the combination of technology and analytics used to process data. The key differences are who generates the underlying data, where that data is generated, and the purposes for which it is generated. In this report, private big data is limited to private data sets generated on the farm or ranch, by the producer, for enhancing the operation. There are a number of key players in the agricultural industry that make private big data possible through the use of technology (e.g., software and hardware) and advanced analytics (e.g., descriptive, predictive, and prescriptive).
Of the different stages of private big data—collection, management, and use—most people are familiar with the end result, or use of, the big data itself. This is frequently because it is the easiest to understand. It is where the producer most interacts with data—after it has been collected, analyzed, and turned into a usable form. For example, big data is used to create prescriptive plans that include recommendations on seed and fertilizer application rates, soil analysis, and localized weather reports. Other stages in the big data cycle, such as the collection and management of data, can be more complex, and frequently hold the biggest challenges for the agricultural industry. 
subsections: section_title: Key Players
paragraphs: The number and type of players involved with private agricultural big data are constantly evolving. For that reason, this report arranges the discussion by examples of players within each stage of the big data process—collection, management, and use. These stages do not operate independently of one another and are, in practice, fluid in nature. In some cases the same player is active in more than one stage, or in all stages, of the process. This section is not intended to serve as an exhaustive list, but rather a starting point to discuss who and what is involved in each stage of the big data process. 
subsections: section_title: Data Collection
paragraphs: The actual collectors of private big data are, in most cases, producers, who collect big data as part of their normal day-to-day activities. The collection stage, however, refers less to who collects the data than to how it is collected. Frequently, data collection involves physical technology, such as sensors, imagery, drones, radar, and other technologies all working together to provide detailed information about soil content, weeds and pests, sunlight and shade, nutrient deficiencies, moisture, and other factors. Physical technology forms only part of the data collection process, though. The other part is the network through which the technology communicates, typically the Internet. This is generally referred to as the "Internet of Things" (IoT) —networks of objects that communicate with other objects and with computers through the Internet. Both have led to questions related to ownership, privacy, and security, among others. 
Data collection is an ever-expanding area of big data and includes a number of key players, including but not limited to
Equipment Manufacturers —In many cases the manufacturers of traditional farm equipment (e.g., tractors, combines, and implements) are well positioned to expand into data collection technologies since, in many cases, the technology is an extension of the equipment already in use. C hemical Companies and Applicators —The application and use of nutrients and pesticides are increasingly complex. As producers look for ways to reduce cost and become more efficient, the application of additives is frequently an area for improvement. Chemical companies are playing an increasing role in the research and development of data collection tools and methods to improve application use. Multi- U se Technologies —The agricultural industry increasingly is finding value in technologies used by other industries. For example, some farmers in the dairy industry are exploring the use of radio frequency identification (RFID), which is more commonly used in the shipping and transportation industries, to track movement, production, feed, and disease outbreaks in herds. Frequently these companies have an interest in multiple industries, one of which could be agriculture.
section_title: Data Management
paragraphs: Private big data management generally covers the organization, administration, and governance of characteristically large volumes of data. The goal of data management is to ensure a high level of data quality and accessibility for the end user, and ultimately the next stage of data use through big data analytics (discussed in the next section). While most see storage as the key function of data management, it often also includes processing and security measures.
Key players, including private data managers and data banks, continue to expand. Others, such as data cooperatives, are entering the mix as producer-owned solutions for small to mid-sized operations that otherwise might face steep market access points individually. In some cases these players collect and organize the data for a fee, while in others they capture the value of the data by acting as a broker to trade or sell the data. Examples of the players include
Producers —In some cases the function of data storage and management is conducted at the farm-level. Concerns about security, and the sensitive nature of the data itself, have resulted in producers opting to store their data locally rather than through a third party or in a cloud computing environment. Data Collectors —In some cases the same companies that offer data collection services also offer data management services. They are generally affiliated with other agricultural products (e.g., equipment, seed, or chemicals). Independent Agricultural Data Banks —These are private companies that offer to store, organize, and, in some cases, analyze data for a fee. They are generally independent of a company that provides other goods or services to the agricultural industry. Additional services may also be included, such as weather information, location services, and benchmarking. Data C ooperatives —These are producer-owned information cooperatives (co-ops) that store, aggregate, and exchange data for their members. Similar to commodity co-ops, data co-ops pool members' data to create economies of scale and generate additional value and negotiating position. Data is then anonymized before being sold to interested parties. In return, members receive a portion of the proceeds from the sale as well as other infrastructure benefits provided by the co-op (e.g., storage and management).
section_title: Data Uses
paragraphs: The final stage of the big data process is the use of the data itself. This stage is where the actual tools are created and the value of private big data occurs for producers. A combination of big data sources is generally analyzed and packaged into an easily understandable and useful product. These can cover the spectrum of qualitative analytical products: descriptive products (e.g., those that provide a better or more advanced way of looking at an operation); prescriptive products (e.g., those that provide timely recommendations for operation improvement based on real-time and historical data); and predictive products (e.g., those that use current and historical data sets to forecast future events and returns). Depending on the size and complexity, this stage may encompass all three types of analytical products. 
The primary players and users of private big data products are the farmers and ranchers themselves. Other interested parties, however, recognize the value of big data and the ways it can be used beyond improving an individual operation (e.g., retail, marketing, or environmental improvement). Examples of key players in the use of private big data include
Farmers and Ranchers —The end user of private big data is frequently the farmer or rancher from which the data originated. The value is derived from a big data product that offers improved production (e.g., lower costs, increased yields, or reduced inputs). Retailers —These are the creators of the big data products themselves. They derive value by analyzing the data, packaging it into a useable and timely product, and selling it to the producer. In some cases these are parties from other stages (i.e., data collectors or data managers) or retailers of other products (e.g., equipment and seed companies) or both. Increasingly, retailers themselves might find value in big data or the products they create separate from the value derived from the producer. It is generally these groups that advocate for big data to get bigger in order to create a greater value for themselves and consumers. Industry Groups —The terms and uses of big data are often confusing and not well defined. Producer concerns related to privacy, security, and ownership abound, leaving many producers to look toward national commodity and agricultural industry groups for guidance regarding licensing language and data contracts. These groups are frequently active in establishing standards and issuing guidance for members on how to navigate big data. Environmental Interests —Economics is only one benefit from the use of private agricultural big data. The power of technology and information can also result in a positive environmental effect through reduced inputs (e.g., fertilizer, pesticides, and water) and efficiencies (e.g., reduced air emissions through reduced tillage overlap). This has caused various environmental interests to pay attention to agricultural big data and in some cases become active players.
section_title: Benefits
paragraphs: As the use of private big data continues to expand, the number and scope of benefits continue to grow. Examples of observed benefits include
Production Benefits —The number of benefits created by private big data—from increases in yields and greater efficiencies to reduced costs, inputs, and farm risk—continues to grow. Most of these benefits are directly received by the producers themselves through increased profits; however, spillover effects from increased farm revenue and environmental sustainability are also possible. Environmental Benefits —Similar to production benefits, the environmental benefits of private big data continue to develop. On-farm environmental benefits, including improved soil quality and water availability, can further benefit production as well as the environment. The reduction of inputs, including pesticides, fertilizer, water, and energy, often results in off-farm benefits through improved water quality and biodiversity. New and Expanded Business Opportunities —The rise of big data and the advances in technology have created a number of new business opportunities around its use. While much of the attention has been directed at the creation of these large, prescriptive, analytic tools, other businesses have developed around or been modified by one or more aspects of the farming operation. For example, in the custom applicator industry, the use of big data allows the third party (i.e., custom applicator) to invest in the technology in order to provide a more precise application of various inputs as a service. For instance, instead of producers investing in the equipment required for certain big data collection, they may contract with another party who can use the same equipment and networks to determine and apply required amounts of inputs. Real Time —Advances in technology have led to the real-time collection and processing of data and congruently to the ability to receive real-time decisionmaking tools. This can further expedite decisionmaking and possibly lead to increases in automation. 
section_title: Challenges
paragraphs: Private big data is constantly and rapidly changing. The complex nature of big data and the pace with which it is moving have created much confusion in the industry. This has led to a number of challenges for production agriculture and raises questions about whether the industry can manage these challenges and whether there is a role for federal involvement. Examples of challenges include
Ownership —This is a widely discussed concern within the agricultural community. Many believe that the owner of private big data is obvious; it is the producer from which it is collected. This is not always true, however, because of how the data moves through the different stages of collection, management, and use. For example, when data is analyzed and provided to the producer during the use stage as a decision tool for their operation, the producer would seemingly be the clear owner because they derive the most value from the data's use. It is not that simple, however, because that same data might also have value to a third party who is able to aggregate it and analyze it for a different purpose (e.g., a company might be interested in seed, yield, and input rates for determining future pricing of their product). Therefore the answer to the question of ownership generally lies in who owns and controls the value of the data. As previously discussed, raw, large data sets frequently hold little value to the end consumer. It is the combination of this data with that of quantitative analytics that creates the value. Other questions also include the following: who owns the secondary and tertiary uses of the data; can this ownership be limited or expanded, and in what way; who is the owner if the data is collected under a separate contract (e.g., custom harvesting or custom applicator); and are the ownership rights of the landowner different from the producer when they are not the same party; what are the options, if any, to not having or limiting data collection? Privacy —Agricultural production presents unique challenges related to localized competition for resources (e.g., access to land and water). In many cases neighbors are competing against one another for access to such resources. The concern of privacy is one that most producers acknowledge in the context of big data. Information related to yields and performance can hold incredible value. The independent nature of farming tends to drive privacy concerns. While most concerns are in respect to market competition, there are an increasing number of privacy concerns related to regulatory and non-agricultural interest groups coming into possession of a producer's data. Security —Many producers are concerned about the security implications of big data getting bigger and requiring the use of more advanced networks. As previously discussed, concerns about security and privacy have resulted in producers opting to store their data locally rather than through a third party or in a cloud computing environment. In some ways, however, this reduces the value of the data because in most cases, the bigger the data set, the greater the additional value created. With recent security breaches in both the public and private sectors it may be unreasonable to believe that all data is fully secure. Market —Unbalanced access to information can frequently distort the marketplace. Some speculate that the use of yield or sales data could create a market advantage for some input companies (e.g., seed and fertilizer companies). Others are concerned that big data will increase the competitiveness of rental agreements, while others see the potential for big data to affect commodity markets. Expense —Similar to the discussion under public big data, the collection, management, and analysis of data are not free. The use of big data products can come at a cost, albeit one which larger operations are able to afford with the anticipation of higher returns. Smaller operators, however, may face challenges related to economies of scale given the high price of equipment and services related to big data. Arguments have also been made that while some private big data products offer greater returns with greater investment, some smaller companies present small operators with options to capture the benefits of big data through more affordable technologies and services. Infrastructure —Recent surveys and studies indicate that, in general, rural areas tend to lag behind urban and suburban areas in broadband deployment. In the context of big data, the ability to send and receive data as well as the speed with which this occurs is central to its use and adoption. The federal government has played a role in addressing rural broadband infrastructure through financial assistance programs and spectrum policy for wireless connectivity. Technology System Failure —Users of technology know that it can be wonderful until it stops working. Basic system failures and limitations will continue to be a challenge for big data. In some cases the collection of the wrong data or inaccurate data can lead to poor decisions, rather than improved ones. Technology Adoption —The adoption of technology itself could prove a hurdle for the agricultural industry. While some producers are open to the adoption of technology on their operations, the sector's aging producer population may be slow to adopt it. 
section_title: Concluding Remarks
paragraphs: Both public and private big data can exist independently of one another; consequently, they are discussed separately in this report. This was done by design to illustrate an increasingly complicated topic. It does not, however, represent how observers think about big data in the context of agriculture. Most see big data in agriculture at the end use point, where farmers use precision tools to potentially create positive results like increased yields, reduced inputs, or greater sustainability. While this is certainly the more intriguing part of the discussion, it is but one aspect and does not necessarily represent a complete picture. 
Big data is a complicated topic, not only from a technological and analytical standpoint, but also from a legal, ethical, and regulatory standpoint. The number of key players continues to grow, as does the list of benefits and challenges. As Congress follows the issue a number of questions may arise, including a principal one—what is the federal role?