Technology Transfer: Federal Laboratory Consortium Should Increase Communication with Potential Customers to Improve Initiatives
Released: 2014-10-03T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The federal government spends about one-third of its annual $145 billion research and development budget at hundreds of federal agency labs. Technology generated by this research may have application beyond agencies' immediate goals if commercialized by the private sector. For example, federal research has contributed to innovative products, including antibiotics and the Internet. FLC—a nationwide consortium of federal labs—helps labs transfer technology to the private sector. In recent years, FLC created new initiatives to provide a clearinghouse—a central point for collecting and disseminating information—for technology transfer opportunities.

GAO was asked to review FLC's efforts to provide information on technology transfer opportunities. This report assesses (1) the extent to which FLC has communicated with potential customers when designing and implementing its clearinghouse initiatives, and (2) how FLC measured the results of those initiatives. GAO reviewed relevant laws and FLC guidance, and interviewed a nonprobability sample of officials from four federal agencies with the highest research budgets, and a spectrum of eight customer groups, among others.

section_title: What GAO Found
paragraphs: The Federal Laboratory Consortium for Technology Transfer (FLC) has taken steps to communicate with potential customers, including small businesses and entrepreneurs, but has not obtained feedback from them to assess their needs when designing and implementing technology transfer clearinghouse initiatives. This resulted in missed opportunities to better meet potential customer needs. For example, in 2012, when developing a web-based search tool to help potential customers identify relevant federal technology transfer opportunities across federal laboratories (labs), FLC discussed how to implement the tool with its federal member labs and agencies. However, FLC did not assess the information needs of potential customers to ensure the tool would provide relevant information in a format that customers consider useful, as called for by leading practices and federal internal control standards on communicating with and obtaining information from stakeholders. FLC officials said they conducted testing to ensure the new website functioned as intended before launching it, but did not involve potential customers in these tests. Moreover, after developing the tool, FLC did not communicate with potential customers to collect feedback from them consistent with leading practices regarding the extent to which the tool met their needs or how it might be improved before implementing it. Potential customers of FLC's initiatives expressed concerns about the extent to which FLC's recent web-based search tool would meet their needs, specifically noting that the tool:

affords customers limited ability to compare technologies across labs; and

FLC faces challenges in communicating with potential customers without also engaging its agency and lab members, given the relatively small size of FLC's annual budget and available staff. By working collaboratively with agency and lab members to collect feedback, FLC can enlist their help in enhancing the information provided through its initiatives.

FLC collects data on the use of its clearinghouse initiatives but has not developed and used performance goals and measures consistent with federal agency leading practices. For example, FLC collects data on the general use of its clearinghouse initiatives, such as the number of technology transfer inquiries it receives, the number of unique views of its web pages, and the average time spent on a web page. However, FLC has not developed performance goals or measures related to the key strategic goals to which its clearinghouse initiatives contribute. Without performance measures, FLC is unable to determine whether its initiatives are having their desired effect or how their performance might be improved. FLC also cannot fully demonstrate in its annual report to Congress its progress toward the achievement of its relevant strategic goals, limiting the information that the administration and Congress receive on the effectiveness of FLC's initiatives.

section_title: What GAO Recommends
paragraphs: GAO recommends, among other things, that FLC work collaboratively with agency and lab members to increase communication with potential customers to obtain feedback and improve its clearinghouse initiatives, and develop performance measures. FLC generally agreed with the report's findings and recommendations.

section_title: Letter
paragraphs: The federal government spends approximately one-third of its annual  $145 billion research and development (R&D) budget on research at  hundreds of federal laboratories (labs) to help meet agencies’ missions.  The knowledge and technological innovations generated by this research  regularly have application beyond agencies’ immediate, mission-related  goals if these innovations can be commercialized by the private sector.  Technology transfer is the term used to refer to the process by which  technology or expertise developed for one purpose is used for another  purpose. In some instances, it involves the transfer of legal rights, such  as licensing a government-owned patent to a private-sector entity.  Technology transfer also includes collaboration between private  companies and federal labs, for example, in the testing of advanced  batteries. In other instances, technology transfer involves the informal  transmission of information, knowledge, and skills through person-to- person or organization-to-organization interaction. Over the past  decades, through technology transfer, federal R&D has contributed to the  development of innovative products and processes in the commercial  marketplace, including advances in airplanes, antibiotics, bioengineered  drugs, computers, plastics, and the Internet, which have created new  industries and significant economic growth.

The Federal Laboratory Consortium for Technology Transfer (FLC) is a  member-based federal entity established in law in 1986 by the Federal  Technology Transfer Act. FLC’s membership is comprised of officials  from federal agencies and approximately 350 federal labs. FLC began  largely as a forum for the education, training, and networking of federal  technology transfer officials to promote the integration of technical  knowledge developed by federal departments and agencies into the U.S.  economy. Over time, FLC’s role has included serving as a  clearinghouse—a central point for collecting and disseminating  information—for federal technologies and assisting outside entities in  identifying technology transfer opportunities at federal labs. Its budget,  which is provided out of its members’ R&D budgets, was more than $2.9  million in fiscal year 2013.

With the lingering effects of the recent economic recession, both the  Obama Administration and congressional committees have emphasized  the importance of transferring available technological innovations out of  federal labs and into the hands of those who can commercialize them and  thereby create jobs and economic growth. For example, among other  actions, in October 2011, the President issued a memorandum to  executive agencies calling for strategies to increase the accessibility and  usefulness of information about federal technology transfer opportunities  and the development of a database of publicly available, federally owned  inventions. FLC took the lead in developing this database.

15 U.S.C. § 3710(e)(6) requires FLC to report on its expenditures and activities annually  to the President, the appropriate authorization and appropriation committees of Congress,  and each agency that provides funding for FLC. said they are nearing completion of a new web-based search tool called  FLCBusiness that is intended to allow users to search for technology  development programs, available funding, and to learn about the  capabilities of federal lab facilities.

Federal technology transfer remains a priority for the President and  Congress. Most recently, in his fiscal year 2015 budget, the President  announced 15 cross-agency priority goals. The focus of one of these  goals, the Lab-To-Market goal, is to increase the economic impact of  federally funded R&D by improving and accelerating technology transfer  from federal labs to the commercial marketplace. Recent congressional  committee hearings have also addressed making opportunities for  technology transfer more readily available to private sector businesses.

You asked us to review FLC’s efforts to make information about  technology transfer opportunities available through its technology  clearinghouse initiatives. This report assesses (1) the extent to which FLC  has communicated with and obtained feedback from potential customers  when designing and implementing its clearinghouse initiatives and (2)  how FLC measured the results of those initiatives.

To assess the extent to which FLC has engaged with potential customers  when designing and implementing its clearinghouse initiatives, and how  FLC measured the results of those initiatives, we reviewed FLC’s  authorizing legislation, bylaws, guidance, and other FLC documents  regarding its goals, organization, initiatives, and metrics. We also  interviewed FLC officials, including members of FLC’s national and  regional leadership, such as the FLC chair and several of FLC’s six  regional coordinators. We used this information to identify how FLC  develops initiatives and measures results, including how FLC engages  with potential customers that may want to participate in technology  transfer opportunities, such as small and large businesses,  entrepreneurs, and organizations focused on state and local economic  development. We compared FLC’s efforts to design, implement, and  measure the results of its initiatives to leading federal practices outlined in  Executive Order 12862, HowTo.gov, the GPRA [Government  Performance and Results Act] Modernization Act of 2010 (GPRAMA),  and GAO standards for internal control in the federal government.

In addition, we conducted interviews using a standard set of questions  with nonprobability samples of officials from the technology transfer offices at four federal agencies:  the National Aeronautics & Space Administration (NASA), Department  of Energy (DOE), National Institutes of Health (NIH), and Department  of Defense (DOD) to obtain their views on FLC and its clearinghouse  initiatives;  representatives from six technology transfer organizations to learn  about their technology transfer clearinghouse initiatives and key  features their customers found useful; and  representatives from eight customer groups who could potentially use  the information provided through FLC’s clearinghouse initiatives.

Our selection of agencies, technology transfer organizations, and  potential customer group representatives was based on nonprobability  samples and their views are not generalizable to those we did not  interview but provide illustrative examples. We also attended FLC’s  annual meeting in April 2014. Additionally, we interviewed officials with  the Office of Science and Technology Policy (OSTP) to obtain their views  on FLC’s role.

We conducted this performance audit from September 2013 to October  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Technology transfer (i.e., the process by which technology or expertise  developed in federal labs for one purpose is used for another purpose) is  a collaborative effort between the federal labs and outside organizations  and traditionally includes:

Technology licensing—businesses can license federal technologies,  such as patented inventions, in order to integrate them into their  products; although not every new or improved technology developed  through federal lab research can be commercialized.

Cooperative research and development agreements (CRADA)— under a CRADA, federal labs collaborate with nonfederal partners to  carry out research projects that will directly benefit lab missions and  the partners’ R&D goals, and a lab may contribute personnel,  equipment, or other resources to a project, while its CRADA partners  may contribute funds or resources, or both.

Work for others—under a nonfederal work-for-others agreement, a  federal lab is paid to conduct research on behalf of a nonfederal  sponsor, such as a university or corporation.

User facility agreements—under a user-facility agreement, scientists  or researchers from outside organizations can use federal lab  equipment for their own research, sometimes in collaboration with  federal lab staff.

According to a report prepared by the Interagency Workgroup for  Technology Transfer in response to the Presidential Memorandum of  2011, other important means of technology transfer include publishing  scientific articles and other technical publications, as well as making  software developed by federal researchers available for download.Figure 1 provides an example of how technology developed in a federal  lab for one purpose may be transferred to the private sector for another  purpose.

Since the early 1980s, the federal government has enacted several laws  related to technology transfer from federal labs. One of the first  technology transfer laws, the Stevenson-Wydler Technology Innovation  Act of 1980, articulated the need for a strong national policy supporting  domestic technology transfer and required federal labs to establish an  office of research and technology applications and devote budget and  personnel resources to promoting technology cooperation and the  transfer of federal technologies. These offices (which we refer to as  technology transfer offices) are generally responsible for coordinating  each lab’s efforts to identify technologies available for transfer, obtaining  patents or other legal protections for lab technologies, and negotiating  technology transfer agreements with outside parties. In addition, the  administration has emphasized the importance of technology transfer  from federal labs. For example, in his October 2011 memorandum,  President Obama directed federal agencies to take actions to accelerate  technology transfer and support private sector commercialization in an  effort to promote innovation and increase the economic and societal  impact of federal R&D investments. Moreover, President Obama’s fiscal  year 2015 budget identified a new Lab-To-Market cross-agency priority  goal to support innovation, economic growth, and job creation.  Specifically, the goal outlines several actions to, over the next four years,  accelerate and improve the transfer of technologies from federal labs to  the commercial marketplace, including optimizing the management,  discoverability, and ease-of-license of federally funded patents and  increasing the utilization of federally funded research facilities by  entrepreneurs and innovators.

Even with this emphasis on technology transfer, we and others have  identified a number of challenges associated with technology transfer  from federal labs. These challenges include the following:

Technology transfer is often not a priority for lab managers and  funding tends not to allow for widespread technology transfer  outreach.

Scientists may not understand the potential commercial applicability of  their innovations.

Rules and requirements that labs must follow in transferring  technology, as well as a lack of standardization thereof, increase the  complexity and length of time of the negotiation process with potential  partners, and create a disincentive to working with the labs.

Lab technologies are often not well-developed enough for use in  market-ready products and may require investment of additional time  and money to develop.

Companies are often not aware of the potentially useful technologies  being developed in federal labs.

Companies and entrepreneurs often prefer access to and personal  contact with researchers to turn an early stage innovation into a  market-ready product, and this may be difficult because conflict-of- interest rules that ensure that public employees do not unfairly benefit  from federally funded inventions prevent scientists from being  involved in business ventures.

One of the federal efforts to facilitate technology transfer from federal labs  to the private sector and others was the creation of FLC. FLC was  established by law in the Federal Technology Transfer Act of 1986 to,  among other things, (1) develop training for federal lab employees  engaged in technology transfer and (2) facilitate communication and  cooperation between federal lab technology transfer offices and regional,  state, and local technology transfer organizations. The Federal  Technology Transfer Act also established requirements for FLC’s  membership and funding.

Since its establishment in 1986, FLC’s mission has evolved. Earlier in its  history, FLC’s efforts, in large part, focused on providing training and  networking for federal technology transfer staff. However, FLC has, in  recent years, taken on more responsibility for helping federal agencies  provide potential customers with information on available technology  transfer opportunities and for functioning as a clearinghouse. These  efforts were bolstered by the President’s 2011 memorandum calling for  an online clearinghouse of federal technologies, and FLC took the lead  role in establishing such a tool. Changes to how FLC defines its mission  from being primarily focused on the federal labs to expanding its focus  beyond the federal labs reflect this transition. For example, in FLC’s 2009  annual report to the President and Congress, FLC defined its mission as  to help federal labs transfer technologies developed through the federal  government’s R&D efforts. In FLC’s 2012 annual report to the President  and Congress, FLC defined its mission as to promote and facilitate the  rapid movement of federal lab research results and technologies into the  mainstream of the U.S. economy.

FLC’s standing committees include a state and local government committee and a  communications committee, among others. NIST is the legislatively designated FLC host  and handles FLC’s finances.

Executive Board. FLC is organized into six regions, and each region has  a regional coordinator, its own website, social media outlets, regional  newsletter, and can implement its own initiatives including training and  information meetings for its members.

section_title: FLC Has Not Fully Communicated with and Obtained Feedback from Potential Customers when Designing and Implementing Clearinghouse Initiatives
paragraphs: FLC has taken steps to communicate with potential customers, including  small business and entrepreneurs, but has not fully communicated with  and obtained feedback from them to assess potential customers’ needs  and incorporate leading practices when designing and implementing  technology transfer clearinghouse initiatives, resulting in missed  opportunities to better meet potential customer needs.

As stated in FLC’s 2012 annual report, FLC carries out its mission to  promote and facilitate the rapid movement of federal lab research results  and technologies into the mainstream of the U.S. economy by  implementing two primary clearinghouse initiatives—the Technology  Locator service and Available Technologies tool—that provide information  on available technology transfer opportunities. Currently, FLC is  developing a third initiative—FLCBusiness—to provide additional  information on available technology transfer opportunities. While FLC  officials said they view federal agencies and labs as FLC’s main clients  for its training and other activities, the principal customers of the  information provided through FLC’s clearinghouse initiatives include the  private sector, and state and local government organizations, among  others, specifically:

Technology Locator service. Since 1987, FLC has provided a  Technology Locator service whereby an FLC staff member will try to  connect customers with relevant labs based on technology needs  submitted by customers through a phone call, e-mail, or online form.  FLC’s online form provides a space for potential customers to explain  their technology request or problem, desired results, and expectations  of a federal lab, among other information. Using this information, FLC  will identify laboratory resources and contacts that can respond to  specific requests and will facilitate communication between the  potential customer and the lab contacts. Once contact is made, further  discussion or negotiation of a technology transfer agreement occurs  between the potential customer and the lab representative. FLC  officials said customers who submit requests usually receive a  response within 48 hours.

Available Technologies tool. In 2012, FLC released the Available  Technologies tool to provide a web-based search tool for collecting  and disseminating information on publicly available federally owned  inventions and, when available, licensing agreements. The tool can  help potential customers identify more than 20,000 available lab  technologies across approximately 225 labs by searching the  websites of 13 participating agencies. FLC reported spending  approximately $30,000 for a contractor to develop the tool, which  relies on a customized Google search tool to link potential customers  to the existing information on agencies’ technology transfer websites  based on key words the customer enters. The key word search can  be refined to include or exclude certain words and may also be  focused on technologies published within a certain time period. In  addition, the Available Technologies tool webpage includes a listing of  the latest technologies available for transfer from federal labs.  According to FLC, it plans to continue to develop this tool as it gains  access to additional data from federal labs through various  government-wide initiatives.

FLCBusiness. FLC is developing a new web-based search tool called  FLCBusiness that is scheduled to become publically available in fall  2014, according to FLC officials, and is intended to allow potential  customers to search for technology development programs, available  funding, and to learn about the capabilities of federal lab facilities. The  tool is designed to allow potential customers to perform key word  searches for labs, or funding or technology development programs,  and to refine searches to a specific agency, geographic location, or  technology area. The tool also is to provide links to additional  information on licenses, CRADAs, and other technology transfer  methods. As of August 2014, FLC reported spending approximately  $150,000 to develop the tool.

In addition, FLC has utilized various communication tools, such as social  media, outreach by regional FLC members to local technology transfer  organizations, its annual meeting, and an online newsletter and news  forum, to communicate with potential customers and promote its  clearinghouse initiatives and technology transfer success stories. For  example, in 2013, FLC began providing an opportunity for industry  representatives from large corporations to present their companies’  technology needs to participating federal lab officials as part of its annual  meeting, according to an industry representative we interviewed. During  the 2014 FLC annual meeting, we observed that industry representatives  described technology needs ranging from advanced materials to robotics.  FLC used the opportunity to do a presentation to publicize its  clearinghouse initiatives and, after the presentation, industry  representatives and agency members had an opportunity to interact and  exchange contact information.

FLC’s process to design and implement its primary clearinghouse  initiatives involves FLC committees and regional coordinators annually  proposing new ideas for consideration by FLC’s Executive Board.  According to FLC officials, the Executive Board considers proposals in  relation to available resources and conducts outreach to agency and lab  members. For example, in 2012 when developing its Available  Technologies tool, FLC reached out to its agency and lab members when  considering possible options for designing a clearinghouse of available  technology transfer opportunities. After receiving input from agency and  lab members, including concerns about the burden, ability, or willingness  to compile data on available technology transfer opportunities for FLC,  FLC officials said they designed the Available Technologies tool to  minimize the burden on agencies to collect and share their data.  Specifically, rather than ask agencies to gather relevant data on federally  owned inventions, which were not already centralized and did not exist in  a standard format, FLC developed a web-based search tool that would  direct potential customers to existing information on federal labs’  websites. FLC officials determined this tool (1) would not require FLC  resources to manipulate agency data and (2) would not require agency  members to update information separately from what they already  planned to include on their websites. FLC officials said they considered  FLC to be best-positioned to take the lead in developing the kind of  clearinghouse tool described in the President’s 2011 memorandum  because it was an interagency body, had already developed a  clearinghouse tool when the memorandum called for the creation of a  similar database, and because FLC’s funding for the clearinghouse tool  was potentially more stable than if the tool were funded by a specific  agency appropriation that needed to be approved each fiscal year.

However, FLC did not assess the information needs of potential  customers of federal lab technologies to ensure the Available  Technologies tool would provide relevant information in a format that  customers consider useful, as called for by leading practices. Federal  leading practices outlined in Executive Order 12862 and HowTo.gov  direct agencies to assess customer needs when designing and  implementing initiatives. For example, Executive Order 12862 directs  agencies to identify and survey customers to determine the kind and  quality of services they want in an effort to develop and meet customer  service standards. HowTo.gov also outlines the need to engage with  customers to collect and address customer feedback through, for  example, online surveys or focus groups, and indicates a need to conduct  regular user testing of government websites with actual customers. In  addition, according to federal standards for internal control, federal  management should ensure that there are adequate means of  communicating with and obtaining information from external stakeholders  that may have a significant impact on an agency’s achieving its goals.  FLC has not taken such steps as part of its approach to design and  implement its clearinghouse initiatives. For example, FLC officials said  that when designing the Available Technologies tool, they conducted  testing to make sure searches on the site functioned as intended, but that  potential customers were not involved in this testing. Moreover, after  developing the tool, FLC did not communicate with potential customers to  collect feedback from them consistent with leading practices regarding  the extent to which the tool met customers’ needs or how it might be  improved before implementing it. Similarly, we found that, while FLC is in  the final stages of developing its new web-based search tool,  FLCBusiness, FLC officials said potential customers had not seen early  versions of the tool and had an opportunity to provide feedback. Instead,  FLC officials said the tool was extensively tested by its web-design  contractor and reviewed by members of its Executive Board. FLC officials  told us that they demonstrated the tool for agency members and sent the  early, or beta, version to laboratory representatives for user testing and  data input. The tool was introduced at FLC’s 2014 annual meeting,  although it was not clear whether any feedback was collected from  potential customers based on our observation of the meeting. Without  communicating with potential customers on FLC’s tools, allowing for  customers to test them, and collecting feedback on their usefulness, FLC  is not following internal control standards or leading practices and does  not have assurance that its tools will meet customer needs.

According to some representatives from customer groups and one technology transfer  organization, federal lab patents may not afford adequate intellectual property protection  to support commercialization because they sometimes protect only the specific version of  the invention relevant to the lab’s mission from infringement, which would potentially allow  a competitor to offer a similar product. across multiple labs, but these may be difficult to identify through  FLC’s Available Technologies tool because labs and agencies  describe technologies differently. For example, some technologies  from NASA are presented as scientific research results, while  technologies with similar search terms from DOD are presented as  potential product components. A representative from one technology  transfer organization we interviewed said that a small company he  worked with would not have been able to identify relevant patents on  its own because the company needed patents from 11 different  federal labs to cover different components of a product it was  developing to filter water used in oil and natural gas drilling. Other  representatives of potential customer groups and technology transfer  organizations said that while large companies have the time and  money to search for technologies across numerous labs, smaller  companies generally do not.

Limited information on technologies’ market relevance: Customers  may benefit from more information on the potential market relevance  of technologies, which is not always provided in the agency and lab  information available through FLC’s Available Technologies tool. For  example, one representative from a technology transfer organization  said that clearinghouse search results that are highly technical—as  with those in FLC’s Available Technologies tool that are primarily  based on patents—may make sense to research scientists who know  relevant technical terms, but not necessarily to entrepreneurs who  may not be aware of the technical key words relevant to a variety of  potential solutions to their technology needs. In addition, some  customer representatives said information on market relevance is  useful because federally patented technologies are generally not  close to being market-ready, and additional R&D is needed to bring  products to market.

We observed a few of these limitations during a keyword search we  conducted using FLC’s Available Technologies tool. In particular, when  we attempted to identify similar technologies across agencies and labs  based on a keyword search, the results were not standardized, and some  results from some agencies offered different information than others. For  example, a result from one agency provided a link to a 15-page research  publication related to the technology, while a result from another provided  a one-sentence overview of the technology’s potential market relevance.  Moreover, not every result included contact information for researchers  affiliated with the technology. For instance, one result provided a list of  links to agency-wide opportunities for partnerships between the private  sector and the agency, but this list did not include contact information for  any researchers, nor was it clear which links on the page related to the  original search term. Figure 2 shows the different types of information that  may be included in FLC search results.

FLC has taken steps that may help address some of the issues  representatives of technology transfer organizations and other potential  customer groups identified and that we observed. For example, in 2012,  FLC began a systematic effort to identify potential customers of federal  technology transfer opportunities, including entrepreneurs and small  businesses, in different regions of the country. This effort started with an  initial study conducted in FLC’s Midwest region that found a number of  unmet customer needs related to technology transfer, including a need for  technical and financial help for entrepreneurs and improved customer  service and dissemination of information on available technologies.  Although this effort represented an important step in engaging with  potential customers, it did not focus on obtaining their input on what  information they might want from an FLC clearinghouse tool and in what  format. In addition, FLC’s clearinghouse tools, taken together when  FLCBusiness becomes publically available, could provide potential  customers with more information on available technology transfer  opportunities than FLC currently provides through the Available  Technologies tool. For example, FLC anticipates using FLCBusiness to  provide information on lab facilities, which representatives of potential  customer groups and other technology transfer organizations noted is  generally missing from the Available Technologies tool. Moreover, if a  potential customer is unable to find a facility or technology in  FLCBusiness or the Available Technologies tool, the potential customer  could contact the Technology Locator service to get in touch with a  laboratory representative. Figure 3 depicts how FLC’s clearinghouse tools  could work together to help potential customers identify technology  transfer opportunities to meet their needs. However, since FLC has not  fully communicated with and obtained feedback from potential customers  in developing FLCBusiness, it lacks assurance that FLCBusiness or its  suite of clearinghouse tools will meet customers’ needs.

FLC faces challenges in its ability to more fully communicate with  potential customers to facilitate technology transfer from federal labs  without also engaging FLC’s agency and lab members. FLC’s web-based  clearinghouse tools depend on information provided by agency and lab  members to help build or update the databases on which the tools are  based. For instance, FLC officials said the Available Technologies tool  depends on information about agency technologies available to FLC  through existing agency websites. As noted above, this information often  varies across agencies and can present similar technologies differently or  use different technical terminology, and representatives of potential  customers said these differences could make it difficult for outside parties  to find the most appropriate information to meet their needs. Agency  officials from NIH, DOE, DOD, and NASA also told us that it could be  challenging for them to justify gathering new or different information  related to technology transfer opportunities until FLC demonstrated that  customer needs would be better met with additional information  presented in a different format. They also noted that FLC may not be well- positioned to conduct additional customer outreach to collect feedback  without working with agency and lab members, making it difficult for the  consortium to improve its existing search tools on its own. Some agency  officials said that any improvements based on customer feedback would  require a significant collaborative effort on the part of FLC and its agency  and lab members, given the relatively small size of FLC’s annual budget  and available staff. By working collaboratively with agency and lab  members to collect feedback on its initiatives’ usefulness, FLC can take  advantage of its members’ knowledge and expertise to better  communicate with potential customers consistent with federal leading  practices.

Even with the challenges FLC faces, several FLC and agency officials as  well as representatives of other technology transfer organizations  supported FLC’s clearinghouse initiatives. Representatives from some  other organizations that provide information on federal technology  transfer opportunities said they only have information about technology at  a small number of federal labs, and one organization said it charges  users or labs a fee for use of its tools. In contrast, some FLC and lab  officials said FLC’s clearinghouse initiatives serve an important function  because they have the potential to provide more comprehensive  information about technology transfer opportunities at federal labs at no  cost to potential customers. One FLC official also noted that, while the  Available Technologies tool has limited capability since the data it reports  are unstructured and not standardized in any way, the relatively low cost  of the tool and the quick time frame over which it was developed made it  an important interim step to providing a global search capability for  available federal technologies.

section_title: FLC Has Not Measured the Performance of Its Clearinghouse Initiatives
paragraphs: Although FLC collects some data related to the use of its clearinghouse  initiatives, it has not used performance goals and measures consistent  with federal agency leading practices to measure results and does not  have the information necessary to determine the extent to which its efforts  help achieve FLC’s overall strategic goals.

FLC tracks the number of requests for assistance it receives through its  Technology Locator service by (1) the technology area the customer is  interested in (e.g., energy, pharmaceutical, or manufacturing); (2) the  nature of the customer’s interest (e.g., informational or technology need);  and (3) type of customer that submitted the request (e.g., academic,  investor, or small business). According to FLC officials, in 2013, the  Technology Locator service received 172 requests for assistance. This  number represented a decrease from the 203 requests for assistance  FLC received in 2012. In addition, FLC collects success stories, which it  posts on its website. For example, one of the success stories describes a  business owner who placed a Technology Locator request to ask for help  with software development—the Locator successfully put her in touch  with relevant staff at federal labs who were able to offer assistance. FLC  also collects data on the general use of the Available Technologies tool,  such as the number of unique views of web pages and the average time  spent on a web page. One report showed that, from January 1 through  July 14, 2014, FLC’s Available Technologies tool had 2,010 page views  from unique customers and 3,240 page views overall.

FLC has developed a number of strategic goals over the years. The last  strategic plan FLC developed was for 2009 and it contained three goals:  (1) develop FLC members to be leaders in technology transfer; (2) foster  an environment for technology transfer; and (3) enhance the professional  organizational structure of FLC. More recently, in its 2012 annual report to  the President and Congress, FLC identified four strategic goals for all  FLC activities: (1) outreach/communication, (2) training/education, (3)  networking, and (4) recognition. An FLC workgroup is currently working  on a new strategic plan, which FLC reported will not be ready until the  end of 2014.

FLC has not, however, developed performance goals or measures to  align its initiatives with its strategic goals, as recommended by leading  practices to improve government performance and results. In particular,  GPRAMA requires that agencies develop strategic plans that include  general, long-term goals and objectives, along with an agency  performance plan that describes how specific performance goals for the  current and subsequent fiscal year contribute to an agency’s strategic  goals and objectives. The performance plan should contain, among other  things, a description of how the performance goals are to be achieved,  including clearly defined milestones, and establish a balanced set of  performance indicators, or measures, that will help demonstrate progress  toward achieving the performance goals. We have found that these  requirements also can serve as leading practices for strategic planning at  lower levels within federal agencies, such as planning for individual  divisions, programs or initiatives. Among these leading practices is  developing and using performance measures, which allow an agency to  track the progress it is making toward its mission and goals, provide  managers information on which to base their organizational and  management decisions, and create powerful incentives to influence  organizational and individual behavior. Further, federal leading practices  outlined by HowTo.gov state that agencies should use performance  metrics to influence the design of and drive improvements for government  websites.

However, FLC did not translate the strategic goals identified in its 2012  annual report into performance goals and measures. In particular, FLC  did not develop any performance goals or measures related to its  strategic goals of outreach or networking, which are, as FLC officials  stated, the current goals to which its clearinghouse initiatives would   In addition, while FLC tracks usage of its clearinghouse contribute. initiatives, FLC has not established performance measures for these  initiatives that would help it assess the extent to which the initiatives are  achieving FLC’s goals of outreach and networking or how performance  might be improved. For example, FLC does not collect information on the  number of connections between lab staff and potential customers that  resulted from use of the Available Technologies tool. As a result, some  agency officials told us it was unclear how well the tool helps to facilitate  technology transfer from federal labs. Working collaboratively with agency  and lab members to develop and collect data on performance measures  for FLC’s clearinghouse initiatives could help FLC demonstrate the  usefulness of its tools to its members while also helping FLC measure  progress toward achieving its own goals. Some representatives of other  technology transfer organizations we interviewed said they measure the  results of their efforts using performance measures, such as the number  of companies formed, the number of joint investment agreements  between labs and private companies, and the number of successful  technology transfer partnerships they help to facilitate. Without some type  of performance measures, FLC will be unable to assess whether its  initiatives are meeting FLC’s goals of outreach and networking. Further,  given that FLC has not developed such measures, its annual report to  Congress does not contain information on the progress toward the  achievement of FLC’s goals of outreach and networking, limiting the  information that the administration and Congress receive on the  effectiveness of FLC’s initiatives.

section_title: Conclusions
paragraphs: Recognized as one of 15 cross-agency priority goals in the President’s  2015 budget, technology transfer has become an increasingly important  strategy to support economic growth, create jobs, and increase the global  competitiveness of U.S. industries. As FLC’s role in promoting technology  transfer has expanded, particularly following the President’s 2011  memorandum, the consortium has launched important initiatives to make  information about federal technology transfer opportunities more available  to potential customers. FLC is uniquely positioned to contribute to the  President’s goal of supporting innovation, economic growth, and job  creation as the only interagency body federally mandated to facilitate  technology transfer.

However, FLC’s efforts to communicate with potential customers often  are not consistent with federal standards for internal control on  communicating with and obtaining information from stakeholders and do  not incorporate leading practices that call for agencies to consider  customer needs when designing and implementing government services.  Without better communicating with potential customers during the design  and implementation of clearinghouse initiatives, FLC may not be able to  effectively assess customer needs, conduct user tests of its web-based  initiatives, and collect customer feedback. As a result, FLC does not have  assurance that its initiatives will meet customer needs, and  representatives of potential customer groups indicated that there was a  significant mismatch between the information that would allow them to  identify opportunities and what FLC’s initiatives currently provide.

While commercialization of technological innovations is a private sector  role rather than a responsibility of the federal government,  representatives of potential customer groups stressed that federal labs  have extensive value and expertise that they find difficult to access.  Officials from federal agencies told us that any improvements to FLC’s  initiatives based on customer feedback would require a significant  collaborative effort on the part of FLC and its agency and lab members,  given the relatively small size of FLC’s annual budget and available staff.  Also, because the information FLC provides through its initiatives  depends on what is available from its agency and lab members, agency  officials said that FLC would need to collaborate with them to help  determine what additional or different information they could provide to  help meet customers’ needs based on any feedback received. By working  collaboratively with agency and lab members to collect feedback on its  web-based initiatives’ usefulness, FLC can take full advantage of its  members’ collective knowledge and expertise to better communicate with  potential customers consistent with federal internal control standards and  leading practices to determine those customers’ needs, enhancing the  clearinghouse initiatives in achieving their technology transfer goals.

In addition, FLC has not established performance goals and measures  that would help it assess the extent to which the initiatives are meeting  FLC’s goals of outreach and networking or how performance might be  improved. Working collaboratively with agency and lab members to  develop and collect data on performance measures for FLC’s  clearinghouse initiatives could help FLC demonstrate the usefulness of its  tools to its members while also helping FLC measure progress towards  achieving its own goals. Representatives of other technology transfer  organizations we interviewed said they measure the results of their  clearinghouse efforts using performance measures. Without performance  measures, FLC cannot fully demonstrate in its annual report to Congress  its progress toward the achievement of its goals of outreach and  networking, limiting the information that the administration and Congress  receive on the effectiveness of FLC’s initiatives.

section_title: Recommendations for Executive Action
paragraphs: To more effectively fulfill its expanded role in providing a clearinghouse of  information on available federal technology transfer opportunities, we  recommend that the Chair of FLC, in coordination with the other members  of the Executive Board, take the following four actions:

Work collaboratively with agency and lab members to take steps to  better communicate with potential customers during the design and  implementation of FLC’s clearinghouse initiatives, including  conducting customer needs assessments, conducting customer  testing of current and future web-based initiatives, and collecting  customer feedback on all FLC initiatives to make the initiatives more  useful.

Once feedback is collected from potential customers, work  collaboratively with agency and lab members to use this feedback to  improve FLC’s initiatives to make them more useful to potential  customers, including asking FLC members for additional or different  information, as appropriate.

Work collaboratively with agency and lab members to develop  performance goals and measures for FLC’s clearinghouse initiatives  and use the results to evaluate progress toward meeting FLC’s goals  on outreach and networking.

Report on FLC’s progress in these efforts in its annual report to the  President and Congress.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to FLC, DOD, DOE, NIH, NASA, and  OSTP for review and comment. In written comments, reproduced in  appendix II, FLC concurred with our findings and recommendations, and  noted that it will take steps to incorporate our recommendations as it  plans, implements, and monitors its efforts to help improve how it  performs outreach beyond its members to the potential end users of its  products. FLC also stated that it is examining how it is organized and is  considering making changes to better leverage government and industry  best practices for the use of digital systems.

In its written comments, FLC also stated that although it generally agrees  with the report’s findings, there are some areas that it believes require  some additional clarification. First, FLC stated that the report implies that  FLC recently assumed a clearinghouse function when it has been  performing this function since its establishment and that it more recently  began to provide digital-based services. We agree that the recent change  in FLC’s clearinghouse efforts has been to provide web-based tools, and  we acknowledged that FLC has offered one of its primary clearinghouse  initiatives—the Technology Locator service—since 1987. In response to  FLC’s comment, we clarified portions of the report to indicate that FLC  has, over time, served as a clearinghouse, but has increased its efforts in  this regard in recent years. Second, FLC noted that it developed the  Available Technologies tool as an initial Phase I version, not as a final  product and that it plans to continue to develop it as additional structured  data from laboratories become available. We revised the report to  acknowledge that FLC plans to further develop this tool. Moreover, our  recommendation that FLC increase communication with potential  customers to obtain feedback on how the tool might be improved will help  ensure the effectiveness of FLC’s efforts. Third, FLC stated that we  should include additional organizations in appendix I. We did not  incorporate FLC’s comments in appendix I because the suggested  additions would have been beyond the scope of the appendix, which was  to provide information on other technology transfer organizations with  technology search tools that we examined during the course of our  review.

DOD, DOE, NASA, and OSTP indicated by e-mail that they had no  technical or written comments. NIH provided technical comments by e- mail but did not provide formal written comments. We incorporated these  technical comments, as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Chair of the Federal Laboratory Consortium, the  Secretary of Defense, the Secretary of Energy, the Secretary of Health  and Human Services, the Administrator of NASA, the Director of OSTP,  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov/.

If you or your staffs have any questions about this report, please contact  me at (202) 512-3841 or neumannj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.

section_title: Appendix I: Selected Technology Transfer Organizations with Technology Search Tools
paragraphs: We identified a number of other organizations with technology search  tools that had features consistent with some of the needs identified by the  representatives of potential customer groups we interviewed. These are  some features that representatives from other technology transfer  organizations said they included in their technology transfer search tools  based on their assessment of customer needs. Common themes among  these tools include a collaborative interface and an ability to contact  researchers affiliated with the technologies. We did not evaluate the  content of any of these search tools, nor are we endorsing any of these  tools or organizations. For more information on these organizations, see  the links below.

AutoHarvest Foundation – http://www.autoharvest.org/

According to its website, AutoHarvest is an online meeting place that  allows users of all types to showcase capabilities, technologies, and  needs, then privately connect with other users to explore technology  and business development opportunities of mutual interest.

Entrepreneurs, industry representatives, and researchers from  universities, federal labs, and private industry can interact with one  another through virtual “collaboration rooms.” These virtual rooms  allow two-way communication between registered users of the tool.

In addition to posting technologies available for licensing, federal  laboratories and universities can also post facility capabilities to  encourage cooperative research and development agreements and  collaborative partnerships.

Collective IP – https://www.collectiveip.com/

According to its website, the Collective IP tool provides technology  transfer staff with a technology marketing platform, while  simultaneously providing a search tool for other users who focus on  identifying opportunities from companies and technology transfer  organizations.

Users can create and receive customized updates about  organizations and researchers from universities, federal labs, or  private industry working on specific technologies or in particular  research areas.

Users can review data on technologies and technology transfer  opportunities in a standardized format from across a variety of  sources, including the U.S. Patent and Trademark Office, federal  grant agencies, and academic literature.  iBridge – http://www.ibridgenetwork.org/ (part of the Innovation  Accelerator Foundation)

According to its website, the iBridge Network serves as an online  community for sharing ideas, research, and knowledge about early  stage technologies and inventions.

Users can search for groups of related patents across multiple  sources, mostly university labs.

Users can also post technologies they wish to license or receive  personalized e-mail updates about technologies that interest them.

Technology Ventures Corporation – http://www.techwhiteboard.com/

According to the Technology Ventures Corporation’s website, its  Technology Whiteboard is a communication hub centered on  Department of Energy (DOE) technology and intellectual property  commercialization.

Users can collaborate and share ideas with entrepreneurs, university  and federal researchers, and investors through social networking.

Users can search for technologies from a number of DOE laboratories  by category, laboratory, or keyword.

section_title: Appendix II: Comments from the Federal Laboratory Consortium for Technology Transfer

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Acknowledgments
paragraphs: In addition to the individual named above, Chris Murray (Assistant  Director), Antoinette Capaccio, John Delicath, Cindy Gilbert, Richard  Hung, Rich Johnson, Mae Liles, Les Mahagan, Rob Marek, David  Powner, Kiki Theodoropoulos, and Michelle Wong made important  contributions to this report.


Question-summary annotations: questions: question: What are the consequences of failing to obtain feedback when the Federal Laboratory Consortium for Technology Transfer (FLC)  try communicate with potential customers?
answer: This resulted in missed opportunities to better meet potential customer needs.
child_questions: question: What is an example of FLC failing to obtain feedback from potential customers?
answer: For example, in 2012, when developing a web-based search tool to help potential customers identify relevant federal technology transfer opportunities across federal laboratories (labs), FLC discussed how to implement the tool with its federal member labs and agencies.

question: How has the development of a web-based search tool in 2012 failed to satisfy the needs of potential customers?
answer: However, FLC did not assess the information needs of potential customers to ensure the tool would provide relevant information in a format that customers consider useful, as called for by leading practices and federal internal control standards on communicating with and obtaining information from stakeholders. FLC officials said they conducted testing to ensure the new website functioned as intended before launching it, but did not involve potential customers in these tests. Moreover, after developing the tool, FLC did not communicate with potential customers to collect feedback from them consistent with leading practices regarding the extent to which the tool met their needs or how it might be improved before implementing it.
child_questions: question: What are the reactions of the potential customers of FLC?
answer: Potential customers of FLC's initiatives expressed concerns about the extent to which FLC's recent web-based search tool would meet their needs, specifically noting that the tool:

questions: question: What kind of data did FCL collect?
answer: For example, FLC collects data on the general use of its clearinghouse initiatives, such as the number of technology transfer inquiries it receives, the number of unique views of its web pages, and the average time spent on a web page.
child_questions: question: How are the collected data underused?
answer: FLC collects data on the use of its clearinghouse initiatives but has not developed and used performance goals and measures consistent with federal agency leading practices.

question: Why did FLC failed to determine whether its initiatives are performing as desired or how to improve their performance?
answer: However, FLC has not developed performance goals or measures related to the key strategic goals to which its clearinghouse initiatives contribute. Without performance measures, FLC is unable to determine whether its initiatives are having their desired effect or how their performance might be improved.
child_questions: question: What other consequence does the lack of performance measures have?
answer: FLC also cannot fully demonstrate in its annual report to Congress its progress toward the achievement of its relevant strategic goals, limiting the information that the administration and Congress receive on the effectiveness of FLC's initiatives.

questions: question: How does the federal government support federal agency labs?
answer: The federal government spends about one-third of its annual $145 billion research and development budget at hundreds of federal agency labs.

question: In what way is the technology generated by this research useful?
answer: Technology generated by this research may have application beyond agencies' immediate goals if commercialized by the private sector. For example, federal research has contributed to innovative products, including antibiotics and the Internet.
child_questions: question: How are the technology transferred to private sector?
answer: FLC—a nationwide consortium of federal labs—helps labs transfer technology to the private sector.

question: What has FCL created in recent years?
answer: In recent years, FLC created new initiatives to provide a clearinghouse—a central point for collecting and disseminating information—for technology transfer opportunities.

questions: question: How did FLC's efforts to provide information on technology transfer opportunities evaluated?
answer: GAO was asked to review FLC's efforts to provide information on technology transfer opportunities.
child_questions: question: What does the report demonstrate?
answer: This report assesses (1) the extent to which FLC has communicated with potential customers when designing and implementing its clearinghouse initiatives, and (2) how FLC measured the results of those initiatives.

question: What information  did GAO used to review FLC's efforts to provide information on technology transfer opportunities?
answer: GAO reviewed relevant laws and FLC guidance, and interviewed a nonprobability sample of officials from four federal agencies with the highest research budgets, and a spectrum of eight customer groups, among others.