International Space Station: Measurable Performance Targets and Documentation Needed to Better Assess Management of National Laboratory
Released: 2015-05-27T00:00:00

Letter

Background

The ISS supports research projects with state of the art facilities for Earth and space science, biology, human physiology, physical science, and materials research, and provides a platform to demonstrate new space- related technologies.

The facilities include modular multipurpose payload racks and external platforms to store and support experiments, refrigerators and freezers for biological and life science samples, research lockers or incubators, and a combustion chamber to observe combustion patterns in microgravity, among other research equipment. The ISS currently has three crew members in the U.S. operating segment who, according to NASA officials, devote a total of approximately 35 hours per week to conduct research. The remaining crew time is used for operations and maintenance of the ISS, training, exercise, and sleep. NASA plans to increase the number of astronauts in the U.S. operating segment of the ISS from three to four once a U.S. capability to transport crew to and from the ISS is available. Cargo transportation to the ISS is done through a commercial resupply services contract that was signed with Orbital Sciences Corporation (Orbital) and Space Exploration Technologies Corporation (SpaceX) in 2008. SpaceX currently has a capsule that can also return significant amounts of cargo to Earth and is the only vehicle currently servicing the ISS that has this capability. Orbital and SpaceX are scheduled to provide 8 and 15 resupply flights, respectively, through December 2017. As of January 2015, SpaceX has launched five successful resupply missions and Orbital has launched two successful resupply missions. Orbital resupply flights to the ISS were deferred pending a review of a mishap that occurred during a resupply launch in October 2014, which resulted in the loss of that mission. According to NASA officials, a “return to flight” plan was submitted by Orbital and accepted by the ISS program in January 2015.

Legislation on Management of the ISS National Laboratory

Since 2005, Congress has directed several changes regarding the management and utilization of the ISS. The NASA Authorization Act of 2005 designated the U.S. segment of the ISS as a National Laboratory.The 2005 act directed the NASA Administrator to seek to increase ISS utilization by other federal entities and the private sector through partnerships, cost-sharing agreements, and other arrangements that would supplement NASA funding of the ISS. It also allowed the Administrator to enter into a contract with a nongovernment entity to operate the ISS National Laboratory.

The NASA Authorization Act of 2008 further directed NASA to establish the ISS National Laboratory Advisory Committee, which was to be composed of individuals representing organizations that had formal agreements with NASA to utilize the U.S. portion of the ISS. act stated that the committee shall monitor, assess, and make recommendations regarding effective utilization of the ISS as a national laboratory and platform for research, and submit a report containing these assessments and recommendations at least annually to the NASA Administrator.

National Aeronautics and Space Administration Authorization Act of 2008, Pub. L. No. 110-422, § 602. designate a NASA liaison, with whom the selected not-for-profit entity would cooperate and consult with in carrying out its responsibilities under the agreement. An individual in the Space Life and Physical Sciences Research and Applications Division of the Human Exploration and Operations Mission Directorate is currently serving as the NASA liaison. The 2010 act outlined seven management and research and development activities that NASA was required to provide funding for the not-for-profit entity to initiate. Those activities stated briefly, are to:

Plan and coordinate ISS National Laboratory research activities;

Develop and implement guidelines, selection criteria, and flight support requirements for non-NASA utilization of the ISS research capabilities and available facilities; Interact with the ISS National Laboratory Advisory Committee and review recommendations provided by that committee;

Coordinate transportation requirements in support of the ISS research

Cooperate with NASA, other departments and agencies of the U.S. government, and commercial entities to sustain ground support facilities for the ISS;

Develop and implement scientific outreach and education activities designed to ensure effective utilization of ISS research capabilities; and

Address other matters relating to the utilization of the ISS National Laboratory for research and development as the Administrator may consider appropriate.

The 2010 act also requires the ISS National Laboratory-managed experiments to be guaranteed access to and use of at least 50 percent of the U.S. research capacity allocation including power, facilities to keep experiments cold, and requisite crew time onboard the ISS through September 30, 2020. The Administrator can allocate additional capacity to the ISS national laboratory if this capacity is in excess of NASA research requirements. If any NASA research plan requires more than the at least 50 percent of the U.S. research capacity allocation of ISS resources, the plan should be submitted for consideration of proposed research to be conducted within the ISS National Laboratory capacity of ISS resources. The person designated as the NASA liaison to the not-for- profit entity has the authority to provide those resources beyond its 50 percent allocation on an exception basis if a proposed experiment is considered essential for purposes of preparing for exploration beyond low-Earth orbit, based on a joint agreement between the NASA liaison and the not-for-profit entity.

CASIS and NASA Responsibilities Outlined in Cooperative Agreement

In August 2011, after a competitive process, NASA signed a cooperative agreement with CASIS, a not-for-profit entity, to manage the activities of the ISS National Laboratory through September 30, 2020. Cooperative agreements differ from contracts. Generally, cooperative agreements are used when the principal purpose of a transaction is to stimulate or support research and development for a public purpose, and substantial involvement is expected between the executive agency and the award recipient when carrying out the activity identified in the agreement. In contrast, contracts are used when the principal purpose is acquisition of property or services for the direct benefit or use of the federal government.

CASIS is bound by the responsibilities outlined in the cooperative agreement, which tasks CASIS with maximizing the value of the ISS National Laboratory by stimulating interest and use of the ISS for scientific research by directly soliciting potential users and fostering a market to attract others. CASIS is also charged with maximizing the use of the ISS for advancing science, technology, engineering, and mathematics (STEM) education. Pursuant to the cooperative agreement, NASA will provide CASIS $15 million annually through 2020, of which it will seek to award at least $3 million for research grants. CASIS officials said that the remainder of NASA funding is used for infrastructure and direct costs such as labor and travel-related expenses. According to the cooperative agreement, CASIS will solicit non-NASA funding for research by targeting various sources—such as government grants, foundation funding, charitable contributions, private equity, venture financing, and private investors—and facilitate matching of projects that meet the research objectives with those qualified funding sources. Additionally, the cooperative agreement requires the development of an annual program plan, which includes a detailed plan of CASIS’s proposed activities for the following year, which CASIS must meet using its “best efforts,” and annual and quarterly performance metrics.

The cooperative agreement outlines responsibilities for NASA such as providing resources and accommodations to CASIS to meet ISS National Laboratory requirements and performing the payload operations integration to ensure safe and effective flight readiness and vehicle integration. The Cooperative Agreement Technical Officer, a NASA employee within the ISS Program Office at Johnson Space Center, is charged with oversight of the cooperative agreement. The Cooperative Agreement Technical Officer is to coordinate the approval of the Annual Program Plan and track performance to the plan using the metrics reflected in CASIS’s quarterly reports.

CASIS Implemented Most of the Required Activities, but Is Not Able to Interact with Advisory Committee

CASIS has taken steps to carry out its responsibilities to manage and promote research activities on the ISS National Laboratory as outlined in its cooperative agreement. For example, CASIS identified key research areas and released seven requests for proposals to solicit interest for research projects. Our survey of a sample of researchers who had submitted proposals to CASIS revealed generally positive comments about CASIS’s management effort. For example, many respondents indicated that CASIS’s processes were clear and that it evaluated their proposals fairly. CASIS, however, has not been able to coordinate with the ISS National Laboratory Advisory Committee (INLAC), as required, because NASA has yet to staff the committee.

CASIS Took Steps to Implement Management Activities

CASIS has taken steps to fulfill its responsibilities contained in its cooperative agreement with NASA, and has initiated the activities required by the NASA Authorization Act of 2010.the activities contained in the 2010 act as well as the corresponding Table 1 summarizes responsibilities for CASIS and NASA outlined in the cooperative agreement.

To determine its research and technology development objectives in accordance with the cooperative agreement, CASIS identified and prioritized the most promising research areas—which CASIS refers to as pathways—with guidance from the Science and Technology Advisory Panel, a CASIS committee comprised of both academic and commercial experts. These pathways are identified by compiling a list of research categories and determining financial feasibility. According to CASIS and NASA officials, research pathways are generated from various sources such as the Decadal surveys—studies conducted once every decade by the National Research Council that prioritize the research focus for the next 10 years in various scientific disciplines—and past NASA studies. To date, CASIS identified protein crystal growth, stem cell research, materials science, enabling technology to support science in space, Earth imaging, and remote sensing as key research pathways and developed a request for proposals (RFP) for each of these research pathways. CASIS released seven RFPs since it was established with the first occurring in June 2012, about 10 months after it was established. CASIS also accepts unsolicited proposals from researchers and other sources such as As of January 2015, CASIS partnership accelerators and competitions.had received 206 proposals from all sources and awarded approximately $20 million in research grants to 77 projects, and paid almost $13 million to the awarded grants. Table 2 shows information related to the types of proposals CASIS has received and the number of grants awarded.

CASIS-sponsored research investigations awarded through its first RFP in 2012—involving protein crystal growth and microgravity—flew to the ISS National Laboratory in April 2014 and were returned to Earth in October 2014. These research investigations are currently in post flight analysis. As of December 2014, there were 11 CASIS-sponsored research investigations being conducted aboard the ISS National Laboratory.

According to NASA and CASIS officials, as CASIS increases the number of experiments for the ISS National Laboratory, the demand for crew time and certain research facilities aboard the ISS is expected to increase and they project the ISS National Laboratory will be challenged with meeting that demand. NASA officials explained that while the demand for crew time is currently manageable, it remains allocated at or near 100 percent, as the three crew members on the U.S. segment of the ISS utilize most of the 35 hours scheduled per week to conduct research. Crew time is expected to double on the ISS National Laboratory once the crew increases from three to four astronauts in fiscal year 2018 because, according to NASA officials, the additional crew member would be able to devote most of his or her time to research. NASA officials stated they are also working with CASIS to build automation into research experiments to reduce the monitoring time by crew members. Both CASIS and NASA expect increased demand for facility resources such as the Animal Enclosure Module used for rodent research and the remote sensing cameras used for Earth observation. Sharing of the ISS National Laboratory facilities requires considerable communication and agreement. NASA and CASIS officials said both organizations have on- going discussions about how to share resources, coordinate research and ensure all users are represented when meeting the demand for crew time and ISS National Laboratory facilities and hardware. NASA officials explained that they reprioritize as necessary to ensure resources are not overstressed.

Develop and Implement Guidelines, Selection Criteria, and Flight Support Requirements

To initiate the development of guidelines and selection criteria, CASIS implemented procedures for prioritizing research, guidelines for proposal development, and evaluation and selection of research proposals, in accordance with the cooperative agreement.

Procedures for prioritizing research: CASIS has implemented a multi-layer review process to identify and develop the overall research portfolio and prioritize future research pathways. See figure 2 for the process CASIS follows to prioritize research pathways.

Guidelines on proposal development and flight support requirements: CASIS established guidelines that are incorporated in the applicable RFP for researchers to follow as they develop their proposals. The RFPs include specific criteria for proposal that CASIS uses as a basis for initial acceptance or denial of proposal submissions. For example, one RFP issued in 2014 contained minimum eligibility criteria such as the research being flight ready within 12 months of award and the research having secured funding, and included provisions that excluded the use of new sensors or instruments for remote sensing, and required that selected proposals be completed by 6 months post-flight. Each RFP also had unique criteria which can be dependent upon the research pathway and the facilities available on the ISS National Laboratory in the proposed time line. CASIS has separate guidance for unsolicited proposal submissions.

CASIS has documented the specific activities for meeting flight requirements, which includes the role of implementation partners and NASA in meeting these requirements. Implementation partners are subcontractors to CASIS and specialize in aerospace technologies and services. They have an integral role in providing hardware, flight integration services, and ground services to support CASIS- sponsored research. NASA performs the activities necessary to incorporate the research on a flight vehicle, such as providing the resources and accommodations to meet ISS National Laboratory requirements, and managing launch operations through payload return to Earth.

Evaluation and selection of research proposals: CASIS implemented a policy that documents the submission and general review process for solicited and unsolicited proposals as well as proposals that it evaluates as part of agreements with outside organizations such as partnerships or subcontracts. The process begins with a two-step initial submission review for solicited proposals and a preliminary review for unsolicited proposals, then a five-step evaluation process. Figure 3 details the CASIS proposal evaluation process.

We surveyed a random sample of 14 researchers who submitted proposals to CASIS from 2012 through 2014 to obtain their perspectives on CASIS’s performance in this and other areas. Although the results of this survey are non-generalizable because of our small sample size, overall the respondents were generally positive about their interaction with CASIS. For example, 11 of the 14 respondents indicated that CASIS’s evaluation criteria were clearly articulated and 12 of 14 respondents believed their proposals were evaluated fairly. Of the 14 respondents, 13 said they were likely to submit future proposals to CASIS. In addition, all 14 respondents indicated that they were notified in a timely manner of the disposition of their proposal. CASIS declined proposals for 8 respondents. Of these 8, 7 said that they were provided feedback concerning why their proposal was declined. Several respondents, however, said that they were provided only a short bulleted response that fell short of addressing the scientific merit of the proposal. One respondent said they received a letter summarizing reviewers’ comments that had several good points and was fair, but it was not detailed as it contained less information than what other grantors provide. According to CASIS guidelines, researchers whose projects are not selected for award are provided feedback and, are invited to revise and resubmit their projects as an unsolicited proposal. Of those that we surveyed, only 1 of 8 respondents who had a proposal declined had resubmitted the proposal, while another respondent said that it was not made clear that proposals could be resubmitted.

Coordination of Transportation Requirements

Under the cooperative agreement, NASA is required to provide the ISS National Laboratory research facilities and resources and coordinate with CASIS when preparing CASIS-sponsored research for launch. CASIS has an integral role in the payload development and integration process during three distinct phases—pre-flight, operations, and post-flight. During the pre-flight phase, the CASIS operations team works with the researcher and implementation partners to understand project objectives and requirements such as power, crew and hardware compatibility needs, flight integration time frames, and design and integration support. CASIS submits the science objectives, requirements, and a development schedule to NASA. The NASA ISS National Laboratory Office also assigns staff to each CASIS-sponsored researcher to help coordinate and navigate the payload development and integration process and ensure that flight planning remains on track. During the operations and in-flight phase, CASIS provides operational support by collaborating with the implementation partner or the researcher to oversee NASA’s integration of the research project or hardware into the flight vehicle. The post-flight phase involves the return of payload samples or hardware from the flight vehicle to the researcher to begin post-processing activities, which CASIS monitors.

The ability to secure transportation for selected research investigations to the ISS facility is outside of CASIS’s control and has presented challenges. NASA provides launch services to the ISS National Laboratory through its commercial resupply services contracts and CASIS receives cargo allocations for its sponsored research. Launch failures and delays, however, have resulted in cost increases. For example, the recent rocket launch failure to the ISS in October 2014 resulted in the loss of several CASIS-sponsored research investigations at a total cost of almost $175,000 which includes hardware and materials, labor consulting and grants. In addition, launch delays for another cargo resupply mission resulted in over $300,000 in cost increases for several researchers for additional materials and samples. CASIS officials explained that the majority of cost increases are related to biological research, which represents approximately 50 percent of the CASIS-sponsored research. These biological payloads have a limited viability or very specific requirements associated with the timing of the payload flight and often require consumables such as gas, nutrients, and water that must be replenished when a launch is delayed. Absorbing the increased cost has been a challenge for CASIS, but it is addressing the increased costs of delays by asking researchers that have biological payloads to identify the impact and associated costs for launch delays in their budgets so it can plan for budget reserves, if necessary.

Cooperative Efforts and Partnerships

The cooperative agreement requires CASIS to manage planning and coordination of research activities for both ground and on-orbit execution. According to CASIS officials, CASIS is addressing this requirement by leveraging the resources of companies that provide hardware, technical expertise and ground support. Eleven implementation partners have received over $5.4 million in funding from CASIS or its sponsored researchers since the establishment of CASIS through September 2014 to provide hardware, flight integration, and ground services for 58 research investigations. CASIS officials reasoned that by leveraging existing companies that can provide specialized hardware and integration capabilities on an as required basis, CASIS can effectively manage the ISS National Laboratory without having to maintain all the requisite skills or capabilities within its organization. CASIS-sponsored researchers are encouraged to select an implementation partner during the proposal submission process from a list of preferred partners. CASIS officials said that they assembled this list of implementation partners beginning with companies that had relationships with NASA for ISS-related operations and expanded the list through its own business development operations. These partners can provide hardware, and technical services, and consultation to researchers that address the project’s science requirements and research needs aboard the ISS National Laboratory. Although CASIS provides a list of implementation partners, the researchers are responsible for entering into formal business arrangements with these partners and including the costs of the implementation partner support in their proposed budget. CASIS officials noted the cost can vary based on the amount of involvement required by the implementation partners and can range from $50,000 to $300,000 per flight.

Scientific Outreach and Educational Initiatives

In accordance with the cooperative agreement, CASIS is building a geographic network to facilitate outreach initiatives and cultivate new partnerships and has implemented educational initiatives that provide opportunities for educators and students to learn about and have access to the ISS National Laboratory. Specifically,

Network outreach: CASIS has organized its outreach to scientific and academic communities in seven geographical areas. These areas are supported by more than 30 CASIS employees and consultants and each area has a research emphasis. See figure 4 for the locations of CASIS’s networks and the research emphasis for each.

The outreach efforts conducted through CASIS’s networks are primarily relationship-based and focused on engaging financial support, forging long term partnerships, and ultimately generating potential research and technology development projects for flight on the ISS National Laboratory. According to CASIS officials, academic institutions, research-specific organizations, philanthropic entities, and industry partners that CASIS identified through this network can benefit from the CASIS-sponsored research and technology development aboard the ISS National Laboratory. For example, Boston was identified as one of the geographic areas because it has over 100 universities and over 300 biotech companies that can support the commercialization of life sciences research and CASIS’s mission.

CASIS is working to expand its network. CASIS has developed 45 new partnerships to date and is leveraging a variety of new partnership opportunities. For example, in 2014, CASIS initiated two strategic campaigns, Good Earth—an international collaboration seeking to maximize ISS Earth observation capabilities—and Good Health—an effort to capitalize on the unique benefits of the microgravity environment so interventions can be developed to preserve health on Earth. CASIS officials expect both campaigns to bring together large scale collaborations to stimulate ISS utilization over the coming years. CASIS also supported the Rice Business Competition by providing a $25,000 grant during 2014 to a startup company that showed the most promise for developing a technology or business that would benefit from access to the ISS National Laboratory. This partnership also gives CASIS access to many forum events and panels.

According to CASIS officials, it has been challenging to raise additional funding from external sources to supplement the amount of funding provided by NASA to support and sustain its operations because CASIS is a new non-profit entity. Although CASIS’s business development team is actively identifying partnerships and funding opportunities with commercial and non-profit granting organizations, CASIS officials said that it takes time to identify, develop, and mature these partnerships. CASIS and NASA officials said that the value of doing research aboard the ISS National Laboratory has to be further demonstrated so commercial industries can be convinced it is worth the high investment. Both NASA and CASIS officials said that demonstrating the value of research on the ISS as a substitute for ground-based research is a tremendous and important effort that is necessary to open a marketplace for space research. NASA officials stated that doing research aboard the ISS National Laboratory can take upwards of 2 to 3 years to plan and execute, time lines that are generally not acceptable to commercial companies that desire a more rapid return on their investments.

Ten of the 14 respondents to our survey reported that CASIS was effective in reaching out to the research community. For example, several researchers were made aware of CASIS opportunities by attending presentations from CASIS staff at industry meetings or campus visits. Respondents also offered areas for improvement for CASIS to increase utilization of the ISS National Laboratory. For example, five respondents said that CASIS could increase its visibility by attending more conferences, using more print ads, and working more with NASA on joint RFPs.

Education: CASIS established its education strategic plan, which included building education programs that promote the ISS as a science, technology, engineering, and mathematics (STEM) learning platform; partnering with existing education entities such as schools, universities, and other educational foundations and associations; and reaching out to underrepresented and nontraditional demographics. CASIS also implemented various educational initiatives that it developed both internally and externally in conjunction with its partners. For example, in fiscal year 2014, CASIS supported 12 educational initiatives. CASIS sponsored the Space Station Academy, a 4-week online program designed to take participants on a simulated mission to the ISS as “virtual astronauts.” This pilot program involved 25 students and 25 educators. In addition, CASIS supports its educational efforts through education grant funding and partnerships. See appendix III for more information on additional CASIS educational initiatives.

NASA Has Not Staffed Advisory Committee with Which CASIS Is Required to Interact

The one required activity in the cooperative agreement that CASIS has been unable to address is its interaction with the ISS National Laboratory Advisory Committee (INLAC) because the committee has not been staffed by NASA. The NASA Authorization Act of 2008 required NASA to establish the INLAC under the Federal Advisory Committee Act. The INLAC was required to include membership from organizations that have formal agreements with NASA to utilize the U.S. portion of the ISS. As outlined in the 2008 act, this committee is required to exist for the lifespan of the ISS and is to function in an advisory capacity to the NASA Administrator by assessing and monitoring ISS National Laboratory resource utilization and reporting its assessments and recommendations at least annually. According to the cooperative agreement, CASIS will coordinate with the INLAC as established under section 602 of the NASA Authorization Act of 2008 and review recommendations provided by the INLAC. Although NASA formally established the committee in 2009, NASA has not fully implemented the 2008 act because the committee has yet to be staffed. NASA officials told us that with CASIS in place, the great majority of non-NASA ISS users do not have an agreement with NASA because they work with CASIS. They added that there are exceptions where NASA works with other agencies, but those are typically for exploration technology or defense-related projects. In addition, NASA officials indicate that the INLAC has not been staffed because they believe that the structure and function of the current CASIS Board of Directors has proven to be a better alternative to a NASA advisory committee since the CASIS board represents a broad experience base including military, medical research, strategic partnerships, and engineering, among others. Further, NASA officials said that the Research Subcommittee of the Human Exploration and Operations Committee to the NASA Advisory Council also provides research advisory oversight of the ISS National Laboratory. This subcommittee’s objectives, however, have a focus on human spaceflight and the membership of this subcommittee is to consist of individuals from the research committee with a broad awareness of human spaceflight related activities.

CASIS officials also believe that their board is performing some of the INLAC’s advisory duties, but acknowledge that the board does not meet the section 602 requirements under the 2008 act—to monitor and report annually to the NASA Administrator its assessments and recommendations of ISS National Laboratory utilization—nor does its membership meet the criteria specified in the act. Without a staffed INLAC, NASA currently lacks a single advisory committee that represents all users of the ISS National Laboratory and provides ongoing monitoring and assessments and recommendations of ISS National Laboratory resource utilization, as required by the charter. As a result, CASIS is not able to fulfill its responsibilities as outlined in the cooperative agreement and as established under section 602 of the NASA Authorization Act of 2008.

CASIS and NASA Have Taken Steps to Measure and Assess CASIS Performance, but Measurable Targets Needed

CASIS’s Metrics Consistent with Most Key Attributes, but Lack Quantifiable Goals

CASIS has established metrics, but not targets against which its performance can be measured by NASA. The metrics CASIS developed in collaboration with NASA for fiscal year 2015 meet most key attributes of successful performance measures. These metrics are based on CASIS responsibilities outlined in the cooperative agreement and are related to CASIS strategic goals and objectives. Metrics are included in an Annual Program Plan, which CASIS prepares with input from NASA.

We have previously reported that successful performance measures as a whole should have four general characteristics: demonstrate results, be limited to a vital few, cover multiple priorities, and provide useful information for decision making. We cited specific attributes as key to successful performance measures, such as linkage, clarity, measurable targets, objectivity, and balance. The four characteristics are overarching, thus they do not necessarily directly link to the attributes. Furthermore, the attributes may not be equal, and a noted weakness does not mean that a measure is not useful. Weaknesses identified should be considered areas for further refinement. Table 3 defines the key attributes of successful performance measures.

We assessed CASIS’s fiscal year 2015 metrics, and found that the metrics met almost all of these key attributes. The results of our assessment are shown in table 4. We also assessed the metrics CASIS had developed for fiscal year 2014, and similarly found that the metrics met most of the key attributes. The results of our assessment of CASIS’s fiscal year 2014 metrics for key attributes of successful performance measures can be found in appendix IV.

Our analyses indicated that CASIS did not establish measurable targets or goals for either fiscal year 2014 or 2015 metrics, which limits its ability to use these metrics to assess performance. We have previously reported that performance metrics should have quantifiable, numerical targets or other measurable values, which help assess whether overall goals and objectives were achieved.it is unclear how NASA objectively assesses CASIS’s performance.

Without defined measurable targets or goals, CASIS officials noted that operating as a new entity with no history made it difficult to establish performance targets, but this is beginning to change. CASIS officials initially told us in July 2014 that establishing targets would be arbitrary because CASIS processes and metrics are still evolving. Subsequently, in January 2015, they indicated that since CASIS now has some operating history, they will be able to do so. The Chairman of the CASIS Board of Directors told us that measurable targets should be developed and that this is a priority for the Board. However, CASIS has not established a date by which measurable targets will be developed. Further, CASIS officials indicated that not all metrics will have measurable targets initially because some metrics are subjective, such as those that attempt to measure the quality of research or a new technology generated by CASIS-sponsored research. The Chairman said that the CASIS Board of Directors is also working to develop targets for subjective measures, and they hope to have them in place in the next several years.

Although the ability to objectively measure performance is limited without measurable targets, CASIS and NASA officials generally agreed about how long-term success for CASIS will be defined. According to CASIS officials, success would ultimately be defined by demonstrating that the research and technology development performed aboard ISS National Laboratory benefits Earth and that commercial markets can be sustained in low-Earth orbit. NASA officials similarly said that developing commercial markets in space and bringing products back to Earth will determine success.

NASA Assesses CASIS Annually, but Performance Assessment Is Not Documented

NASA performs an annual assessment of CASIS’s performance consistent with its responsibilities in the cooperative agreement, but this assessment is not documented. The Cooperative Agreement Technical Officer (CATO) uses the metrics in CASIS’s quarterly and annual reports to monitor CASIS’s efforts. The cooperative agreement also requires CASIS to propose an adjustment to the metrics if performance is not going to be met. However, without performance targets, CASIS cannot determine whether the metrics need to be adjusted. Further, without these targets, NASA and CASIS cannot conduct assessments that are measurable or conclusive and, therefore, the assessments are subjective. According to the CATO, during the annual program review, he assesses CASIS metrics for trends, looking for improvements over time and questioning any perceived lack of progress. The CATO added that he discusses any issues identified during the annual review with CASIS officials, NASA management, and stakeholders. CASIS officials concurred, and told us this discussion with NASA highlights areas for further refinement. For example, as a result of such discussion, CASIS is now more proactively engaging NASA technical expertise on available flight hardware, and has broadened business development efforts aimed at attracting new commercial users of the ISS National Laboratory.

Both CASIS and NASA officials told us that NASA does not document its annual program review of CASIS performance. Federal standards for internal controls call for information to be recorded and communicated to management and others who need it to carry out their responsibilities. This type of documented information is important to support decision making and conduct assessments. CASIS officials have not asked for a formal summary of the results of NASA’s annual program review because CASIS receives informal feedback on quarterly reports provided to NASA. CASIS also maintains minutes of regularly scheduled meetings with NASA where any issues that need to be discussed between CASIS and NASA are addressed. While NASA does not document this annual assessment, NASA officials told us that they were generally satisfied with CASIS performance. CASIS officials, however, said that the results of the annual review should be reported in some sort of formal manner to make the information more actionable.

Because CASIS is allocated at least 50 percent of ISS research capacity, future success of the ISS as a research platform is partially dependent on the efforts CASIS has undertaken. However, without definitive and documented assessment factors, NASA will be challenged to take action in response to CASIS performance. For example, without documentation, NASA lacks support to terminate the cooperative agreement, if deemed necessary. Conversely, NASA also would have no record to justify extending the cooperative agreement to support a possible ISS life extension. The cooperative agreement will expire at the end of fiscal year 2020, but includes a provision for an extension.

Conclusions

The ISS offers the potential for scientific breakthroughs, a unique test bed for new technologies and applications, and a platform for increased commercial and academic research. Achieving greater utilization of the ISS and its unique capabilities, showing the benefit of commercial and academic research, and demonstrating success to generate increased interest from potential users could help NASA get a better return on its significant investment in the ISS. NASA currently lacks an advisory committee established under the Federal Advisory Committee Act that is composed of individuals representing organizations who have formal agreements with NASA to use the U.S. portion of the ISS. As a result, CASIS is not able to fulfill its responsibility as outlined in the cooperative agreement that requires it to coordinate with INLAC as established under the NASA Authorization Act of 2008 and review recommendations originated by the INLAC. A fully staffed and operational INLAC could provide information to senior NASA management on how to better utilize the constrained resources of the ISS—which could affect how CASIS attracts new users and fulfills its responsibility to increase utilization of the ISS National Laboratory.

In addition, clearly defined measurable targets are essential for CASIS to demonstrate results, allow NASA to objectively assess CASIS performance, and help stakeholders assess whether overall goals and objectives for the ISS National Laboratory are achieved. Finally, NASA’s annual performance assessment of CASIS is not documented and the results are provided to CASIS on an informal basis. Not documenting the results of the annual program assessment is a practice contrary to good internal controls, which call for information to be recorded and communicated to management and others who need it to carry out their responsibilities, to include taking appropriate corrective actions. Without a clear, well-documented assessment of CASIS performance, NASA management and stakeholders could also be missing information important for decision making, for example, deciding to extend the cooperative agreement with CASIS beyond the September 2020 expiration if the service life of the ISS is extended or terminate the agreement, if necessary.

Recommendations for Executive Action

We recommend that the NASA Administrator take the following three actions: In order for NASA to fully implement the NASA Authorization Act of 2008 and for CASIS to fulfill its responsibility as outlined in the cooperative agreement, direct the Associate Administrator for the Human Exploration and Operations Mission Directorate to fully staff the INLAC.

In order to set clear goals to allow NASA to objectively assess CASIS performance, require the ISS Program Manager work with CASIS to collectively develop and approve measurable targets for metrics for fiscal year 2016 and beyond.

In order to provide CASIS management actionable information to better fulfill its responsibilities and NASA management with additional information by which to make future decisions concerning the extension of the agreement with CASIS, require the ISS Program Manager to document the annual program assessment of CASIS performance.

Agency and Third- Party Comments and Our Evaluation

NASA and CASIS each provided written comments on a draft of this report, which are reprinted in appendix V and appendix VI, respectively. NASA and CASIS also provided technical comments, which have been incorporated into the report, as appropriate. NASA partially concurred and CASIS non-concurred with one of our recommendations and both NASA and CASIS concurred with the other two recommendations.

NASA partially concurred and CASIS non-concurred with our recommendation directing the Associate Administrator of the Human Exploration and Operations Mission Directorate within NASA to staff the INLAC. In response to this recommendation, both NASA and CASIS raised concerns that the current requirements for membership of the INLAC would create a conflict of interest. Specifically, NASA stated that the individuals who would make up the committee would likely have user agreements with CASIS and, in many cases, would be receiving funding from CASIS and NASA. Furthermore, because these entities would be competing for CASIS resource allocations, CASIS believes that they would not be sufficiently independent to perform the functions required of the committee. In response to these concerns, CASIS indicated the composition of membership as defined in the NASA Authorization Act of 2008 should be amended. NASA also responded that while meeting statutory obligations and obtaining knowledgeable input and recommendation to achieve optimal utilization of the ISS is important, it is the agency’s position that the CASIS Board of Directors serves the intent of the INLAC charter by providing recommendations regarding effective utilization of the ISS. As a result, NASA indicated that it plans to work with the Congress to adjust the INLAC requirement to address these concerns. We continue to believe our recommendation is valid. We do not see that staffing the INLAC as directed in the 2008 act would necessarily result in a conflict of interest and that the entities would be competing for CASIS resource allocations. The act required an advisory committee that represents all users of ISS National Laboratory and that provides ongoing monitoring and assessment and makes recommendations. According to the cooperative agreement between CASIS and NASA, CASIS is directed to coordinate with the INLAC and review the committee’s recommendations. The INLAC, however, functions only in an advisory capacity; therefore we do not see how a conflict of interest would be created by the membership of the INLAC. Furthermore, according to the NASA Authorization Act of 2010, CASIS shall be guaranteed access to not less than 50 percent of the United States research capacity allocation. Because CASIS has to agree with NASA for an allocation of resources at a level below 50 percent, we do not see how the composition of the INLAC would create a competition for resource allocation with CASIS. In addition, it was not clear to us in our review that the existing mechanisms in place accomplish these requirements. If NASA were to seek relief or changes to this requirement, it should clearly outline how these requirements can be met through existing bodies and processes.

NASA and CASIS concurred with our recommendation directing the ISS Program Manager to work with CASIS to collectively develop and approve measurable targets for metrics in fiscal year 2016 and beyond. In response to this recommendation, NASA stated that fiscal year 2016 is a reasonable time to establish measurable targets with CASIS because the non-profit will be entering its fourth full year of operations. Similarly, CASIS responded that it is now in a position to develop targets for key metrics and plans to formalize the process in fiscal year 2016. NASA indicated that these targets should be established by December 31, 2015. Once complete, this action should address our recommendation to develop and approve measurable targets for CASIS’s metrics.

NASA and CASIS also concurred with our recommendation directing the ISS Program Manager to document the annual program assessment of CASIS performance. In response to this recommendation, NASA said that it would begin documenting the agency’s annual program assessment in response to CASIS’s 2015 annual report. Once complete, this action should address our recommendation to document NASA’s annual assessment of CASIS’s performance.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. We will send copies of the report to NASA’s Administrator and interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to assess the extent to which (1) the Center for the Advancement of Science in Space (CASIS) has initiated and implemented the required management activities for research aboard the International Space Station (ISS) National Laboratory, and (2) the National Aeronautics and Space Administration (NASA) and CASIS measure and assess CASIS’s performance.

To determine the extent to which CASIS is performing the required management activities for non-NASA research aboard the ISS National Laboratory, we obtained and reviewed relevant legislation and documentation, and interviewed ISS program and CASIS officials. We reviewed the NASA Authorization Act of 2005, which designated the U.S. Operating Segment of the ISS as a National Laboratory; the NASA Authorization Act of 2008, which directed NASA to establish an ISS National Laboratory Advisory Committee; and the NASA Authorization Act of 2010, which required NASA to enter into a cooperative agreement with a nonprofit organization to manage the activities of the ISS National Laboratory. We also reviewed the cooperative agreement between NASA and CASIS and the CASIS fiscal year 2014 and 2015 Annual Program Plans for CASIS responsibilities related to the required activities outlined in Section 504(c) of the NASA Authorization Act of 2010. We examined the CASIS portfolio management and research prioritization process and various market analyses and studies that CASIS considered in establishing research areas. We reviewed the CASIS proposal review and evaluation process for solicited and unsolicited proposals as well as the Requests for Proposals that CASIS had issued to solicit research proposals. We studied fiscal year 2014 quarterly and annual reports to gain insight into the activities CASIS had undertaken to meet its responsibilities. We reviewed CASIS business development efforts, including funding and marketing processes and outreach efforts. We reviewed the partnerships CASIS has established with philanthropic institutions that could provide additional resources to sponsor research aboard the ISS National laboratory and implementation partners that provide logistical assistance to researchers. Additionally, we reviewed CASIS education efforts, particularly science, technology, engineering, and mathematics activities. We also reviewed GAO, NASA Inspector General, and NASA reports on sustaining the ISS. We interviewed several ISS program officials including the ISS Program Director, ISS Program Manager, ISS Program Scientist, and the Cooperative Agreement Technical Officer to gain their perspectives on the work CASIS was performing. We also interviewed officials in the Space Life and Physical Sciences Research and Applications division, including the NASA Liaison to CASIS, to gain perspective on the work NASA is sponsoring aboard the ISS. In addition, we interviewed the CASIS President and Executive Director, the CASIS Chief Operating Officer, the CASIS Chief Financial Officer, and the Chairman of the CASIS Board of Directors to better understand the processes and procedures being implemented, how proposals are evaluated, and the challenges that CASIS faces to further implement the responsibilities outlined in the cooperative agreement.

To obtain additional information on CASIS’s performance and the effectiveness of its implementation of some of the required activities, we used information provided to us by CASIS to select a random sample of 20 principal investigators who had submitted either a solicited or unsolicited research proposal to CASIS. Of the 20 researchers selected, we conducted structured interviews with 14 researchers to obtain additional insights into CASIS’s performance. Although the randomly selected researchers are, in part, representative of the population of 172 researchers who had submitted proposals to CASIS through July 2014, the descriptive nature of the responses and the relatively small sample size does not permit the development of reliable, quantitative estimates that are generalizable to the population. However, we believe our interview results provide us with valuable information about researcher’s experiences and perspectives on CASIS’s performance in the area of soliciting, reviewing and providing feedback on proposals.

To determine whether CASIS, in collaboration with NASA, has established performance metrics, we reviewed CASIS metrics as presented in its fiscal years 2013 to 2015 Annual Program Plans. We concentrated on fiscal year 2014 and 2015 metrics, but examined the previous metrics to determine how performance measures evolved. We also reviewed CASIS quarterly reports for fiscal year 2014 and the first quarter of fiscal year 2015 and the fiscal year 2014 annual report to determine how performance was measured and reported to NASA. We analyzed CASIS’s fiscal year 2014 and 2015 metrics to evaluate whether they adhered to GAO’s key attributes of successful performance measures, which were identified in previous work. Judgment was required to determine which attributes were applicable to assess and whether the performance measures met the definition of the attributes selected. To determine how NASA assesses CASIS performance, we reviewed the cooperative agreement to determine relevant NASA responsibilities, including the roles of the Cooperative Agreement Technical Officer and NASA Liaison. We also interviewed the NASA Liaison to CASIS, the Cooperative Agreement Technical Officer, and CASIS officials to gain their perspective on the evolution of metrics and how they are used to assess CASIS’s performance.

Our work was performed at NASA Headquarters in Washington, D.C., and Johnson Space Center in Houston, Texas. We also visited CASIS headquarters in Melbourne, Florida.

We conducted our review from April 2014 to April 2015 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings based on our audit objectives.

Appendix II: International Space Station National Laboratory Management Activities Required in the NASA Authorization Act of 2010

The National Aeronautics and Space Administration (NASA) Authorization Act of 2010 directed that the Administrator shall provide initial financial assistance to the organization with which the Administrator enters into a cooperative agreement to manage the International Space Station (ISS) National Laboratory. In August 2011, after a competitive process, NASA signed a cooperative agreement with the Center for the Advancement of Science in Space, Inc. (CASIS), a not-for-profit entity, to manage the activities of the ISS National Laboratory through September 30, 2020. The 2010 act outlined several management and research and development activities for CASIS, as the not-for-profit entity selected, to initiate, as follows: 1. Planning and coordination of the ISS national laboratory research activities. 2. Development and implementation of guidelines, selection criteria, and flight support requirements for non-NASA scientific utilization of ISS research capabilities and facilities available in United States-owned modules of the ISS or in partner-owned facilities of the ISS allocated to United States utilization by international agreement. 3. Interaction with and integration of the International Space Station National Laboratory Advisory Committee established under section 602 of the National Aeronautics and Space Administration Authorization Act of 2008 (42 U.S.C. 17752) with the governance of the organization, and review recommendations provided by that Committee regarding agreements with non-NASA departments and agencies of the United States Government, academic institutions and consortia, and commercial entities leading to the utilization of the ISS national laboratory facilities. 4. Coordination of transportation requirements in support of the ISS national laboratory research and development objectives, including provision for delivery of instruments, logistics support, and related experiment materials, and provision for return to Earth of collected samples, materials, and scientific instruments in need of replacement or upgrade. 5. Cooperation with NASA, other departments and agencies of the United States Government, the States, and commercial entities in ensuring the enhancement and sustained operations of non- exploration-related research payload ground support facilities for the ISS, including the Space Life Sciences Laboratory, the Space Station Processing Facility and Payload Operations Integration Center. 6. Development and implementation of scientific outreach and education activities designed to ensure effective utilization of ISS research capabilities including the conduct of scientific assemblies, conferences, and other fora for the presentation of research findings, methods, and mechanisms for the dissemination of non-restricted research findings and the development of educational programs, course supplements, interaction with educational programs at all grade levels, including student focused research opportunities for conduct of research in the ISS national laboratory facilities. 7. Such other matters relating to the utilization of the ISS national laboratory facilities for research and development as the Administrator may consider appropriate.

Appendix III: The Center for the Advancement of Science in Space (CASIS) Educational Activities

Educational Initiatives The Center for the Advancement of Science in Space (CASIS) Developed Educational Initiatives National Design Challenge Pilot Project

A national education campaign that provides educators and their students the opportunity to design and implement an authentic research experiment on the International Space Station (ISS).

Houston, Tex. – Six educators and 220 students completed experiments to fly to the ISS on Orb-3 in October 2014. Denver, Colo. – Three schools are currently developing experiments that will be sent to the ISS in spring on 2015. The pilot includes 105 middle and high school students.

Brings middle and high school students to the Kennedy Space Center Visitor Complex and the Space Life Science Lab to interact with an astronaut and research scientist to send their experiment to the ISS.

Six CASIS Academy Live events have been held at the Space Life Sciences Lab and the Kennedy Space Center for 390 Central Florida middle and high school students.

Created to educate middle school students about the ISS.

There have been nearly 15,000 total views of the CASIS Academy student website with the monthly average of 2,492 views. The educators’ webpage has a total of 1,530 views averaging 255 monthly.

Work with volunteers across the nation who communicate the CASIS mission and information about recent research conducted on board the ISS National Laboratory. Volunteers serve as pilot tester, focus group and provide local training on CASIS education programs.

Program brought awareness of the ISS and CASIS science, technology, engineering, and mathematics (STEM) activities to 450 educators and students at various workshops and presentations.

Partnership with the Professional Golfers’ Association of America Center for Golf Learning and Performance, Cobra Puma Golf and St Lucie County Schools to bring together science and golf by offering a 5-day golf summer cap to underprivileged middle school students teaching them math and physics .

Sixty-three middle school students participated in the Professional Golfers’ Association of America STEM Camp in summer 2014.

CASIS Educational Partnership Programs Zero Robotics Middle School Program

Five-week summer program for middle school students to work in teams with program staff, mentors, and scientists to learn about programming, robotics and space engineering while getting hands-on experience working with and programming Synchronized Position Hold, Engage, Reorient, Experimental Satellites.

There were 550 students and 110 teachers from 9 different states who participated in the program in summer 2014.

Offers students the ability to participate in near real-time life science research onboard the ISS to study foraging ant behavior.

The ant experiment was flown to the ISS in December of 2013. A total of 8,814 students in 32 states participated in program in FY2014.

Educational Initiatives Story Time From Space

Objectives Videotapes of astronauts reading selected stories from the ISS.

Status The videotapes were downloaded in January 2014. Fundraising efforts continue for Phase 2 in parallel with the development of the demonstration kit of materials that will complement the science content in the books. There have been 6,500 student s and educators participating in the program in 2014.

Students engage in the experiments design and proposal writing process that culminates in flying an experiment on the ISS.

CASIS is a national sponsor of Missions 5 and 6 in fiscal year 2014. This represents more than 8,000 students actively engaged in authentic research experiences. CASIS presented to 400 of these students and their parents at the Student Spaceflight Experiments Program National conference in Washington D.C.

A 4-week online program designed to take participants on a simulated mission to the ISS as “virtual astronauts”. Offered to middle and high school students and children and adults outside of the school system.

A total of 25 students and 25 educators participated in the prototype version of the Space Station Academy in July 2014.

The High School Students United with NASA to Create Hardware program is a partnership between high schools and NASA where students design, build and implement an experiment in microgravity.

The experiment is being developed by a team of students at Lakewood High School in Colorado.

CASIS entered into a partnership with National Geographic Learning/Cengage to help develop an online interactive science program for grades K-6.

Appendix IV: Assessment of the (CASIS) FY 2014 Metrics Against GAO’s Key Attributes of Successful Performance Measures

Performance Metric (18) Number of total flight projects manifested as a result of solicited proposals or investments (20) Describe intended impacts/ outcomes of ISS NL research and development to life on Earth (21) Report scientific or technological breakthroughs related to use of the ISS NL (22) Report transformational/ translational science (23) Report projects or activities contributing to national scientific, educational, or technology initiatives (26) Report new initiatives to solicit interest in/engagement with CASIS toward broader utilization of the ISS (27) Number of awards given to unsolicited proposals (28) Dollar ($) amount given to unsolicited proposals (29) Number and dollar ($) amount of awards by type of responding organization (other government agencies, academic, individual, commercial, other) (30) Dollar ($) amount contributed to projects by non-CASIS sources, and their origins (including targeted giving, commercial entities, private investments) (31) Dollar ($) amount and description of flight projects provided by other government agencies (32) Describe actual impacts of ISS NL research and development to life on Earth (specific examples, as they occur)

Appendix V: Comments from the National Aeronautics and Space Administration

Appendix VI: Comments from the Center for the Advancement of Science in Space

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Shelby S. Oakley, Assistant Director; Richard A. Cederholm; Virginia Chanley; Maria Durant; Laura Greifner; Ralph Roffo; Sylvia Schatz; and Roxanna T. Sun made key contributions to this report.