Developing Metrics for Assessing the Impact of Open Science Services Huajin Wang, Melanie Gainey, Patrick Campbell, Katie Behrman, Sarah Young University Libraries, Carnegie Mellon University, Pittsburgh, PA, USA Summary Preliminary results Methods Logic Model Metrics framework development Program overview Future assessment development Research Data Management DMP Tool Open Science Symposium AIDR Collaborative Bioinformatics Hackathon dataCoLab protocols.io Open Science Framework KiltHub R Python OpenRefine Citizen Science LabArchives APC Fund Open Educational Resources ●Usage and attendance data were collected from vendors and dashboards for the following tools and platforms: KiltHub (Institutional Repository), Open Science Framework, LabArchives, protocols.io, OSDC Newsletter, open science-themed workshops, Carpentries workshops, Open Science Symposium, and AIDR. ●The KiltHub dashboard includes departmental affiliation for all staff, faculty, and graduate students at CMU. We used that dataset to determine the departmental affiliation for users and attendees of all other platforms and events. Since undergraduates are not included in the KiltHub dashboard, we did not include their departmental affiliation and they are represented in our data as “undergraduates”. ●Data collected between January and April 2021 for each platform or event was used to generate summary data on departmental and institutional affiliation of users. ●Platform usage over time data for KiltHub, LabArchives, and protocols.io were collected in December 2021. University Libraries’ Open Science and Data Collaborations (OSDC) Program at Carnegie Mellon University (CMU) was created in 2018 to meet the growing needs and interest from researchers in making data and other research products publicly available and reproducible. The program offers five categories of support from librarians and staff with expertise in research and data sharing: tools, trainings, events, assessment, and collaboration. Our services provide end-to-end support throughout the research lifecycle. After three years of building the program, we are currently assessing the success of our initiatives with the development of a logic model and usage metrics. Development of metrics will allow us to take the scattered usage or attendance data from each platform and event and integrate them into a single dataset. We can then answer questions about who is using our services, how they are using them, and what impact the services are having on research workflows. Preliminary data suggest that users come from a broad swath of disciplines with researchers from Heinz College of Information Systems and Public Policy being the heaviest users of program services. The department with the highest number of public items on KiltHub was the Software Engineering Institute, but these deposits represented many users that each have a low number of uploads. Superusers (users with a high number of public items) came from other departments such as Psychology, Computer Science, and Materials Science Engineering. Platform usage for KiltHub, LabArchives, and protocols.io has increased each year since 2019 when we licensed protocols.io and LabArchives. Metrics data and feedback from the Advisory Group will be used to design targeted surveys. Advisory group from top users (2021) User data Targeted surveys & interviews ●How much time / money are we saving researchers ●How many grant awards / publications / career opportunities are we helping researchers to obtain ●How many communities are we helping to adopt open science ●How much are we contributing to the cultural shift ●Who uses our tools and participates our activities? ●Who are our top users? ●Which disciplines are the most engaged? ●How do people use our tools or activities? ●Why do people use our tools or activities? ●What impact are we making? Integrated user data Scattered data from each service Questions 5 Ws and 1 H Metric Variable(s) Source of Data Who User affilia