TITLE: Evaluating Collaboratory Cultures
AUTHORS: M Fox, Phil Bourne

[DESCRIPTION]
The disciplinary silos in Higher Education were organized around defined fields of study in order to facilitate student instruction—law students receive their education in the Law College, medical students at the School of Medicine, etc.  But these traditional arrangements are not appropriate for 21st century science, which is organized around research relationships in order to optimize diversity and accelerate innovation.  The collaboratory structure, such as the School of Data Science (SDS) Collaboratories, strikes a balance between UVA’s pride in tradition and its Great and Good strategic plan to celebrate its tricentennial as the world’s best public university. 
But does this innovative structure provide a measurable improvement to tradition?  How do collaboratory practices facilitate future trends in science, such as convergence research with international teams and cross-functional capabilities?  How can UVA students, faculty, trustees, and administration use insights into their collaboratory functioning to improve decisions?  Looking forward to 2030, how can UVA leverage its unique advantage in collaboratory expertise toward becoming the nation’s premier Collaboratory Research Center with training, certification, and services for industry, academia, civil society, and government?
The topic of collaboration as a research subject has been well-studied in most fields, and some metadata about collaboratories is well-documented (e.g., h-index, bibliometrics, award amounts,  recipient demographics, etc.); however, metrics and reference data that can be used to accurately assess the functioning of the collaboratory unit have yet to be established and modeled.  Indeed, there is not a single published dataset or evidence-based best practice to inform the design and/or operation of the research collaboratory.  In short, the collaboratory as a subject of scientific research remains unexamined.  This systemic lack of performance metrics results in persistent shortcomings that pervade every level of decision-making—from program selection by graduate school applicants to strategic planning within the Office of the President.

[STEPS]
SECTION: Landscaping | Facilitation | Framework Development &amp; Testing
1. Phase I: Landscaping
This phase of the study is focused on document review to develop benchmarks and semi-structured interviews to examine the assumptions, values, practices, challenges, needs, perceptions, and services that constitute UVA’s research landscape.
SECTION: Landscaping | Facilitation | Framework Development &amp; Testing
1.1. Document Review
Published policies and documents relating to interdisciplinary research initiatives, guidelines, and services will be found posted online in UVA department sites and at the UVA Library.Unpublished assessments, reports, meeting notes, strategic plans, and other documents will be found offline and may be identified and produced upon request by individuals involved in the decision-making process.
SECTION: Landscaping | Facilitation | Framework Development &amp; Testing
1.2. Semi-structured Interviews
Findings from the Document Review process will inform the specific set of questions posed to members of the UVA research community. Likewise, the responses to the interviews will inform the development of baseline metrics to capture the current state of interdisciplinary values and commitments on campus, and then measure progress toward community-defined goals going forward. Nonetheless, a preliminary line of inquiry has been prepared to address the practice and perception of interdisciplinarity on campus.
SECTION: Landscaping | Facilitation | Framework Development &amp; Testing
2. Phase II: Facilitation 

SECTION: Landscaping | Facilitation | Framework Development &amp; Testing
3. Phase III: Framework Development & Testing 
The first Collaboratory Cultures framework that we will test is the FAIR + OA Framework. This approach applies FAIR data science principles to the Open Access movement and expresses the framework output as a series context-specific of use cases and applications.
At this time, analysis of semi-structured interviews and surveys to be conducted in Phases I-II will be required to develop framework outputs.In the future, we plan to develop tooling that will automate context-specific use case and application outputs, similar to the bottom two rows of the example rubric above.

SECTION: Landscaping | Facilitation | Framework Development &amp; Testing
3.1. Surveys
Assign a value to each practice (No=0 | Yes=1)

Communication practices 
Establish and explain agile communication practices early 
Co-create the communication platforms and tools strategy
Define the purpose of the collaboration
Describe, debate, and refine purpose in the first team meeting
Elicit co-creation/participatory design input early and often
Assign roles and clarify responsibilities for all collaboratory members involved in the project lifecycle, including principal investigators and team leads 
Outline benchmarks of success (i.e., project milestones and schedule of deliverables)
Introduce collaboration tools and how they relate to the purpose of the project, such as communication platforms and meeting schedules
Provide collaboratory orientation (team introductions, SOP, tool and method training info)

 
Video conference practices
Optimize productivity by adopting agile techniques and anti-fatigue good practices
    oDisable mirror cam function during meetings
    oDisable cams during presentation of meeting materials
    oReduce cam time as much as possible to pull attention to agenda topics and reduce bandwidth issues
Prioritize inclusivity by normalizing privacy practices, supporting cultural differences, and offsetting nonverbal cues
    oNever ask a participant to turn on their cam if they have chosen to participate off camera
    oEnforce the use of the “Raise Hand” function to support uninterrupted speech sans nonverbal cues
Provide secure, stable links to project meetings (including agenda) for asynchronous participation 

Participation best practices
Always ask for clarification when you are uncertain
Offer your own disciplinary input to help refine or expand issues
Respect and encourage a diversity of opinions, backgrounds, and experiences 
Avoid jargon when a synonym can be used
When jargon is unavoidable, define terms 
Address conflicting ideas or approaches as they arise and employ co-creation techniques in the resolution process

Communicating multidisciplinary concepts
Schedule enough time in the meeting agenda to introduce new terms and concepts 
Practice iterative review to raise competency across the entire collaboratory
Issue a project handbook or SOP and add terms and definitions, as necessary

Collaboratory tools
Document work on shared docs that allow multiple users to take notes, edit, and comment in real time.
Allow team members to schedule meetings, set deadlines, assign time-sensitive tasks, and track project milestones on a shared calendar. 
Use instant messaging when it is the prescribed method of communication in the SOP (avoid using IM as an ad hoc communication tool to prevent fatigue)
Use data repositories to track version control, multiple threads, and updates.
Practice project archiving to establish a stable record of project assets and to track progress via reflexive practice.
Establish reproducible and executable software configuration practices, commonly referred to as containerization, bundles together software, configuration files, dependent versions, and data so that projects can be reproduced, regardless of future changes in operating systems and software versions. 
Preserve data assets with computational notebooks and tutorials that combine code, results, and descriptive text into a computational narrative. 
Use scientific discovery platforms to enable collaboration with big data assets.

Data management best practices
Make data as FAIR as possible. 
Revisit the data management plan frequently during the project and make changes as necessary.
Consider legal, ethical, and cultural obligations when drafting sharing policies.
Implement a stewardship best practices strategy and reflexive methodology at the project outset.

Code writing practices
Good practices for creating sustainable, accessible codes:
    oProvide a step-by-step user manual for tools whenever possible
    oAlways maintain an up-to-date Readme file where latest updates and requirements are listed
    oProvide high-level comments at the beginning of each file and throughout the code as needed
    oFollow consistent naming convention across your codes
    oDo version-control
Data curation:
    oAutomate as many of the processes involved in data access, storage, and reformatting as possible.
    oKeep separate copies of the original (raw) data and the curated data
Data analysis:
    oResearch and employ common, successful analysis methods.Do not reinvent the wheel.
    oMap your method to the research questions you are trying to answer.Do not try to fit your method to an application.
    oMake a tutorial-style document that explains your analytical method in simple language.
Data visualization practices:
    oUse visual graphics to communicate results with collaborators. 
    oUse high quality formats to produce images.
    oAutomate data visualization as much as possible.
    oAim for users being only a single click away from reproducing everything.
Build stand-alone tools:
    oIf possible and where appropriate, build a Graphical User Interface (GUI) that allows your collaborators to tweak parameters and apply their expertise to parameter evaluation and exploratory analyses.
    oWhen appropriate, build add-on packages and libraries, or even lean software tools.

Participatory design practices
Create a flexible-by-design, agile management framework that can accommodate variable scope and unanticipated results. 
Specify the distinct contribution that each collaborator has to offer to their field.
Identify inclusive objectives and/or outputs that allow each contributor to advance their own professional goals and research agendas.
Specify how the results of collaborative research, including data science methods, will ultimately be evaluated by disciplinary experts.
Ensure that individual contributions will contribute to the research portfolio of the data science practitioners, and vice versa.
Revise and improve the research plan whenever new collaborators are onboarded.

Roles and responsibilities
Co-create a project management plan with milestones and deadlines that lead to clearly defined project goals by assigning roles and tasks according to the strengths and interests of each team member.
Invest in the personal contribution of individual team members by working one-on-one to design and refine assigned tasks.
Check-in regularly to ensure that each individual is on track with the tools and resources they need to succeed, and adjusting the plan as needed.