# Presentation: 182528

## Archiving Experiment and Data Access

- What are the requirements for a Data Architecture?
- Storage
- Raw Data – Stored forever and available anytime on request (immiedately). Location Irrelevant.
- Many Copies will exist. Must determine who has the master copy and which are replicas.
- Is a copy complete? Users need to know.
- Requirement: determine what needs to be stored. Must do your curation well.

- Geist

## Archiving Experiment and Data Access

- Need hashes of data to insure it has integrity.
- What about data from simulations and analysis? Does SNS have responsibility for part or all of that?
- Must determine the lifetime of the data. Set data retention periods: raw, reduced, analyses,...

- Geist

## Archiving Experiment and Data Access

- Define a Metadata minimum.
- Will be instrument specific
- Every bit of Data will be saved in a Nexus file.
- Thus metadata will be preserved.
- Temp. slit settings, ...
- Should we connect information about runs, users, proposals, samples, ...
- In Nexus?
- Indirectly?
- ...?

- Geist

## Archiving Experiment and Data Access

- Keep Brent’s data until he dies.
- Is there a need to keep raw data for ex-post re-calibration  efforts? ---Y
- Raw file needs to insure that it is write-protected.
- Is Metadata attached to raw data changeable? A: No, but you can annotate.

- Geist

## Archiving Experiment and Data Access

- Def: Raw data, by def doesn’t change. Other metadata that could be annotated, would by def be non-raw data.

- Geist

## Archiving Experiment and Data Access

- Goal is science: Knowledge is fusion of Th, exp, sim
- Archive & Access needed at
- Raw data
- Metadata
- Information

- Geist

## Archiving Experiment and Data Access

- We need to think of SNS data as community data. We may discover more in the data afterward.
- Analogy: Genomes and genome annotations.
- Policy is needed for a data “safe harbor” period.
- < 5 years
- Maybe 1-2 months
- Astronomy is 2 years (or is it 5?)
- Make sure you don’t create incentive to hide meta-data.
- Raw data is available at ISIS now.
- Propose to release Data elements at publication.
- Remain private until reasonable publication time – 2 years.
- What is definition of private: unviewable or not allowed to be used? <we didn’t answer>

- Geist

## Archiving Experiment and Data Access

- Methods:
- Portals
- Electronic Notebooks
- Direct Access
- Stand alone tools
- Mirrored datasets.

- Geist

## Archiving Experiment and Data Access

- Raw Data storage:
- Should it be the responsibility of the user? Controversial.
- Raw data is of much greater value when it is documented by publication and SW and versions used to reduce, analyze, ...

- Geist

## Archiving Experiment and Data Access

- If you want to access control data, then how?
- PW protect for a year, then free. User can free sooner.
- Or Track it by user.
- ???
- Look at existing tools, like  Secure Oracle
- Sophisticated joins and queries.
- Field by Field Security

- Geist

## Archiving Experiment and Data Access

- Facility Services:
- Lifetime issues
- Facility
- Beamlines
- Data
- SW, community codes
- Linus/Linux as a Model: Dist. Dev, single kernel.

- Geist

## Archiving Experiment and Data Access

- Who sets policy over data?
- Community
- User
- Facility
- Different entities set policies on different parts of data.
- Make data proprietary only if user pays fully burdened beamtime cost.
- Does data access depend on use of instrument or on nature or sample?
- Cannot decide these policy issues here.
- Storage requirements and security requirements may be dramatically altered by technology advances.

- Geist

## Archiving Experiment and Data Access

- Who is the proper data steward: user or facility?
- Not both.
- Franz: User is responsible
- No-one but the user uses the data
- Others: facility is responsible
- Metadata is more interesting than data often.

- Geist

## Archiving Experiment and Data Access

- Need scientific annotation – accurate and complete data
- Who certifies that D & MD is complete?
- Funding Agencies (esp. Europe) looking at data as the _product_ of their funding.
- I’ll show you mine if you’ll show me yours.
- Should be standard
- If someone publishes from my data – that’s life!?
- We can learn from bad experiments.	“My life’s purpose is to be a warning to others”

- Geist

## Archiving Experiment and Data Access

- Need to be flexible to adjust to changing statutory and regulatory requirements on data retention.
- Annotation into data and metadata.
- What about protected information?
- Black?
- Proprietary?
- Propose Facility policy is to delete it from archive.

- Geist

## Archiving Experiment and Data Access

- Tool interfaces (part of data architecture)
- Data Formats (plural)
- Encourage defining a leadership on standardization.
- NeXus
- Define Data transfer and tranform protocols
- Define how to:
- Meta-search for data
- Backup data
- Localize data for analysis

- Geist

## Archiving Experiment and Data Access

- Need to define:
- Path forward
- Data storage needs
- 26 TB raw data
- How much metadata?
- Lifetime(s)
- Define who is responsible for what data?
- Where? – anywhere near a big pipe.
- Raw storage at SNS facility probably
- May reveal data via other means.
- Need to integrate data from multiple facilities.
- Solve in a community fashion.

- Geist

## Archiving Experiment and Data Access

- Computing needs:
- How much?
- SNS provides or do others provide?
- We will use large machines if we have problems and SW that solve it.
- If not provided at facility, does facility budget for it?
- Does a user submit a proposal for beamtime and Cycles?
- Probably yes
- Dual allocation
- Need time definite availability
- Remote computing requires data winnowing to save network BW.
- Post-processing needs to go on at SNS
- Simulation data might be disconnected.
- Data mining is a tougher problem
- ISIS uses can use cluster cycles now. Insatiable appetite for cycles.
- Want “personal supercomputer”
- Work with Synergy with Computational groups
- Maintain SW

- Geist