okfn/openspending — Issue #277: Private datasets
Author: pudo
Created: 2011-10-29 13:28:34+00:00
State: open
Labels: feature.simplified-load

I'm creating this as a debate stub for the question of whether or not we should have private datasets on OpenSpending. There are several levels at which this could be implemented, some of which have alternative solutions:

* Permanently private datasets, no big use case here.
* Embargoed datasets, i.e. private until creator decides to publish. This is a clearly set condition expressed by virtually all journalists we have spoken to regarding the use as OpenSpending as a research platform for arbitrary datasets. The alternative is promoting OS as a place to research already published datasets such as UK25k or FTS, but I have not yet met anyone who was actually interested in this. Journalists are incredibly interesting here because they have a strong motivation to find *interesting* datasets and to work with them. Giving them a tool where they can do that and then publish actually is a huge promotion of the idea of publication when compared with the status quo (Excel files on hard disks or in Fusion Tables). Of course, incentives could be found to promote the release of data at the earliest stage (extreme case: pay for private).
* Review datasets, i.e. private until accepted by customer or checked by OS core team. While most of this could also be done by using separate systems, that approach has many disadvantages: you always need to do the work twice or implement a mirroring service which is easy in theory but incredibly hard in practice. Publication is not a simple operation, but a whole second iteration on the process. If our goal is to have 10 datasets in a year, this is OK. If our goal is to have 500, then it just means we have to hire many people to do the sysadmin involved in this. Having two systems has AFAIK never worked for a community project, not Wikipedia, not OSM not anything else.
* Private-during-load datasets. There is no alternative I know to this, other than renaming the flag for ideological reasons.

All of these workflows require pretty much the same core features, which have these advantages:

* Fine grained publication control,
* Built-in QA workflow,
* Plausible SpendingStories pitch,
* Possibility to re-combine sandbox and (community) production

The disadvantages are:

* Ideological concerns about promoting closed knowledge - which I think are unfounded as the greater danger here is in semi-publication (i.e. vis but no raw data etc.), not in a structured workflow, which is actually a means of promoting publication with some training wheels.
* Need for authorization checks on page views. - ~40 SLOC