thehyve/transmart-data — pull request #27 by cataphract (2014-07-16 21:15:19+00:00)
Title: External studies

* **Do not stored the studies params files in transmart-data**. These should now be included inside the tarball. Example of an old tarball:

```
.
`-- clinical
|-- E-GEOD-8581.sdrf-rewrite.txt
|-- E-GEOD-8581_columns.txt
`-- E-GEOD-8581_wordmap.txt
```

Example of a tarball in the new format:
```
.
|-- clinical
| |-- E-GEOD-8581.sdrf-rewrite.txt
| |-- E-GEOD-8581_columns.txt
| `-- E-GEOD-8581_wordmap.txt
`-- clinical.params
```

* **Do not store the list of datasets inside transmart-data**. This is replaced by a system of _feeds_. Feeds are listed in `samples/studies/public-feeds` and, optionally, in a git-ignored `private-feeds`, in the root of the project. Each file contains a list of feeds in the following format:

```

```

The two supported feed types right now are `http-index` and `ftp-flat`. Examples:
```
http-index http://studies.thehyve.net/datasets_index
ftp-flat ftp://studies.thehyve.net/
```

The type `http-index` points to a plain text file that lists the available datasets in the format ` `. Example:
```
Cell-line acgh Cell-line/Cell-line_acgh.tar.xz
Cell-line clinical Cell-line/Cell-line_clinical.tar.xz
```

The type `ftp-flat` points to an FTP directory that should store, directly underneath it, all the dataset tarballs.

The list of datasets are automatically downloaded when needed, but it is not automatically updated. To update it, run at the root:

make update_feeds

* **Study data is stored under `samples/studies`**. The dataset files are no longer downloaded into `samples/common`. They are now instead downloaded into `samples/studies`. All the subdirectories underneath `samples/studies` are git-ignored.

* **Provided repository maintenance scripts under `samples/maint`**. Read the [README](samples/maint/README.md) for more information.