timrdf/csv2rdf4lod-automation — Issue #242: cr-create-versioned-dataset-dir.sh should recognize file extension via conneg and do the right thing
Author: olyerickson
Created: 2011-10-11 12:08:58+00:00
State: open

Sometimes the URLs of targets don't indicate the actual format of the file being downloaded; sometimes the text on the HTML page is not correct (e.g. labeling a zipped CSV as a CSV). The user must experiment, first executing a "raw" pcurl.sh, then a pcurl.sh with -n an -e params, followed by a punzip.sh.

A use case is Data.gov dataset 2203 http://explore.data.gov/download/eqfa-2yin/CSV

Currently cr-create-versioned-dataset-dir.sh (called by retrieve.sh) does not properly handle such datasets. Desired outcome:

* Be able to specify desired file name (for resulting csv) at the retrieve.sh level
* allow cr-create-versioned-dataset-dir.sh to do the right thing, i.e. figuring out that the unknown download is a zip (in the above use case, it is revealed during conneg), name the zip properly ("2203.zip"), and generate the csv properly ("2203.csv"). And of course all the pml...