snowplow/snowplow — pull request #624 by mrwalker (2014-04-07 13:37:04+00:00)
Title: RDS postgres loading

This PR hard-codes RDS postgres loading via streaming local files into stdin. We used this for several months at Radico to load a postgres data warehouse before moving to Qubole. Additional work needs to be done to surface this as an option rather than hard-code it.

This PR has two commits; one that was our initial attempt at loading RDS postgres, and the second that addressed an issue we had with silent PGRES_FATAL_ERRORS that did not fail the Jenkins job that was running the load.

An example failure from the first era (note the build continues):
```
Loading Snowplow events into radico-postgres (PostgreSQL database)...
Opening database connection ...
Running COPY command with data ...
Result of COPY is: PGRES_COMMAND_OK
Opening database connection ...
Running COPY command with data ...
Result of COPY is: PGRES_FATAL_ERROR
Archiving SnowPlow events...
```

And from the second (note the build halts with an error):
```
Loading Snowplow events into radico-postgres (PostgreSQL database)...
Opening database connection ...
Running COPY command with data from: data/run=2014-03-23-00-18-19/part-00001
Result of COPY is: PGRES_FATAL_ERROR
Running COPY command with data from: data/run=2014-03-23-00-18-19/part-00002
Result of COPY is: PGRES_FATAL_ERROR
Running COPY command with data from: data/run=2014-03-23-00-18-19/part-00000
Result of COPY is: PGRES_FATAL_ERROR
Build step 'Execute shell' marked build as failure
```

The actual underlying cause of the above issue is revealed earlier in the log, indicating that it might be necessary to do some more work on this PR to better expose the cause of PGRES_FATAL_ERRORs:
```
WARNING: pgstat wait timeout
PG::Error error executing VACUUM ANALYZE atomic.events;: ERROR: could not extend file "base/16388/16395_vm": No space left on device
HINT: Check free disk space.
Downloading SnowPlow events...
```

Hope this helps!