elasticsearch/logstash-contrib — pull request #117 by DaveTaboola (2014-09-29 14:39:48+00:00)
Title: New features to google_bigquery.rb

I've added\changed the following features since my last commit:
1. added table_expirationTime parameter (and handling), which sets the epoch time for the table deletion.
the action is being done by google itself, once the epoch reach the designated time.
the parameter accepts strings in the following format
a. d - days starting the current time
b. w - weeks starting the current time
c. m - months starting the current time
d. y - years starting the current time

2. changed dataset and table creation once not present.
found out that the query for dataset and table present isn't working
therefore every load job indicates for table creation, instead the query was fixed and the dataset\table are created prior to the data loading

3. gzip compression
as bigquery can handle compressed files, i've changed my previous version of gzip compression to be handled by the OS
the main reason for that was the fact that the compression processes made the logstash application to reach the memory heap to the point it failed (in large fails) therefore i've changed that to run in os mode instead