Oh alright, so the reason you're skipping the first is because we dont want
an hour with incomplete data in druid. It makes sense.


On 10 December 2013 11:28, Hagen Rother <notifications@github.com> wrote:

> In dumbo.rb:
>
> > +        result[:events] += event_count
> > +      end
> > +    end
> > +  end
> > +
> > +  results
> > +end
> > +
> > +conf[:db].each do |db_name, options|
> > +  puts "Scanning #{db_name}"
> > +  intervals = []
> > +  files = []
> > +
> > +  hdfs_content = scan_hdfs(options[:input][:path], conf[:check_window_days])
> > +
> > +  hdfs_intervals = hdfs_content.keys.sort[1...-1].map{|check_time| [check_time, check_time + 1.hour]} # skip first and last hour
>
> That's legacy from the old importer. As that randomly started at some
> hour, the first segment is always assumed to be incomplete.
>
> Given camus, cleaning up the beginning of import is a trivial "rm", so
> refactor to [0...-1]
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/liquidm/druid-dumbo/pull/1/files#r8228979>
> .
>



-- 
*Fabio Roger Manera* Senior Software Developer | LiquidM
------------------------------
LiquidM Deutschland GmbH
Paul-Lincke-Ufer 39-40 | 10999 Berlin | Germany
twitter<https://twitter.com/goliquidm> |
facebook <https://www.facebook.com/pages/LiquidM/1422869821267572> |
LinkedIn<http://www.linkedin.com/company/3488199?trk=tyah&trkInfo=tas%3AliquidM%2Cidx%3A1-2-2>
Phone:+44 7511 02 60 62 Internet:www.liquidm.com LiquidM is hiring,
check: LiquidM
Career Center <http://www.liquidm.com/#careers>