If we filter here using "<", BatchID which equals to exclusiveUB will never
be returned. Is this what the planner assumes?



On Mon, Dec 23, 2013 at 1:58 PM, P. Oscar Boykin
<notifications@github.com>wrote:

> In
> summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/DirectoryBatchedStore.scala:
>
> > +  protected def getLastBatchID(exclusiveUB: BatchID, mode: Mode) = {
> > +    mode match {
> > +      case Hdfs(_, conf) => {
> > +        def hdfsPaths: List[String] = {
> > +          val path = new Path(rootPath)
> > +          Option(path.getFileSystem(conf).globStatus(path))
> > +            .map{ statuses: Array[FileStatus] =>
> > +              statuses.map {_.getPath.getName}
> > +            }
> > +            .getOrElse(Array[String]())
> > +            .toList
> > +        }
> > +
> > +        val lastBatchStatus =
> > +          hdfsPaths.map(getFileStatus(_, conf))
> > +            .filter{input => input._1 && BatchID(input._2) <= exclusiveUB}
>
> exclusiveUB needs to be < not <=.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/twitter/summingbird/pull/407/files#r8535554>
> .
>



-- 
Kevin Lin | Twitter, Inc.
1355 Market St. | San Francisco, CA | 94103

Follow me: @reconditesea <https://twitter.com/reconditesea>