NICTA/scoobi — Issue #16: StackOverflowError when using Join or CoGroup
Author: agemooij
Created: 2011-12-04 21:32:42+00:00
State: open

Today I tried applying the new Join operations on my data but I ran into StackOverflowErrors every time.

I have two DLists. The first contains a mapping of low-level id to higher-level id, resulting from an earlier processing step. the second one contains a mapping of low-level id to a PageView instance, which is my raw data. I would like to create a join to get a DList containing (higher-level id, Pageview) pairs so I can later group by higher level ids, combine the pageviews, and do my final analysis. See the code below for an impression.

When I run this (with standalone hadoop), I get a stack overflow (see stacktrace below). I looked at the code for both Join and CoGroup and I narrowed it down to the following code:

val joined = (d1s ++ d2s).groupByKey ...

The `groupByKey` causes the stack overflow, although I have no idea why. Since this code is shared between Join and CoGroup, both operations result in the stack overflow.

Unfortunately, this is where my insight in how Scoobi works ends (at the moment). I hope you guys can help because I _really_ don't want to go back to plain Hadoop or Cascading.

Regards,
Age

#### The Code
...
val visitorIdToTrueVisitorId: DList[(Long, Long)] = ...
val visitorIdToPageView: DList[(Long, PageView)] = ...

val joined = join(visitorIdToTrueVisitorId, visitorIdToPageViews)
...

#### The Stack Trace

Exception in thread "main" java.lang.StackOverflowError
at com.nicta.scoobi.impl.plan.AST$GroupByKey.gd7$1(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.equals(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$Combiner.gd3$1(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$Combiner.equals(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.gd2$1(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.equals(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.gd7$1(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.equals(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$Combiner.gd3$1(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$Combiner.equals(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.gd2$1(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.equals(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.gd7$1(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.equals(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$Combiner.gd3$1(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$Combiner.equals(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.gd2$1(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.equals(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.gd7$1(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.equals(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$Combiner.gd3$1(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$Combiner.equals(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.gd2$1(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GbkMapper.equals(AST.scala:73)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.gd7$1(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$GroupByKey.equals(AST.scala:175)
at com.nicta.scoobi.impl.plan.AST$Combiner.gd3$1(AST.scala:92)
at com.nicta.scoobi.impl.plan.AST$Combiner.equals(AST.scala:92)
at scala.collection.immutable.Set$Set1.contains(Set.scala:76)
at com.nicta.scoobi.impl.plan.MSCR$$anonfun$containingOutput$1.apply(MSCR.scala:90)
at com.nicta.scoobi.impl.plan.MSCR$$anonfun$containingOutput$1.apply(MSCR.scala:90)
at scala.collection.Iterator$class.find(Iterator.scala:709)
at scala.collection.immutable.TrieIterator.find(TrieIterator.scala:20)
at scala.collection.IterableLike$class.find(IterableLike.scala:80)
at scala.collection.immutable.HashSet.find(HashSet.scala:33)
at com.nicta.scoobi.impl.plan.MSCR$.containingOutput(MSCR.scala:90)
at com.nicta.scoobi.impl.exec.Executor$$anonfun$com$nicta$scoobi$impl$exec$Executor$$executeMSCR$1.apply(Executor.scala:112)
at com.nicta.scoobi.impl.exec.Executor$$anonfun$com$nicta$scoobi$impl$exec$Executor$$executeMSCR$1.apply(Executor.scala:110)
at scala.collection.immutable.Set$Set2.foreach(Set.scala:106)
at com.nicta.scoobi.impl.exec.Executor$.com$nicta$scoobi$impl$exec$Executor$$executeMSCR(Executor.scala:110)
at com.nicta.scoobi.impl.exec.Executor$$anonfun$com$nicta$scoobi$impl$exec$Executor$$executeMSCR$1.apply(Executor.scala:112)
at com.nicta.scoobi.impl.exec.Executor$$anonfun$com$nicta$scoobi$impl$exec$Executor$$executeMSCR$1.apply(Executor.scala:110)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at com.nicta.scoobi.impl.exec.Executor$.com$nicta$scoobi$impl$exec$Executor$$executeMSCR(Executor.scala:110)
at com.nicta.scoobi.impl.exec.Executor$$anonfun$com$nicta$scoobi$impl$exec$Executor$$executeMSCR$1.apply(Executor.scala:112)
at com.nicta.scoobi.impl.exec.Executor$$anonfun$com$nicta$scoobi$impl$exec$Executor$$executeMSCR$1.apply(Executor.scala:110)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at com.nicta.scoobi.impl.exec.Executor$.com$nicta$scoobi$impl$exec$Executor$$executeMSCR(Executor.scala:110)
...

Those last four lines repeat until the stack overflows.