Thanks Matei for the comments. I will address it today whenever I find a time. 
 My major problem is the following. I appreciate the suggestions. 

def execute() = {
    /**
     *  mutableRow is GenericMutableRow type and only created once.
     *  mutableRow is upadted at each iteration inside the while loop. 
     */
    val mutableRow = new GenericMutableRow(attributes.length)

    val res = inputRdd.context.runJob(inputRdd,(iter: Iterator[_]) => {
      /** rddBuffer keeps track of all the transformed rows.
       *  needed later to create finalRdd 
       */
      val rddBuffer = mutable.ArrayBuffer[Row]()
      while (iter.hasNext) {
        val row = iter.next()
        val values = row match {
          case Array(deserializedRow: AnyRef, partitionKeys: Array[String]) =>
            attributeFunctions.map(_(deserializedRow, partitionKeys))
          case deserializedRow: AnyRef =>
            attributeFunctions.map(_(deserializedRow, Array.empty))
        }
        var i = 0
        val len = values.length
        while ( i < len ) {
          values(i) match {
            case n: String if n.toLowerCase == "null" => mutableRow.setNullAt(i)
            case varchar: org.apache.hadoop.hive.common.`type`.HiveVarchar =>
              mutableRow.update(i,varchar.getValue)
            case decimal: org.apache.hadoop.hive.common.`type`.HiveDecimal =>
              mutableRow.update(i,BigDecimal(decimal.bigDecimalValue))
            case other => mutableRow.update(i,other)
          }
          i += 1
        }
        rddBuffer += new GenericRow(mutableRow.toArray)
      }
      rddBuffer
    })
    /** finalRdd ... equivalent to Rdd generated from inputRdd.map(...) */
    val concatArray = Array.concat[mutable.ArrayBuffer[Row]](res)
    inputRdd.context.makeRDD(concatArray)
    
  }

In the execute() method of hiveOperators.scala, I take an iterator over all the partitions in the inputRDD and pass it to the runJob() method as argument. The return from runJob() method is an Array[ArrayBuffer[Row]]. I need to generate the output rdd from that and the only way I can think of is concatenating all the ArrayBuffer[Row]s and generating one Array[Row] and then generating a new finalRDD from that. I don't think this is the right way to do that ( by doing concatenation, we are kind of localizing the already partitioned data and then again partition it by creating a new RDD ) ... Does anyone have a suggestion? 

This is the main reason that I cannot pass the regression tests ...