TODO: 1) separate the creation of input dir of each task from fetch 2) filter unpaired partitions