Q:

Will Spark Coalesce perform Shuffle

Assume I have a 8 node Spark cluster with 8 partitions (i.e each node has 1 partitions)
Now if I try to reduce the number of partitions to 4 by using coalesce(4), 
1.  Will coalesce perform shuffle ? 
2.  If yes, then in which nodes will the newly created 4 partitions reside ? 

A:

If You Check Spark API documentation of Coalesce.  Then that is following
coalesce(int numPartitions, boolean shuffle, scala.math. Ordering<T> ord)

by default the shuffle Flag is False.  Repartition calls same method by changing shuffle flag to True.  With This info, Now let us answer your question
To Change Number of Partitions From 8 to 4, Shuffle has to happen.  But here you are explicitly saying No to shuffle.  So Number of Partitions in This case will not change.  
even If you Try to Increase number of Partitions, It will not change.  Since shuffle flag is False.  Hope This Helps
Cheers! 

A:

Coalesce by default has shuffling flag set to False.  
If you have to increase the partitions, you can either use coalesce with shuffle flag set to true(with false, partition remains unchanged) or use repartition
If you are decreasing partitions, better to use coalesce with flag set to False as it avoids full shuffle unlike repartition where shuffling is guaranteed. 
Coalesce with false shuffling moves data on 1 partition to another existing partition thereby avoiding full shuffle giving better performance. 
say, data from partitions 5,6,7,8 will be moved to existing partitions 1,2,3,4 rather than shuffling the data of all 8 partitions
Determining on which node data resides is decided by the partitioner you are using

