Q:

Effect of "finally" block on return values from "try" block

I was reading this question and I got the following code snippet:
public void testFinally(){
    System.out.println(setOne().toString());

}

protected StringBuilder setOne(){
    StringBuilder builder=new StringBuilder();
    try{
        builder.append("Cool");
        return builder.append("Return");
    }finally{
        builder.append("+1");
    }
}

The answer is: CoolReturn+1
Okay then I tried the same with String and int following is my code snippet with String:
public void testFinally(){
    System.out.println(setOne().toString());
}

protected String setOne(){
    String str = "fail";
    try{
        str = "success";
        return str;
    }finally{
        str = str + "fail";
    }
}

Why the answer is: success.  Why not successfail, as in 1st case finally appends value and here I am doing concatenation? 
Also I tried with primitive type int
public void testFinally(){
    System.out.println(setOne());
}

protected int setOne(){
    int value = 10;
    try{
        value  = 20;
        return value ;
    }finally{
        value  = value  + 10;
    }
}

Here too why the answer is: 20 why not 30. 

A:

TL;DR: You've already told the method what to return.  The finally block happens after you've done that.  Additionally, reference variables are just pointers to objects, and while the finally block can't change the reference itself, it can certainly change the object that the reference points to. 
Long answer:
There are two things going on here:

First, you've already told the method what value to return.  The JLS states it fairly simply, from 14.20.2 (emphasis mine):

A try statement with a finally block is executed by first executing the try block.  

That is, the try block executes completely before the finally block is run.  Also, from 14.7:

... execution of such a return statement first evaluates the Expression. 

The implication is that any and all effects the try block had (including specifying the value that the method should return) are complete, and that the value to be returned has been completely evaluated and is now "set in stone", so to speak. 
Let's concentrate on this first.  Here, we look at a slightly modified version of your primitive example.  The original example you posted isn't a great example, because the initial value 10 of the integer plus the 10 in the finally block also happens to equal 20, which clouds what is truly happening.  So let's consider this instead:
protected int setOne(){
    int value = 5;
    try{
        value = 20;
        return value;
    }finally{
        value = value + 10;
    }
}

The return of this method is 20.  Not 15, not 30, but 20.  Why?  Because in the try block, you set value = 20, then you tell the method to return 20; value is evaluated in the return statement, and its value at that time is 20.  Nothing the finally block does can change the fact that you've already told the method to return 20. 
Ok, easy. 

Now the second thing that is happening, in your other examples, is that reference variables point to objects.  That is, they are essentially primitive integer variables that hold the memory address of an object.  They follow the same rules as primitive types above!  Before we look at the rest of your examples, consider the following:
int array[] = new int[] { 100, 200, 300 };

int example () {
    int index = 1;
    try {
       return index;
    } finally {
       array[index] = 500;
       index = 2;
    }
}

This method returns 1, not 2 (for reasons explained above).  The finally block also modifies array[1].  So what, then, does value contain after the following:
int index = example();
int value = array[index];

It's 500, of course.  We can see this without much explanation being required.  The example method returns an index into an array.  The finally block modifies the data in the array.  When we look at the data at that index later, we see it contains 500, because the finally block set it to 500.  But changing the data in the array is unrelated to the fact that the returned index is still 1. 
This is exactly the same as returning a reference.  Think of a reference variable as a primitive integer that is essentially an index into a large array of memory (the heap).  Modifying the object that a reference points to is like modifying data in that array.  Now, the rest of the examples should make more sense. 
So let's look at your first example:
protected StringBuilder setOne(){
    StringBuilder builder=new StringBuilder();
    try{
        builder.append("Cool"); // [1]
        return builder.append("Return"); // [2]
    }finally{
        builder.append("+1"); //[3]
    }
}

In your question, you've stated that you are confused because this method returns "CoolReturn+1".  However, this statement fundamentally doesn't make much sense!  This method is not returning "CoolReturn+1".  This method is returning a reference to a StringBuilder that happens to contain the data, "CoolReturn+1". 
In this example, first line [1] is evaluated.  Then line [2] is evaluated, and .append("Return") is executed.  Then the finally block happens, and line [3] is evaluated.  Then, since you've already told the method to return a reference to that StringBuilder, that reference is returned.  The StringBuilder pointed to by the returned reference has been modified by finally, and that's OK.  That does not affect the value returned by the method, which is simply a reference to an object (i.e. the "index" into that big array of memory I described earlier). 
Ok, so let's look at your second example:
protected String setOne(){
    String str = "fail";
    try{
        str = "success";
        return str;
    }finally{
        str = str + "fail";
    }
}

This returns a reference to a String that contains the data, "success".  Why?  For all the reasons already described above.  This line:
str = str + "fail";

Simply creates a new String object that is the concatenation of the two strings, and then assigns str a reference to that new object.  However, as with the primitive int example, we've already told the function to return a reference to the "success" String, and we cannot change that no matter what we do!  

Conclusion:
You can come up with an infinite number of examples, but the rules will always be the same: The return value is evaluated in the return statement, and the value cannot be changed later.  Reference variables are simply values that hold memory addresses of objects, and that memory address value cannot be changed, even though the object at that address can certainly be modified by finally. 
Note also that immutability is irrelevant to the general concept here.  In the String example it is a bit of a red herring.  Keep in mind that even if Strings were mutable, we would never expect a binary + operator to modify the fields of its left operand (e.g. even if strings had, say, an append() method, a = a + b would not be expected to modify any fields of a, it would be expected to return a new object then store a reference to that in a, leaving the original a untouched).  One source of confusion here is that Java allows + on String as a convenience; no other objects directly support operators like that (not counting automatic unboxing of primitive wrappers). 
I did mark this question as a duplicate of Why does changing the returned variable in a finally block not change the return value?.  I believe that, once you have your head wrapped around the concepts here, it will be clear that the question and answer there are essentially the same as the question and answers here.  

A:

The first and second methods return the reference to the object, the finally block is executed later, what happens in the first example is that you still keep the reference to the object (the builder) so you can modify it. 
In the second example you have a string, which also is inmutable, so you are not able to modify it, only assign it a new object to the variable.  But the object which you have returned is not modified.  So when you do str = str + "fail"; you assign to your variable str a new object. 
In the third example you have an integer, which is not an object, it returns it´s value, later in the finally block you assign your variable to a new integer, but the returned one is not modified
Detailed explanation:
Imagine a fourth case:
    public static class Container{
    public int value = 0;
}

protected static Container setOne(){
    Container container = new Container();
    try{
        container.value  = 20;
        return container ;
    }finally{
        container.value  = container.value + 10;
    }
}

This function retrieves the reference to a variable called container, and after returning it it increments the container's value field to +10, so when you exit the function, the container.value will be at 30, just like in the StringBuilder example. 
Let´s compare this method with the third example (the int method):
If you get the bytecode of this two methods you get:
For the example of the int:
bipush 10
istore_0
bipush 20
istore_0
iload_0
istore_2
iinc 0 10
iload_2
ireturn   <- return point  
astore_1
iinc 0 10  <- retrieve the variable value and add 10 to it's value
aload_1  <- Store the value of the result of the sum. 
athrow

For the example of the Container wrapper class:
new Test4$Container
dup
invokespecial Test4$Container/<init>()V
astore_0
aload_0
bipush 20
putfield Test4$Container/value I
aload_0
astore_2
aload_0
aload_0
getfield Test4$Container/value I
bipush 10
iadd
putfield Test4$Container/value I
aload_2
areturn  <-- Return point
astore_1 <-- Stores the reference
aload_0
aload_0
getfield Test4$Container/value I <-- gets the value field from the object reference
bipush 10 
iadd      <-- add the value to the container.value field
putfield Test4$Container/value I <-- Stores the new value (30) to the field of the object
aload_1
athrow

As you can see, in the second case the finally statement access to the referenced variable incrementing it's value.  but in the int example it only adds 10 to the value of the variable, assigning a new value to the variable. 
I've used this example because it's bytecode is easier to be read than the stringbuffer one, but you can do it with it and you will have a similar result.