Once the Rounded Rectangle is used to make a Layer Mask around the picture the corners of the picture will be smooth. Pictures in the natural state have pointed edges and the Rounded Rectangle Layer Mask applied to the picture gives it a smooth round edge.