Firstly, the sinner mourns their sins before God because they genuinely feel sorry for them. It's the same guilt one feels when they offend someone they cherish. The speaker relates to the sinner's brokenness but rejoices at the same time because they know that sinner will soon be free of both sin and guilt. With their empathic nature, one can also say the persona feels the joy of heaven. After all, they speak of joining the angels to sing.